For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Learn about LLM gateway concepts, supported providers, and common enterprise challenges that agentgateway solves.

Agentgateway provides seamless integration with various Large Language Model (LLM) providers. This way, you can consume AI services through a unified interface while still maintain flexibility in the providers that you use.

What is an LLM?

Large Language Model (LLMs) are very large, deep learning models that are pre-trained on massive amounts of unlabeled and self-supervised data, often containing billions of words. LLMs extract meanings from a sequence of text and can understand the relationship between words and phrases in it. Based on this understanding, the LLM can then predict and generate natural language, and perform related tasks, such as answering questions, summarizing text, translating between languages, or writing content. LLMs can be fine-tuned on a smaller, more specific data set so that it can perform a more specific task more accurately.

While LLMs were originally designed to be trained on text and natural language, they are now becoming multimodal, supporting different media types, such as images and videos.

Common problems with LLM consumption

When adopting an LLM on an enterprise-level, you typically run into the following issues:

• Data leakage and privacy: When interacting with LLMs, it is essential to comply with federal laws and to not leak any PII (personally identifiable information) or other sensitive information in prompts. For example, users might enter personal credit card or customer related information, such as account numbers and names. Attackers might also trick the model into revealing previous user data that is still stored in the context. To prevent sensitive information from leaking to the LLM provider, prompt guards must be in place that help to filter, block, monitor, and control LLM inputs and outputs to find offensive content, prevent misuse, and ensure ethical and responsible AI usage.

• Cost controls (budget and spend limits): Compared to traditional APIs, LLM queries are long-running, meaning the LLM must parse through its large data set iteratively to compute one token at a time. All tokens must then be summarized to generate a prediction. Given the large amounts of data that the LLM parses, GPUs are required to run LLMs efficiently, which are costly to run and maintain. Limiting the amount of tokens that can be used and monitoring token consumption are critical to keep costs to a minimum.

• Fan-out patterns: LLMs are limited by static training data. This limitation can make seemingly simple questions difficult to answer. For example, answering a simple question, such as What is the weather today? requires the LLM to have access to several real-time information, such as the location, time, and weather forecast. Integrating LLMs with other MCP servers and agents to perform these tasks becomes essential to accurately answer questions of this nature.

• Security: To protect your LLM from being consumed by unauthorized users, you typically want to authenticate with the LLM before you start sending requests. Managing credentials, such as API keys, and monitoring the traffic to the LLM becomes essential to protect access to the LLM .

• Scalability and reliability: When relying on an LLM to perform specific tasks in your environment, you must put measures in place that prevent your applications from failing, being overwhelmed, or performing inefficiently. This includes retries and timeouts, request rate limiting, and multi-model failovers.

Supported providers

Review the following table to compare agentgateway’s support of different LLM provider APIs.

Provider	Chat Completions	Responses	Messages	Embeddings	Realtime	Count Tokens	Rerank
OpenAI	✅	✅	✅¹	✅	✅	✅²	-
Anthropic	✅¹	◇	✅	-	-	✅	-
Bedrock	✅¹	✅¹	✅¹	✅¹	-	✅⁴	✅¹
Azure	✅	✅	✅¹	✅	-	✅²	⚠️³
Gemini	✅	✅¹	✅¹	✅	-	✅²	-
Vertex AI	✅⁴	◇	✅⁴	✅¹	-	✅⁴	✅¹
Copilot	✅	✅	✅¹	◇	-	✅²	⚠️³
Cohere	✅	✅¹	✅¹	✅	-	✅²	✅
Ollama	✅	✅	✅¹	✅	-	✅²	-
Baseten	✅	✅¹	✅	-	-	✅²	-
Cerebras	✅	✅¹	✅¹	-	-	✅²	-
Deepinfra	✅	✅¹	✅	✅	-	✅²	-
Deepseek	✅	✅¹	✅	-	-	✅²	-
Groq	✅	✅	✅¹	-	-	✅²	-
Hugging Face	✅	✅	✅¹	-	-	✅²	-
Mistral	✅	✅¹	✅¹	✅	-	✅²	-
OpenRouter	✅	✅	✅	✅	-	✅²	✅
Perplexity	◇	✅	✅¹	-	-	✅²	-
Together AI	✅	✅¹	✅¹	✅	-	✅²	✅
xAI	✅	✅	✅¹	-	✅	✅²	-
Fireworks	✅	✅	✅	✅	-	✅²	✅
Meta	✅	✅	✅	◇	◇	✅²	◇

Legend:

How requests reach a provider

Clients can call the gateway in several LLM API formats, such as OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and native Gemini. The route type tells the gateway which format the client sent. On an AgentgatewayModel, the standard serving paths, such as /v1/messages, set the route type. On an AgentgatewayBackend, you map paths to route types in policies.ai.routes. When the provider supports that format, the request is sent to the provider unchanged in format. When the provider does not support it, the request is converted to a format that the provider supports, and the reply is converted back to the format that the client called. For the route types, see API types.

Query strings on converted requests

A client query string belongs to the API format that the client called. For the Completions, Messages, Responses, and GenerateContent route types, the client query string is kept when the provider receives the same format and is dropped when the request is converted to another format. For example, a Messages request to /v1/messages?beta=true that is converted to Chat Completions for a Gemini provider reaches the provider without ?beta=true.

These cases differ:

• An Amazon Bedrock request on the Runtime endpoint always drops the client query string, because Runtime serves only the Bedrock APIs.
• A built-in provider on an AgentgatewayBackend that sets host without pathPrefix receives the client path and query string unchanged.
• A Passthrough route forwards the client query string, because no conversion happens.

Choose how to expose models

Clients select a model with the request’s model field. Choose a matching pattern to control which models they can request and how you name them.

Each pattern sets the name that clients request in an AgentgatewayModel. For a wildcard pattern, set spec.match.model. Otherwise, omit spec.match to match metadata.name exactly.

For API details, see About models. For configurations that use an AgentgatewayBackend and an HTTPRoute, see Model aliasing.

Provider prefix

Match names with a provider prefix, such as openai/gpt-5-mini, and strip the prefix before forwarding the request. One entry exposes all of the provider’s models, including new releases.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: openai
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  match:
    model: "openai/*"
  provider: OpenAI
  policies:
    auth:
      secretRef:
        name: openai-secret
    transformations:
    - field: model
      expression: 'llmRequest.model.stripPrefix("openai/")'

Create one resource like this for each provider, such as anthropic/* with provider: Anthropic.

The /v1/models endpoint lists prefixed names from the model catalog, such as openai/gpt-5-mini.

To restrict access, list exact model names.

Exact model name

Match a single model name and forward it unchanged. Add an entry for each model you allow clients to use.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  # With no spec.match, clients request the model by this name
  name: gpt-5.6-luna
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  provider: OpenAI
  policies:
    auth:
      secretRef:
        name: openai-secret

Set spec.match.model for names that Kubernetes does not allow in metadata.name, such as those containing uppercase letters, /, or :.

Prefix without a provider

Match a model family, such as gpt-* or claude-*, and forward the name unchanged. Maintain one entry per family.

This pattern has the following limits.

• Each family maps to one provider. To offer the same model through Anthropic and Amazon Bedrock, for example, use a provider prefix.
• A prefix matches only the names that start with it. For example, gpt-* does not match the OpenAI models o3 or text-embedding-3-small.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: claude-models
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  match:
    model: "claude-*"
  provider: Anthropic
  policies:
    auth:
      secretRef:
        name: anthropic-secret

Catch-all

Match any model name with * and forward it unchanged. Use this pattern to expose all models from one provider. Unsupported model names fail at the provider rather than the gateway.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: all-openai-models
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  match:
    model: "*"
  provider: OpenAI
  policies:
    auth:
      secretRef:
        name: openai-secret

Custom model name

Publish a custom name, such as fast or support-bot, and map it to a provider model. Change that mapping to switch models without updating clients.

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayModel
metadata:
  name: fast
  namespace: agentgateway-system
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
    sectionName: http
  provider: OpenAI
  policies:
    auth:
      secretRef:
        name: openai-secret
    transformations:
    - field: model
      expression: '"gpt-5-mini"'

The model transformation sets the model that the provider receives.

To split traffic or fail over between models behind one custom name, use a virtual model.