For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Overview of supported LLM providers and their capabilities

Agentgateway provides seamless integration with various Large Language Model (LLM) providers. This way, you can consume AI services through a unified interface while still maintaining flexibility in the providers that you use.

What is an LLM?

Large Language Models (LLMs) are very large, deep learning models that are pre-trained on massive amounts of unlabeled and self-supervised data, often containing billions of words. LLMs extract meanings from a sequence of text and can understand the relationship between words and phrases in it. Based on this understanding, the LLM can then predict and generate natural language, and perform related tasks, such as answering questions, summarizing text, translating between languages, or writing content. LLMs can be fine-tuned on a smaller, more specific data set so that it can perform a more specific task more accurately.

While LLMs were originally designed to be trained on text and natural language, they are now becoming multimodal, supporting different media types, such as images and videos.

Common problems with LLM consumption

When adopting an LLM on an enterprise-level, you typically run into the following issues:

• Data leakage and privacy: When interacting with LLMs, it is essential to comply with federal laws and to not leak any PII (personally identifiable information) or other sensitive information in prompts. For example, users might enter personal credit card or customer related information, such as account numbers and names. Attackers might also trick the model into revealing previous user data that is still stored in the context. To prevent sensitive information from leaking to the LLM provider, prompt guards must be in place that help to filter, block, monitor, and control LLM inputs and outputs to find offensive content, prevent misuse, and ensure ethical and responsible AI usage.

• Cost controls: Compared to traditional APIs, LLM queries are long-running, meaning the LLM must parse through its large data set iteratively to compute one token at a time. All tokens must then be summarized to generate a prediction. Given the large amounts of data that the LLM parses, GPUs are required to run LLMs efficiently, which are costly to run and maintain. Limiting the amount of tokens that can be used and monitoring token consumption are critical to keep costs to a minimum.

• Fan-out patterns: LLMs are limited by static training data. This limitation can make seemingly simple questions difficult to answer. For example, answering a simple question, such as What is the weather today? requires the LLM to have access to several real-time information, such as the location, time, and weather forecast. Integrating LLMs with other MCP servers and agents to perform these tasks becomes essential to accurately answer questions of this nature.

• Security: To protect your LLM from being consumed by unauthorized users, you typically want to authenticate with the LLM before you start sending requests. Managing credentials, such as API keys, and monitoring the traffic to the LLM becomes essential to protect access to the LLM.

• Scalability and reliability: When relying on an LLM to perform specific tasks in your environment, you must put measures in place that prevent your applications from failing, being overwhelmed, or performing inefficiently. This includes retries and timeouts, request rate limiting, and multi-model failovers.

Supported providers

Agentgateway supports native, OpenAI-compatible, and self-hosted LLM providers.

Native providers

Review the following table to compare agentgateway’s support of different LLM provider APIs.

Provider	Chat Completions	Responses	Messages	Embeddings	Realtime	Count Tokens	Rerank
OpenAI	✅	✅	✅¹	✅	✅	✅²	-
Anthropic	✅¹	◇	✅	-	-	✅	-
Bedrock	✅¹	✅¹	✅¹	✅¹	-	✅⁴	✅¹
Azure	✅	✅	✅¹	✅	-	✅²	⚠️³
Gemini	✅	✅¹	✅¹	✅	-	✅²	-
Vertex AI	✅⁴	◇	✅⁴	✅¹	-	✅⁴	✅¹
Copilot	✅	✅	✅¹	◇	-	✅²	⚠️³
Cohere	✅	✅¹	✅¹	✅	-	✅²	✅
Ollama	✅	✅	✅¹	✅	-	✅²	-
Baseten	✅	✅¹	✅	-	-	✅²	-
Cerebras	✅	✅¹	✅¹	-	-	✅²	-
Deepinfra	✅	✅¹	✅	✅	-	✅²	-
Deepseek	✅	✅¹	✅	-	-	✅²	-
Groq	✅	✅	✅¹	-	-	✅²	-
Hugging Face	✅	✅	✅¹	-	-	✅²	-
Mistral	✅	✅¹	✅¹	✅	-	✅²	-
OpenRouter	✅	✅	✅	✅	-	✅²	✅
Together AI	✅	✅¹	✅¹	✅	-	✅²	✅
xAI	✅	✅	✅¹	-	✅	✅²	-
Fireworks	✅	✅	✅	✅	-	✅²	✅
Meta	✅	✅	✅	◇	◇	✅²	◇

Legend:

First-class providers

Many providers now have dedicated integrations with preconfigured base URLs and request formats:

• Baseten
• Cerebras
• DeepInfra
• xAI (Grok)
• Cohere
• Together AI
• Groq
• DeepSeek
• Mistral
• Hugging Face
• OpenRouter
• Fireworks AI
• Meta

Self-hosted solutions

Run models locally or in your own infrastructure:

• Ollama
• vLLM
• LM Studio

Custom providers

Use Custom provider for other providers without direct support such as Perplexity, vLLM, or LM Studio. Agentgateway supports all of the common LLM formats and can generally integrate with any provider (file an issue if one is missing!).

Using the API

Agentgateway exposes multiple different API endpoints, including OpenAI Chat Completions, Anthropic Messages, and more. Depending on the API used in the request, and the provider selected, agentgateway can pass the request through or translate it as needed.

This enables a unified API regardless of the provider used, allowing seamlessly connecting clients (regardless of which API they use) to any provider.

Below shows some basic examples using the Chat Completions API

curl 'http://localhost:4000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
  "model": "gpt-3.5-turbo",
  "messages": [
    {
      "role": "user",
      "content": "Tell me a story"
    }
  ]
}
'

import openai

client = openai.OpenAI(
    api_key="anything",
    base_url="http://localhost:4000/v1"
)

response = client.chat.completions.create(model="gpt-4o-mini", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
])

print(response)

import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: "anything",
  baseURL: "http://localhost:4000/v1",
});
const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "this is a test request, write a short poem" }]
});

console.log(response);

Model routing and aliases

Model routing is configured within the llm section of your agentgateway configuration file. The llm section offers a simplified, model-centric approach compared to the gateways and routes model; for more details on the two approaches, see LLM configuration modes. The model configurations shown in this section live under the llm.models key.

Agentgateway routes requests by matching an incoming model name, and then sending it to the configured model. The outgoing model can be passed through from the incoming model, be transformed, or be a static model.

Agentgateway recognizes the standard LLM serving paths, such as /v1/chat/completions, /v1/messages, /v1/responses, /v1/audio/transcriptions, and /v1/models, only when the request path matches exactly. Provider-specific paths that carry the model in the path, such as the Gemini and Vertex AI :generateContent paths, match by their pattern instead. A request to any other path, such as /other/v1/messages or /v1/messages/extra, is not rejected. Agentgateway still selects the model from the request body, but forwards the request to the provider as passthrough, without format conversion and without the params.model override.

To serve the standard LLM paths under a base path, set llm.pathPrefix. For example, pathPrefix: /tenant-a lets clients call /tenant-a/v1/chat/completions. The gateway removes /tenant-a before model routing and before forwarding the request to the provider, and returns a 404 error for requests outside the prefix. The prefix must start with /, and cannot include a query or a fragment. You can set only one prefix, and a trailing slash is ignored.

When the selected model changes the upstream model name, the gateway rewrites the model before forwarding the request. The rewrite applies to JSON request bodies and multipart form data, such as /v1/audio/transcriptions. For multipart requests, file fields and non-model fields are preserved while each model form field is rewritten.

Choose how to expose models

Clients select a model with the request’s model field. Choose a matching pattern to control which models they can request and how you name them.

Each pattern sets the name field of an llm.models entry.

For matching priority when you mix patterns, see Combine patterns.

Provider prefix

Match names with a provider prefix, such as openai/gpt-5-mini, and strip the prefix before forwarding the request. One entry exposes all of the provider’s models, including new releases.

llm:
  models:
  - name: "openai/*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      model: llmRequest.model.stripPrefix("openai/")
  - name: "anthropic/*"
    provider: anthropic
    params:
      apiKey: "$ANTHROPIC_API_KEY"
    transformation:
      model: llmRequest.model.stripPrefix("anthropic/")

The /v1/models endpoint lists prefixed names from the model catalog, such as openai/gpt-5-mini.

To restrict access, list exact model names, or set per-key patterns such as openai/gpt-5* with allowedModels.

Exact model name

Match a single model name and forward it unchanged. Add an entry for each model you allow clients to use.

llm:
  models:
  - name: gpt-5.6-luna
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
  - name: claude-sonnet-4-5
    provider: anthropic
    params:
      apiKey: "$ANTHROPIC_API_KEY"

Prefix without a provider

Match a model family, such as gpt-* or claude-*, and forward the name unchanged. Maintain one entry per family.

This pattern has the following limits.

• Each family maps to one provider. To offer the same model through Anthropic and Amazon Bedrock, for example, use a provider prefix.
• A prefix matches only the names that start with it. For example, gpt-* does not match the OpenAI models o3 or text-embedding-3-small.

llm:
  models:
  - name: "gpt-*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
  - name: "claude-*"
    provider: anthropic
    params:
      apiKey: "$ANTHROPIC_API_KEY"

Catch-all

Match any model name with * and forward it unchanged. Use this pattern to expose all models from one provider. Unsupported model names fail at the provider rather than the gateway.

The * pattern has the lowest priority, so it can serve as a fallback for unmatched names. See Combine patterns.

llm:
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"

Custom model name

Publish a custom name, such as fast or support-bot, and map it to a provider model. Change that mapping to switch models without updating clients.

llm:
  models:
  - name: fast
    provider: openAI
    params:
      model: gpt-5-mini
      apiKey: "$OPENAI_API_KEY"
  - name: support-bot
    provider: anthropic
    params:
      model: claude-haiku-4-5
      apiKey: "$ANTHROPIC_API_KEY"

The params.model field sets the model that the provider receives.

To split traffic or fail over between models behind one custom name, use a virtual model.

Combine patterns

This example exposes OpenAI and Anthropic models with provider prefixes. It also maps support-bot to a specific model and uses OpenRouter for unmatched names.

llm:
  models:
  - name: "openai/*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      model: llmRequest.model.stripPrefix("openai/")
  - name: "anthropic/*"
    provider: anthropic
    params:
      apiKey: "$ANTHROPIC_API_KEY"
    transformation:
      model: llmRequest.model.stripPrefix("anthropic/")
  - name: support-bot
    provider: anthropic
    params:
      model: claude-haiku-4-5
      apiKey: "$ANTHROPIC_API_KEY"
  - name: "*"
    provider: openrouter
    params:
      apiKey: "$OPENROUTER_API_KEY"

When patterns overlap, agentgateway selects a match in the following order.

1. A virtual model with the exact requested name.
2. The entry with the most characters other than *, regardless of list order. Exact names take priority over matching wildcards, and openai/* takes priority over *.
3. The first listed entry if matches have the same number of characters other than *. For example, list order breaks the tie between claude-* and *-latest for claude-sonnet-latest.

Agentgateway skips entries if their matches headers do not match or their authorization rules deny the request. It returns an error only if no matching entry allows the request.

A broader entry can therefore serve a request denied by a more specific entry. Apply the same authorization rules to broader matches, including the catch-all, or remove those entries.