Route models and control access

A route maps a requested model name to the provider that serves it. In the AI Gateway, the route table is the models list on each entry in providers. Change the provider behind a model by moving the model name to another provider, without updating clients.

For the legacy AI Gateway, see legacy gateway model routing.

Route a specific model

List each model under the provider that serves it:

values.yaml

ai-gateway-v2:
  providers:
    - kind: openai
      baseUrl: https://api.openai.com
      models: ['gpt-5', 'gpt-5-mini']
      credentialSecretRef:
        name: ai-gateway-v2-openai
        key: api-key
    - kind: anthropic
      baseUrl: https://api.anthropic.com
      models: ['claude-sonnet-5']
      credentialSecretRef:
        name: ai-gateway-v2-anthropic
        key: api-key

Write each model name exactly as the provider expects it. The gateway sends the name upstream unchanged, and the pricing catalog uses the same name to price the request. claude-sonnet-4-5 and claude-sonnet-4-5-20250929 are two different routes and two different prices.

How the gateway picks a provider

The gateway looks up the requested model within the client API the request arrived on, so a client calls each model through an API its provider speaks. See Supported providers.

Within one client API, a model name maps to exactly one provider. Listing the same model under two providers that share an API, such as anthropic and bedrock-anthropic, is a configuration error. Use distinct model names for the two, such as claude-sonnet-5 and us.anthropic.claude-sonnet-5.

There is no default route. A request for a model that no provider lists returns 404, so only the models you list are reachable.

Control which models each caller can use

Model access limits each caller to the models their directory grants cover. With model access on, the gateway refuses a request for a model the caller has no grant for with 403, before it calls any provider. The model discovery endpoints list only the models the caller can use.

Grants come from model sets, named collections of models that you manage on the Models page in the console's administration experience and grant to users or groups. The All Users set is held by every active user. It starts out granting every model, so turning model access on changes nothing until you narrow it.

Turn model access on in the chart values:

values.yaml

ai-gateway-v2:
  modelAccess:
    enabled: true

Model access uses the same receiver, token, and certificate authority as budget enforcement, and the directory connection used for virtual API keys. The chart refuses to render when one of these is missing. See Configure the AI Gateway for the platform values it depends on.

Model access fails closed for both requests and discovery calls. If the gateway can't get a decision for a request, it refuses the request. If it can't get a decision for a discovery call, it returns 503 with model access unavailable instead of the full model list.

Discover models

Clients list the models they can call through the discovery endpoints. Each endpoint returns the response shape its client family expects:

Endpoint	Response shape	Lists
GET /v1/models	OpenAI	Models served on Chat Completions and Responses
GET /anthropic/v1/models	Anthropic	Models served on Anthropic Messages
GET /v1beta/models	Gemini	Models served on the Gemini API, with their actions

A GET /v1/models request that carries an anthropic-version header returns the Anthropic shape and models.

Where the gateway knows a model's context window, it includes it in the response: context_window in the OpenAI shape, max_input_tokens in the Anthropic shape, and inputTokenLimit in the Gemini shape. The gateway ships a catalog of context windows generated from public model metadata. A model missing from the catalog is listed without a context window, so clients fall back to their own default.

Retries and timeouts

The gateway applies one fixed resilience policy to every provider:

• Retries. The gateway retries a request up to two times when the connection to the provider fails, or when the provider answers 502, 503, or 504. It retries only before any response bytes reach the client, and always against the same provider.
• Response headers. The gateway waits up to 5 minutes for the provider to start responding.
• Stalled streams. The gateway ends a stream that sends nothing for 5 minutes.
• Stream lifetime. The gateway ends any stream after lifecycle.maxStreamDuration, which defaults to 15m.

Budget admission runs once per request, before the provider call, and the request is charged once for the usage the provider reports. Retries don't add charges.

Limits worth knowing

• Each model is served by one provider. The gateway has no weighted routing across providers and no failover to a second provider.

Next steps

• Budgets and pricing to price every model you route. An unpriced model is refused.
• Screen prompts for injection and PCI/PII controls to decide what content is allowed through the gateway.