Skip to main content

Collect gateway telemetry

The AI Gateway exports OpenTelemetry traces and metrics, and can serve its metrics to Prometheus. Configure both under telemetry in your platform values. Telemetry is off by default. This guide covers connecting a collector, counting token usage, and adding request and identity context to spans so you can break traffic down by project, team, or department.

Every signal carries the service name ai-gateway-v2, and every gateway metric carries the label implementation="v2". Filter on that label when the legacy AI Gateway reports into the same metric store.

For the legacy gateway, see legacy gateway telemetry.

Export traces and metrics to a collector​

Set telemetry.otlp.endpoint to your collector's OpenTelemetry Protocol (OTLP) HTTP receiver, then turn on each signal you want:

values.yaml
ai-gateway-v2:
  telemetry:
    otlp:
      enabled: true
      endpoint: 'otel-collector.observability.svc:4318'
    tracing:
      enabled: true

telemetry.otlp.enabled pushes metrics and telemetry.tracing.enabled pushes traces, both to telemetry.otlp.endpoint. The gateway exports over OTLP/HTTP, usually port 4318, and uses TLS unless you set telemetry.otlp.insecure: true for a collector on a trusted private network.

Each model request produces an ai_gateway.chat_completion span carrying attributes from the OpenTelemetry generative AI semantic conventions, such as gen_ai.request.model and gen_ai.provider.name. When the provider's response reports token usage, the span also carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus gen_ai.usage.cost when budgets are configured. The gateway sets user.id to the caller's identity.

To share one collector across the platform, set global.stacklok.telemetry.otlp.enabled and global.stacklok.telemetry.otlp.endpoint instead. The gateway falls back to the global endpoint when it has no local one.

Tracing samples every request by default. To cut span volume, set telemetry.tracing.samplingRate to the fraction of root spans to keep, between 0.0 and 1.0. The default, 0, keeps every root span. The gateway always keeps a request whose inbound traceparent is sampled, whatever the rate. A cost platform fed by these spans sees only the sampled requests, so it no longer counts all traffic.

Send traces to a separate destination​

Some cost and usage platforms, such as CloudZero, ingest these spans directly at a fixed URL. Set telemetry.otlp.tracesEndpoint to send traces there. The gateway posts to the URL exactly as written, without appending /v1/traces, and telemetry.otlp.endpoint keeps receiving metrics if you set it.

If the destination requires an API key, store the OTLP headers string under the headers key of a Secret in the platform namespace:

kubectl create secret generic cloudzero-otlp-headers \
  -n <NAMESPACE> \
  --from-literal=headers='Authorization=<API_KEY>'

Then name the Secret in telemetry.otlp.headersSecretRef:

values.yaml
ai-gateway-v2:
  telemetry:
    tracing:
      enabled: true
    otlp:
      tracesEndpoint: 'https://telemetry.cloudzero.com/v2/telemetry/<SINK>'
      headersSecretRef:
        name: cloudzero-otlp-headers

The headers require an https:// URL for tracesEndpoint. The gateway sends the headers on every OTLP signal, so the chart refuses them while OTLP metrics push is on, locally or through global.stacklok.telemetry.otlp. Scrape metrics with Prometheus instead, or relay traces through your own collector that holds the credential.

Scrape metrics with Prometheus​

Set telemetry.prometheus.enabled to serve /metrics on a dedicated port, 9464 by default (change it with telemetry.metrics.port):

values.yaml
ai-gateway-v2:
  telemetry:
    prometheus:
      enabled: true

The gateway pods carry prometheus.io/scrape annotations, so a Prometheus server that discovers targets by annotation picks them up automatically. The metrics listener is unauthenticated. When the chart's network policy is on, only pods in the namespaces under networkPolicy.ingress.monitoringNamespaces (default monitoring) can reach it.

Record token usage​

The gateway records the stacklok.ai_gateway.token.usage counter for every request whose provider reports usage, labeled with token_type (input or output), provider, and model. Output tokens include reasoning tokens for every provider. In Prometheus, the counter appears as stacklok_ai_gateway_token_usage_total. For example, this query returns tokens per second by model:

sum by (model, token_type) (rate(stacklok_ai_gateway_token_usage_total{implementation="v2"}[5m]))

To break usage down by a header the caller sends, map it under telemetry.headerLabels with the closed set of values to keep:

values.yaml
ai-gateway-v2:
  telemetry:
    headerLabels:
      - header: x-team
        attribute: team
        permitted:
          - payments
          - platform

A value outside permitted becomes other, which keeps the number of metric series bounded however the header varies. Per-user and per-request identifiers belong on spans instead.

Request rate, latency, and status codes come from the standard http.server.request.duration histogram, labeled with http.route and http.response.status_code. It covers every HTTP request, including ones refused by authentication.

Add request headers to spans​

For per-request context that has too many distinct values for a metric label, such as a project or ticket ID, telemetry.spanAttributes copies a header the caller sends onto that request's span:

values.yaml
ai-gateway-v2:
  telemetry:
    tracing:
      enabled: true
    spanAttributes:
      - header: x-cz-context-project
        attribute: cz.context.project

You can configure up to 20 entries, and tracing must be on. A request that omits the header gets no attribute, and values longer than 256 bytes are truncated. The value is whatever the client sent, so treat it as an attribution hint.

Header names are lowercase. Names the gateway already sets are reserved: anything under gen_ai. (except gen_ai.client.app), stacklok., or budget., plus the caller identity attributes such as user.id and user.email. To keep secrets and personal data out of your traces, the chart refuses credential and personal-data headers such as authorization, cookie, x-api-key, x-user-email, and x-forwarded-for.

Attribute traffic to organizational groups​

The gateway can look up each caller in the directory and attach the labels of their derived groups, such as department or cost_center, to request spans and audit records. You can then break activity down by department or cost center without changing any client. You define these labels on derived groups in the directory; see Label derived groups.

Point the gateway at the directory's gRPC address, give it a credential to present, and list the labels to resolve:

values.yaml
ai-gateway-v2:
  directory:
    endpoint: '<ENTERPRISE_MANAGER_SERVICE>.<NAMESPACE>.svc:9091'
    insecure: true
    groupsResolutionIssuer: '<IDP_ISSUER_URL>'
    tokenProjection:
      enabled: true
    groupLabels:
      - department
      - cost_center

Use the same directory connection values as in Configure the AI Gateway.

The directory must allow the gateway's ServiceAccount to call it. The platform chart adds system:serviceaccount:<NAMESPACE>:<RELEASE_NAME>-ai-gateway-v2 to enterprise-manager.grpc.callerAuth.saSubjectAllowlist by default when the AI Gateway is enabled. If you override that list, include this entry.

The directory accepts gRPC calls from AI Gateway pods in the Enterprise Manager's namespace, which is where the platform chart installs both.

For requests that use a virtual API key, the gateway resolves labels through the key's owner. directory.groupsResolutionIssuer is the issuer the directory knows those owners under. The chart refuses to render groupLabels without it.

How labels appear​

Labels appear as attributes on request spans under their own names and under subjects on audit records. They're kept off metrics because per-caller values would multiply metric series. The gateway forwards each label to its governance checks as a header, such as X-Department, and removes that header before the request reaches the provider.

When a caller belongs to two derived groups with different values for the same label, the gateway omits that label and logs a warning naming it.

The gateway looks up every request, without caching. If the directory doesn't answer, the gateway refuses the request with 503 and logs directory group resolution failed; refusing request, so it never forwards an incomplete identity.

Next steps​

  • Forward audit logs to send these records, group labels included, to your security information and event management (SIEM) system.
  • Budgets and pricing to turn attributed usage into enforced spend limits.

Troubleshooting​

No traces or metrics reach the collector

Confirm the gateway has a destination for each signal. Metrics go to telemetry.otlp.endpoint when telemetry.otlp.enabled is set. Traces go to telemetry.otlp.tracesEndpoint when set, otherwise to telemetry.otlp.endpoint, and only when telemetry.tracing.enabled is set. The gateway speaks OTLP over HTTP, so point it at the collector's HTTP receiver, usually port 4318.

With the chart's network policy on and networkPolicy.allowExternalEgress set to false, the chart opens egress on the ports named in telemetry.otlp.endpoint and telemetry.otlp.tracesEndpoint, to any destination. Confirm that each port in those values matches the port the collector listens on.

Spans carry no group labels

Confirm that tracing is on and that the label is listed in directory.groupLabels. The gateway resolves only the labels you list. If one label is missing while the others appear, check the caller's group memberships in the console for two groups with conflicting values.