Skip to main content

Forward audit logs to your SIEM

The AI Gateway emits structured JSON audit events to pod standard output. It writes one event for each model interaction, one for each rejected authentication attempt, and one for each model list request that model access filters down to nothing.

Events record decisions and metadata only. They never carry prompts, completions, credentials, request or response bodies, token claims, or email addresses.

Every event carries the fields required by NIST SP 800-53 controls AU-2, AU-3, and AU-8: a unique audit ID, a UTC timestamp, the outcome, the component, the source, an optional target, and an extra block for forensic correlation. Events for authenticated requests also carry the subject.

For the legacy AI Gateway, see legacy gateway audit logs.

Enable audit events​

Audit events are off by default. Turn them on for the gateway:

values.yaml
ai-gateway-v2:
  audit:
    enabled: true

Setting global.stacklok.audit.enabled: true also turns them on, and while it's true, audit.enabled: false has no effect. The change applies when the gateway pods roll out; see How configuration changes apply.

Enabled audit events cover every interaction without sampling, and there are no per-event toggles. Filter and route events in your log pipeline.

What gets captured​

Event typeFires whenOutcome
aigw.v2.interaction.completedAn authenticated generation request reaches its final outcome, including an early refusalsuccess, denied, failure, or error
aigw.v2.authentication.failedThe gateway rejects a credential on a protected route, or can't reach a service it needs to validate itfailure, or error when unreachable
aigw.v2.models.no_grantModel access filters a non-empty model list to nothing, and the caller receives a 200denied

Interaction events cover the OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini generate and stream-generate endpoints. A streaming interaction emits one event when it finishes, never one per frame. Health endpoints emit no events, and a model list request that returns at least one model emits none.

Event schema​

{
  "time": "2026-10-16T14:22:31.418Z",
  "level": "INFO",
  "msg": "audit_event",
  "logger": "audit",
  "audit_id": "b3d4f7a2-8c11-4f6e-9b0a-1c2d3e4f5a6b",
  "type": "aigw.v2.interaction.completed",
  "logged_at": "2026-10-16T14:22:31.418Z",
  "outcome": "success",
  "component": "v2",
  "source": { "type": "local", "value": "ai-gateway-v2-6d8f9c7b5-x2k4q" },
  "subjects": { "user": "00u1a2b3c4d5e6f7g8h9", "department": "engineering" },
  "target": {
    "model": "claude-sonnet-4-6",
    "provider": "anthropic",
    "route": "/v1/messages"
  },
  "metadata": {
    "extra": {
      "schema_version": 1,
      "request_id": "c0ffee01-2a3b-4c5d-6e7f-8091a2b3c4d5",
      "interaction_id": "7e1d5390-41e5-4f71-b3c2-9a0815793c4a",
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7",
      "http_method": "POST",
      "http_status": 200,
      "dialect": "anthropic-messages",
      "streaming": true,
      "duration_ms": 8421,
      "reason_code": "delivered",
      "usage_present": true,
      "input_tokens": 1834,
      "output_tokens": 412
    }
  }
}

Subjects appear only after authentication succeeds. user is the caller's canonical subject, and virtual_key_id identifies the virtual key on virtual-key requests. Directory group labels named in directory.groupLabels, such as department, are added alongside them. A rejected authentication event carries no subjects.

Targets carry model, provider, and route, each only once the gateway has resolved it. provider is the provider kind from the gateway configuration: openai, openrouter, anthropic, bedrock-anthropic, gemini-aistudio, or vertex. route is the endpoint path template.

Source is the gateway pod that wrote the event. Events carry no client IP address or user agent.

Common extras are schema_version, request_id, http_status, and, when tracing is on, trace_id and span_id for OpenTelemetry correlation. The gateway also returns request_id to the client in the X-Request-ID response header. Authentication events add auth_method: jwt, virtual_key, or unknown.

Interaction events add interaction_id, http_method, dialect, streaming, duration_ms, reason_code, and usage_present, plus input_tokens and output_tokens when the provider reported usage. reason_code says what ended the interaction, such as delivered or governance_denied. When a governance hook denies or fails a request, hook names the hook entry.

Required governance hooks can report annotations under hook_annotations, keyed by hook entry name. The budgets entries report the charged subject and amount, and the scanning entry reports the entity types it acted on, such as detection_types for a block and redact_types for a redaction.

Separate audit records from application logs​

Each event is one line of JSON on the pod's standard output. Every audit record carries "logger":"audit" and "msg":"audit_event", so shippers should filter on that pair to separate audit records from ordinary application logs on the same stream.

Ship them​

Each example below filters on those same two keys. Adapt hostnames, credentials, and <NAMESPACE> for your environment.

Fluent Bit to Splunk​

[INPUT]
    Name              tail
    Path              /var/log/containers/*<NAMESPACE>*.log
    multiline.parser  docker, cri
    Tag               kube.audit.*
    Refresh_Interval  5

[FILTER]
    Name    grep
    Match   kube.audit.*
    Regex   log "logger":"audit"
    Regex   log "msg":"audit_event"

[FILTER]
    Name          parser
    Match         kube.audit.*
    Key_Name      log
    Parser        json
    Reserve_Data  On

[OUTPUT]
    Name          splunk
    Match         kube.audit.*
    Host          <SPLUNK_HEC_HOST>
    Port          8088
    Splunk_Token  ${SPLUNK_HEC_TOKEN}
    TLS           On
    event_source  aigw-audit
    event_index   aigw_audit

A record passes the grep filter only when both Regex rules match, so the filter works whatever order the two keys appear in.

Fluent Bit to Elastic​

Keep the input and both filters from the Splunk example above, which tag records kube.audit.*, and swap the output. The Match value has to be that same tag, or the output receives nothing:

[OUTPUT]
    Name            es
    Match           kube.audit.*
    Host            <ELASTICSEARCH_HOST>
    Port            9200
    HTTP_User       ${ES_USER}
    HTTP_Passwd     ${ES_PASSWORD}
    tls             On
    Logstash_Format On
    Logstash_Prefix aigw-audit
    Replace_Dots    On

Vector to Kafka​

sources:
  kubernetes_logs:
    type: kubernetes_logs
    extra_namespace_label_selector: 'kubernetes.io/metadata.name=<NAMESPACE>'

transforms:
  parse_json:
    type: remap
    inputs: [kubernetes_logs]
    source: |
      . = parse_json!(.message)
      if .logger != "audit" || .msg != "audit_event" { abort }

sinks:
  kafka:
    type: kafka
    inputs: [parse_json]
    bootstrap_servers: '<KAFKA_BROKERS>'
    topic: aigw-audit
    encoding:
      codec: json
    compression: snappy

Promtail to Loki​

Promtail's match stage selects on labels, so this pipeline promotes logger and msg to labels, drops every line that isn't an audit record, then removes those two labels.

scrape_configs:
  - job_name: aigw-audit
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        regex: <NAMESPACE>
        action: keep
      - source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        separator: /
        target_label: __path__
        replacement: /var/log/pods/*$1/*.log
    pipeline_stages:
      - cri: {}
      - json:
          expressions:
            logger: logger
            msg: msg
            type: type
            outcome: outcome
            component: component
      - labels:
          logger:
          msg:
      - match:
          selector: '{logger!="audit"}'
          action: drop
      - match:
          selector: '{msg!="audit_event"}'
          action: drop
      - labeldrop:
          - logger
          - msg
      - labels:
          type:
          outcome:
          component:

On nodes that use the Docker runtime, replace cri: {} with docker: {}. Promtail has reached end of life. For new installs, use Grafana Alloy, which converts this configuration with alloy convert --source-format=promtail.

CloudWatch and rsyslog​

The CloudWatch agent cannot filter on a JSON key natively. Either ingest everything and apply a log-group metric filter:

{ $.logger = "audit" && $.msg = "audit_event" }

or run Fluent Bit alongside it with the cloudwatch_logs output, using the filters above so only audit lines reach the dedicated log group.

For rsyslog, match both markers before forwarding:

if $msg contains '"logger":"audit"' and $msg contains '"msg":"audit_event"' then {
    action(type="omfwd"
           target="<SIEM_HOST>"
           port="514"
           protocol="tcp"
           template="RSYSLOG_SyslogProtocol23Format")
    stop
}

Retention, tamper evidence, and GDPR​

Configure retention, tamper evidence, and privacy controls in your ingest and storage systems. Use lifecycle policies and write-once storage where your compliance requirements call for them.

Apply field-level redaction before events reach long-term storage. The gateway does not retain or replay emitted audit events.

The Kubernetes API audit log covers control-plane activity such as kubectl calls and Helm upgrades. Configure it through the API server audit policy.

Conversation content reaches no audit event. Screening sends prompt text to Presidio and AWS Bedrock for analysis, and two settings persist prompt content:

  • Conversation capture to the budgets service, configured with budgets.capture. See Connect the gateway to budgets.
  • Snapshot capture, configured with the top-level capture value, which records full request and response bodies for testing. It's unset by default; leave it unset in production.

Next steps​