feat: export audit records and diagnostics to an OTLP collector

README.md

@@ -496,12 +496,15 @@ Gotchas specific to the image:
   has to be readable by uid 65532 — a host-side `0600` file owned by your
   account is not, so `chown 65532` it or run with `podman --userns=keep-id`.
 - **The audit stream needs a persistent volume.** With
-  `BUGWARDEN_AUDIT_CONFIG` set, the guard writes JSONL files itself — it
-  exports nothing — so the directory in `path` must be a volume writable by
-  uid 65532. Without one the records live in the container's writable layer,
-  which `--rm` throws away; with one that uid 65532 cannot write, startup
-  fails outright, and over HTTP the default fail mode is `closed_all`, so a
-  directory that becomes unwritable later stops the server serving.
+  `BUGWARDEN_AUDIT_CONFIG` pointing at a file, the guard writes JSONL
+  itself, and the file is authoritative whether or not
+  [OTLP export](#opentelemetry-export) is also on — so the directory in
+  `path` must be a volume writable by uid 65532. Without one the records
+  live in the container's writable layer, which `--rm` throws away; with
+  one that uid 65532 cannot write, startup fails outright, and over HTTP
+  the default fail mode is `closed_all`, so a directory that becomes
+  unwritable later stops the server serving. `BUGWARDEN_AUDIT_CONFIG=none`
+  with an OTLP endpoint writes no file and needs no volume.
 - **Signals.** The process handles `SIGINT` and `SIGTERM`. Over HTTP both
   cancel the transport token and let axum drain. Over stdio both end the
   process immediately (status 0): rmcp reads stdin on a blocking thread
@@ -557,7 +560,14 @@ Command-line arguments take precedence over environment variables.
 | — | `BUGWARDEN_HTTP_READ_TOKEN` | — | Bearer token granting the **read** scope over http: the read tools only. Same rules, and it must differ from the write token. Either token may be set alone |
 | `--read-only` | `MCP_READ_ONLY` | `false` | Disable all write tools. Tighten-only: ORed with the policy's `global.read_only`; cannot re-enable writes a policy forbids. As an environment variable it takes the literal `true` or `false` — `1`, `yes` and an empty value are a usage error, not a synonym |
 | `--policy <PATH>` | `BUGWARDEN_POLICY` | — | Path to the guard policy TOML. Without it, an allow-all policy applies (with private comments off and the 2 MiB attachment cap still in force) |
-| `--audit-config <PATH>` | `BUGWARDEN_AUDIT_CONFIG` | — | Path to the audit stream configuration TOML (worked example in [`examples/audit.toml`](examples/audit.toml)). Without it, no audit stream is written. Records carry W3C trace ids when the client sends a `traceparent` in the request's `_meta`, enabling correlation with client-side traces |
+| `--audit-config <PATH>` | `BUGWARDEN_AUDIT_CONFIG` | — | Path to the audit stream configuration TOML (worked example in [`examples/audit.toml`](examples/audit.toml)), or the exact value `none` to disable the file (OTLP-only when an endpoint is set). Unset with no OTLP endpoint writes no stream; an endpoint with no file decision, or `none` with no endpoint, is a startup error. Records carry W3C trace ids when the client sends a `traceparent` in the request's `_meta`, enabling correlation with client-side traces |
+| — | `OTEL_EXPORTER_OTLP_ENDPOINT` | — | Base URL of an OpenTelemetry collector, e.g. `http://127.0.0.1:4318`; bugwarden appends `/v1/logs` itself. Unset or empty means no export at all — no task, no thread, no request. See [OpenTelemetry export](#opentelemetry-export) |
+| — | `OTEL_EXPORTER_OTLP_HEADERS` | — | Headers added to every export request, `key=value` separated by commas — typically the collector's own credential. Environment only, like the bearer tokens, and never logged (I12). Values are used verbatim: percent-encoding is **not** decoded |
+| — | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | The OTLP transport. `http/protobuf` is the only value this build speaks; anything else is a startup error. Not consulted while export is off |
+| — | `OTEL_SERVICE_NAME` | `bugwarden` | `service.name` on the exported records |
+| — | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | — | Logs-specific endpoint. Overrides `OTEL_EXPORTER_OTLP_ENDPOINT` per the OTLP spec, and is used **as given** — write the whole URL including `/v1/logs`. Set alone it still turns export on |
+| — | `OTEL_EXPORTER_OTLP_LOGS_HEADERS` | — | Logs-specific headers; overrides `OTEL_EXPORTER_OTLP_HEADERS`. Same secrecy rules |
+| — | `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | — | Logs-specific protocol; overrides `OTEL_EXPORTER_OTLP_PROTOCOL`. Same single accepted value |
 | — | `RUST_LOG` | `info` | Tracing filter for the diagnostic log, which always goes to **stderr** — stdout belongs to the stdio transport. An unparsable value falls back to `info` |
 
 An empty value counts as unset for `--api-key`, `--api-key-file`,
@@ -568,7 +578,8 @@ is a startup error of its own), `MCP_ALLOWED_HOSTS=` names no host, leaving
 `Host` validation off as if it were never set, and an emptied token variable
 is an unset one — which over http means the deny-by-default refusal, not an
 open port. An empty `BUGWARDEN_POLICY` or `BUGWARDEN_AUDIT_CONFIG` is a usage
-error.
+error. `OTEL_EXPORTER_OTLP_ENDPOINT=` follows the same "cleared variable"
+reading, and there it is the off switch for the whole export.
 
 Exit status: `0` on clean shutdown, `1` on a startup or runtime failure (a
 missing or malformed http bearer token, an unreadable policy or audit
@@ -754,14 +765,21 @@ nonexistent one, filtered search results vanish without a trace, and no rule
 is ever named in a response. The audit stream is the other half of that
 bargain — the operator's own record of what was asked and what the guard
 decided. It carries exactly the facts the client must never see, which is
-why it goes only to a local file the operator controls: no MCP surface can
-read it, and it is never mixed into the diagnostic stderr stream.
-
-Auditing is off until `--audit-config` / `BUGWARDEN_AUDIT_CONFIG` names a
-configuration file; a commented example ships in
-[`examples/audit.toml`](examples/audit.toml). Over the http transport,
-starting without one logs a warning. Parsing is strict — unknown keys are a
-startup error, so a typo cannot silently disable a setting.
+why it goes only to the sinks the operator named — a local file, an OTLP
+collector, both, or neither: no MCP surface can read it, and it is never
+mixed into the diagnostic stderr stream.
+
+Auditing is off until at least one sink is configured.
+`--audit-config` / `BUGWARDEN_AUDIT_CONFIG` names a file configuration; the
+exact value `none` disables the file so a collector can be the only sink.
+An OTLP endpoint without a file decision is a startup error — say `none` or
+point at a file — and so is `none` with no endpoint. A commented file
+example ships in [`examples/audit.toml`](examples/audit.toml). Over the http
+transport, starting with no sink at all logs a warning. Parsing is strict —
+unknown keys are a startup error, so a typo cannot silently disable a
+setting. Getting the records off the host — which is what makes them
+tamper-evident — is covered under [OpenTelemetry export](#opentelemetry-export)
+below.
 
 ### Audit configuration reference
 
@@ -861,6 +879,91 @@ A reader of the file should skip empty lines and tolerate at most one
 unparsable line per outage: a failed write can leave a partial line, and the
 stream heals itself on the next successful record.
 
+### OpenTelemetry export
+
+A configured collector is a load-bearing audit sink, not a copy of the
+file. Bugwarden exports every persisted record to it, and sends its own
+diagnostics along the same connection. Off until
+`OTEL_EXPORTER_OTLP_ENDPOINT` names one — and that endpoint must be paired
+with a file decision (`--audit-config` / `BUGWARDEN_AUDIT_CONFIG` pointing
+at a file, or set to `none` for collector-only):
+
+```bash
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
+export OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer <collector token>'
+```
+
+[`examples/otel-collector.yaml`](examples/otel-collector.yaml) is a working
+collector configuration for this, with both receivers: `otlp` for the
+native export and `filelog` tailing the JSONL file. Run both — they carry
+the same records, and they answer different questions. The export is live
+and correlated with the client's traces; the tail is what puts the log
+beyond the reach of the host that wrote it, which is what makes it
+tamper-evident — at the price of re-reading what it already read if the
+collector restarts without a `file_storage` extension, so a consumer
+collapses the two copies on `session` plus `seq`.
+
+What goes over the wire:
+
+- **One log record per audit record**, tagged `bugwarden.stream=audit`. The
+  body is the audit line *verbatim* — the same bytes the file holds when
+  there is one, or the line the file would have carried on an OTLP-only
+  sink — so nothing extra is added.
+  Attributes lift the fields worth querying on out of the body:
+  `bugwarden.event`, `.seq`, `.transport`, `.session.id` and, on a tool
+  call, `.tool`, `.verdict` and `.rule`. The record's `trace_id` and
+  `span_id` are the ones the client sent in its `traceparent`, so a guard
+  decision joins the client trace that caused it. Severity follows the
+  record *kind*: `audit_gap` is an error, everything else is info — the
+  verdict is an attribute to filter on, not a severity.
+- **The server's diagnostics**, tagged `bugwarden.stream=diagnostics`,
+  under the same `RUST_LOG` filter stderr uses. Nearly the same events:
+  stderr additionally keeps whatever is emitted before the exporter starts
+  and everything from the export's own machinery — this module and the
+  HTTP stack it posts through — which is never exported, because a flush
+  that logs would otherwise be the reason for the next one.
+
+What it costs when the collector is not there: the same as a full disk.
+Delivery is probed at startup (the server refuses to start if the collector
+will not take a record) and watched while serving. A collector that is
+down, slow or refusing marks the audit sink failing and the configured
+`fail_mode` decides what happens to tool calls — `open` keeps serving and
+accounts the window with `audit_gap`; `closed_all` (the http default)
+refuses with the tool's usual failure text. The record still reaches the
+file first, when there is one, and the exporter only afterwards. Diagnostics
+stay best-effort on a separate queue: a dropped log line is counted (a
+warning at 1, 2, 4, 8 … drops) and never stops the guard. A served call's
+response is byte-identical with export on, off or failing. The stream is
+reachable through no MCP surface either way.
+
+Two things to know before pointing this anywhere:
+
+- **The exported stream is as sensitive as the file**, because it *is* the
+  file: verdicts, rule names and withheld bug ids included. Its destination
+  is a policy decision. `suppressed_ids = false` in the audit configuration
+  drops the ids and keeps the counts if the collector cannot be trusted
+  with them.
+- **`OTEL_EXPORTER_OTLP_HEADERS` is a credential**, and it is handled like
+  the bearer tokens: environment only — no command-line option exists, so
+  it never reaches `ps` — and it is never logged, never in an error, never
+  in a record. That one is absolute: nothing formats it, and reqwest's
+  byte-level connection tracing, which would dump the header, stays off.
+  The endpoint is a weaker promise and worth stating plainly: bugwarden
+  never logs it — the drop warning carries a count and one of
+  `queue_full`, `network`, `http_status` or `shutdown`, and nothing else —
+  and it never reaches the exported stream or an audit record. But at
+  `RUST_LOG=debug` the HTTP client underneath prints the collector's host
+  and port as any HTTP client does. That is left alone deliberately: it is
+  what you read when the collector will not answer.
+
+`http/protobuf` is the only transport (`OTEL_EXPORTER_OTLP_PROTOCOL`); gRPC
+is deliberately absent, which is what keeps the export on the same rustls
+stack the Bugzilla client uses and adds no dependency to the binary. The
+logs-specific `OTEL_EXPORTER_OTLP_LOGS_*` variables override their general
+counterparts, as the OTLP specification requires — so a fleet that names
+only `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` gets its logs exported rather than
+silently nothing.
+
 ## Tool reference
 
 Two rules cut across the tools that name bugs. A single call may reference at

compose.yaml

@@ -16,6 +16,14 @@ services:
       # Bugzilla key in the ApiKey header instead.
       BUGZILLA_API_KEY_FILE: /run/secrets/bugzilla-key
       # BUGWARDEN_AUDIT_CONFIG: /etc/bugwarden/audit.toml
+      # OTLP export to the sidecar below. A configured collector is
+      # load-bearing: delivery is probed at startup and an outage engages
+      # fail_mode. Pair it with BUGWARDEN_AUDIT_CONFIG (a file, or `none`
+      # for collector-only); an endpoint with no file decision is a
+      # startup error. The service name resolves on compose's network,
+      # which is why the collector must not bind loopback — see the note
+      # on its own receiver in examples/otel-collector.yaml.
+      # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
     # Deny-by-default: without a token the server refuses to start. Kept in a
     # file rather than the shell environment so the value stays out of argv.
     env_file:
@@ -41,3 +49,32 @@ services:
       - ALL
     security_opt:
       - no-new-privileges:true
+
+  # Sidecar collector, commented out because its destination is a policy
+  # decision (see examples/otel-collector.yaml): the stream carries guard
+  # verdicts, rule names and withheld bug ids, so it is at least as
+  # sensitive as the bugs it describes.
+  #
+  # Two ways records reach it, and the example config takes both: the OTLP
+  # endpoint above, and the filelog receiver tailing the shared audit
+  # volume. The tail is the tamper-evidence path — once the lines are off
+  # the host, the host cannot rewrite its own history.
+  #
+  # bugwarden writes the audit file 0600 and its directory 0700 as uid
+  # 65532, so the collector has to run as the SAME uid to read it. Do not
+  # loosen the file, and do not run the collector as root: that would swap
+  # a permission problem for a collector that can rewrite the record it
+  # exists to preserve.
+  # otel-collector:
+  #   image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:latest
+  #   user: "65532:65532"
+  #   command: ["--config=/etc/otelcol/config.yaml"]
+  #   volumes:
+  #     - ./examples/otel-collector.yaml:/etc/otelcol/config.yaml:ro
+  #     # The same directory bugwarden writes, read-only on this side.
+  #     - ./audit:/var/log/bugwarden:ro
+  #   read_only: true
+  #   cap_drop:
+  #     - ALL
+  #   security_opt:
+  #     - no-new-privileges:true

crates/bugwarden/Cargo.toml

@@ -45,8 +45,12 @@ rmcp = { version = "3.1", features = [
 axum = { version = "0.8", default-features = false, features = ["http1", "tokio"] }
 # Same crate rmcp uses to parse Host authorities (`parse_allowed_authority`).
 http = "1"
-tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "net"] }
+tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "net", "time"] }
 tokio-util = "0.7"
+# OTLP/HTTP export (issue #31) posts protobuf with the same rustls stack
+# bugwarden-core already resolves for Bugzilla, so this adds no crate to the
+# lock file and no second TLS implementation.
+reqwest = { version = "0.13", default-features = false, features = ["rustls"] }
 
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"

crates/bugwarden/completions/_bugwarden

@@ -25,7 +25,7 @@ stdio\:"Stdio transport. The API key comes from \`--api-key\` / \`BUGZILLA_API_K
 '--api-key=[Bugzilla API key. Required for --transport stdio (no HTTP headers exist there) unless --api-key-file provides it. Environment variable BUGZILLA_API_KEY can also be used. Ignored for --transport http (clients send the key per-request via the API key header; use --api-key-file for a server-held key)]:API_KEY:_default' \
 '--api-key-file=[Path to a file holding the Bugzilla API key (e.g. a container secret or systemd LoadCredential path). Mutually exclusive with --api-key. Over http this selects server-held key mode\: every request is served with this key and the per-request API key header is not consulted. An empty value counts as absent, like --api-key (so \`BUGZILLA_API_KEY_FILE=\` is an unset, not an error)]:API_KEY_FILE:_files' \
 '--policy=[Path to the guard policy TOML file. Environment variable BUGWARDEN_POLICY can also be used. Without it an allow-all default policy is used]:POLICY:_files' \
-'--audit-config=[Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. Without it no audit stream is written]:AUDIT_CONFIG:_files' \
+'--audit-config=[Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. The exact value \`none\` disables the audit file (OTLP-only when an endpoint is set). Without it, and with no OTLP endpoint, no audit stream is written. An endpoint with no file decision, or \`none\` with no endpoint, is a startup error]:AUDIT_CONFIG:_files' \
 '--use-auth-header[Use '\''Authorization\: Bearer'\'' header instead of the api_key query parameter (required for some Bugzilla instances). Environment variable BUGZILLA_USE_AUTH_HEADER=true can also be used]' \
 '--read-only[Disables all tools which modify the state of a bug. Environment variable MCP_READ_ONLY=true can also be used. Can only tighten the guard policy, never loosen it]' \
 '--insecure-no-auth[Serve the http transport without bearer authentication. Only for a trusted, isolated network\: every caller that reaches the port gets the full write scope. Command line only, with no environment variable, so no ambient value can turn authentication off. Tokens are never taken from the command line either (argv is world-readable)\: set BUGWARDEN_HTTP_TOKEN / BUGWARDEN_HTTP_READ_TOKEN in the environment]' \