From 1781ee65f040eb686db0da4af54554f07ea75579 Mon Sep 17 00:00:00 2001 From: Martin Pluskal Date: Tue, 18 Aug 2026 21:05:05 +0200 Subject: [PATCH] feat: export audit records and diagnostics to an OTLP collector MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A configured collector is a load-bearing audit sink: delivery is probed at startup, a failed export engages the same FailMode gate a failed file write does, and the file can be turned off only with the exact `none` sentinel. Diagnostics stay best-effort on a separate queue. The two ambiguous spellings (endpoint with no file decision, `none` with no endpoint) refuse to start. Configuration is the standard OTel environment surface, off unless OTEL_EXPORTER_OTLP_ENDPOINT (or the logs-specific variant) is set; OTEL_EXPORTER_OTLP_HEADERS is credential material under I12. The encoder is hand-written against opentelemetry-proto logs.proto — zero new crates, Cargo.lock unchanged. Closes #31 --- README.md | 135 +- compose.yaml | 37 + crates/bugwarden/Cargo.toml | 6 +- crates/bugwarden/completions/_bugwarden | 2 +- crates/bugwarden/completions/bugwarden.fish | 2 +- crates/bugwarden/man/bugwarden.1 | 38 +- crates/bugwarden/src/audit.rs | 737 +++++- crates/bugwarden/src/bin/bugwarden-gen.rs | 39 +- crates/bugwarden/src/config.rs | 7 +- crates/bugwarden/src/lib.rs | 3 + crates/bugwarden/src/main.rs | 299 ++- crates/bugwarden/src/otel.rs | 2228 +++++++++++++++++ crates/bugwarden/src/server.rs | 2 +- crates/bugwarden/tests/audit_wiremock.rs | 2 +- crates/bugwarden/tests/binary_shutdown.rs | 16 + crates/bugwarden/tests/binary_user_agent.rs | 32 +- crates/bugwarden/tests/env_config.rs | 139 +- crates/bugwarden/tests/http_auth_wiremock.rs | 16 + .../tests/http_transport_wiremock.rs | 4 +- crates/bugwarden/tests/otel_diagnostics.rs | 342 +++ crates/bugwarden/tests/otel_wiremock.rs | 770 ++++++ docs/DESIGN.md | 325 ++- examples/audit.toml | 12 +- examples/otel-collector.yaml | 257 ++ 24 files changed, 5212 insertions(+), 238 deletions(-) create mode 100644 crates/bugwarden/src/otel.rs create mode 100644 crates/bugwarden/tests/otel_diagnostics.rs create mode 100644 crates/bugwarden/tests/otel_wiremock.rs create mode 100644 examples/otel-collector.yaml diff --git a/README.md b/README.md index 9106081..f9aad80 100644 --- a/README.md +++ b/README.md @@ -496,12 +496,15 @@ Gotchas specific to the image: has to be readable by uid 65532 — a host-side `0600` file owned by your account is not, so `chown 65532` it or run with `podman --userns=keep-id`. - **The audit stream needs a persistent volume.** With - `BUGWARDEN_AUDIT_CONFIG` set, the guard writes JSONL files itself — it - exports nothing — so the directory in `path` must be a volume writable by - uid 65532. Without one the records live in the container's writable layer, - which `--rm` throws away; with one that uid 65532 cannot write, startup - fails outright, and over HTTP the default fail mode is `closed_all`, so a - directory that becomes unwritable later stops the server serving. + `BUGWARDEN_AUDIT_CONFIG` pointing at a file, the guard writes JSONL + itself, and the file is authoritative whether or not + [OTLP export](#opentelemetry-export) is also on — so the directory in + `path` must be a volume writable by uid 65532. Without one the records + live in the container's writable layer, which `--rm` throws away; with + one that uid 65532 cannot write, startup fails outright, and over HTTP + the default fail mode is `closed_all`, so a directory that becomes + unwritable later stops the server serving. `BUGWARDEN_AUDIT_CONFIG=none` + with an OTLP endpoint writes no file and needs no volume. - **Signals.** The process handles `SIGINT` and `SIGTERM`. Over HTTP both cancel the transport token and let axum drain. Over stdio both end the process immediately (status 0): rmcp reads stdin on a blocking thread @@ -557,7 +560,14 @@ Command-line arguments take precedence over environment variables. | — | `BUGWARDEN_HTTP_READ_TOKEN` | — | Bearer token granting the **read** scope over http: the read tools only. Same rules, and it must differ from the write token. Either token may be set alone | | `--read-only` | `MCP_READ_ONLY` | `false` | Disable all write tools. Tighten-only: ORed with the policy's `global.read_only`; cannot re-enable writes a policy forbids. As an environment variable it takes the literal `true` or `false` — `1`, `yes` and an empty value are a usage error, not a synonym | | `--policy ` | `BUGWARDEN_POLICY` | — | Path to the guard policy TOML. Without it, an allow-all policy applies (with private comments off and the 2 MiB attachment cap still in force) | -| `--audit-config ` | `BUGWARDEN_AUDIT_CONFIG` | — | Path to the audit stream configuration TOML (worked example in [`examples/audit.toml`](examples/audit.toml)). Without it, no audit stream is written. Records carry W3C trace ids when the client sends a `traceparent` in the request's `_meta`, enabling correlation with client-side traces | +| `--audit-config ` | `BUGWARDEN_AUDIT_CONFIG` | — | Path to the audit stream configuration TOML (worked example in [`examples/audit.toml`](examples/audit.toml)), or the exact value `none` to disable the file (OTLP-only when an endpoint is set). Unset with no OTLP endpoint writes no stream; an endpoint with no file decision, or `none` with no endpoint, is a startup error. Records carry W3C trace ids when the client sends a `traceparent` in the request's `_meta`, enabling correlation with client-side traces | +| — | `OTEL_EXPORTER_OTLP_ENDPOINT` | — | Base URL of an OpenTelemetry collector, e.g. `http://127.0.0.1:4318`; bugwarden appends `/v1/logs` itself. Unset or empty means no export at all — no task, no thread, no request. See [OpenTelemetry export](#opentelemetry-export) | +| — | `OTEL_EXPORTER_OTLP_HEADERS` | — | Headers added to every export request, `key=value` separated by commas — typically the collector's own credential. Environment only, like the bearer tokens, and never logged (I12). Values are used verbatim: percent-encoding is **not** decoded | +| — | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | The OTLP transport. `http/protobuf` is the only value this build speaks; anything else is a startup error. Not consulted while export is off | +| — | `OTEL_SERVICE_NAME` | `bugwarden` | `service.name` on the exported records | +| — | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | — | Logs-specific endpoint. Overrides `OTEL_EXPORTER_OTLP_ENDPOINT` per the OTLP spec, and is used **as given** — write the whole URL including `/v1/logs`. Set alone it still turns export on | +| — | `OTEL_EXPORTER_OTLP_LOGS_HEADERS` | — | Logs-specific headers; overrides `OTEL_EXPORTER_OTLP_HEADERS`. Same secrecy rules | +| — | `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | — | Logs-specific protocol; overrides `OTEL_EXPORTER_OTLP_PROTOCOL`. Same single accepted value | | — | `RUST_LOG` | `info` | Tracing filter for the diagnostic log, which always goes to **stderr** — stdout belongs to the stdio transport. An unparsable value falls back to `info` | An empty value counts as unset for `--api-key`, `--api-key-file`, @@ -568,7 +578,8 @@ is a startup error of its own), `MCP_ALLOWED_HOSTS=` names no host, leaving `Host` validation off as if it were never set, and an emptied token variable is an unset one — which over http means the deny-by-default refusal, not an open port. An empty `BUGWARDEN_POLICY` or `BUGWARDEN_AUDIT_CONFIG` is a usage -error. +error. `OTEL_EXPORTER_OTLP_ENDPOINT=` follows the same "cleared variable" +reading, and there it is the off switch for the whole export. Exit status: `0` on clean shutdown, `1` on a startup or runtime failure (a missing or malformed http bearer token, an unreadable policy or audit @@ -754,14 +765,21 @@ nonexistent one, filtered search results vanish without a trace, and no rule is ever named in a response. The audit stream is the other half of that bargain — the operator's own record of what was asked and what the guard decided. It carries exactly the facts the client must never see, which is -why it goes only to a local file the operator controls: no MCP surface can -read it, and it is never mixed into the diagnostic stderr stream. - -Auditing is off until `--audit-config` / `BUGWARDEN_AUDIT_CONFIG` names a -configuration file; a commented example ships in -[`examples/audit.toml`](examples/audit.toml). Over the http transport, -starting without one logs a warning. Parsing is strict — unknown keys are a -startup error, so a typo cannot silently disable a setting. +why it goes only to the sinks the operator named — a local file, an OTLP +collector, both, or neither: no MCP surface can read it, and it is never +mixed into the diagnostic stderr stream. + +Auditing is off until at least one sink is configured. +`--audit-config` / `BUGWARDEN_AUDIT_CONFIG` names a file configuration; the +exact value `none` disables the file so a collector can be the only sink. +An OTLP endpoint without a file decision is a startup error — say `none` or +point at a file — and so is `none` with no endpoint. A commented file +example ships in [`examples/audit.toml`](examples/audit.toml). Over the http +transport, starting with no sink at all logs a warning. Parsing is strict — +unknown keys are a startup error, so a typo cannot silently disable a +setting. Getting the records off the host — which is what makes them +tamper-evident — is covered under [OpenTelemetry export](#opentelemetry-export) +below. ### Audit configuration reference @@ -861,6 +879,91 @@ A reader of the file should skip empty lines and tolerate at most one unparsable line per outage: a failed write can leave a partial line, and the stream heals itself on the next successful record. +### OpenTelemetry export + +A configured collector is a load-bearing audit sink, not a copy of the +file. Bugwarden exports every persisted record to it, and sends its own +diagnostics along the same connection. Off until +`OTEL_EXPORTER_OTLP_ENDPOINT` names one — and that endpoint must be paired +with a file decision (`--audit-config` / `BUGWARDEN_AUDIT_CONFIG` pointing +at a file, or set to `none` for collector-only): + +```bash +export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 +export OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer ' +``` + +[`examples/otel-collector.yaml`](examples/otel-collector.yaml) is a working +collector configuration for this, with both receivers: `otlp` for the +native export and `filelog` tailing the JSONL file. Run both — they carry +the same records, and they answer different questions. The export is live +and correlated with the client's traces; the tail is what puts the log +beyond the reach of the host that wrote it, which is what makes it +tamper-evident — at the price of re-reading what it already read if the +collector restarts without a `file_storage` extension, so a consumer +collapses the two copies on `session` plus `seq`. + +What goes over the wire: + +- **One log record per audit record**, tagged `bugwarden.stream=audit`. The + body is the audit line *verbatim* — the same bytes the file holds when + there is one, or the line the file would have carried on an OTLP-only + sink — so nothing extra is added. + Attributes lift the fields worth querying on out of the body: + `bugwarden.event`, `.seq`, `.transport`, `.session.id` and, on a tool + call, `.tool`, `.verdict` and `.rule`. The record's `trace_id` and + `span_id` are the ones the client sent in its `traceparent`, so a guard + decision joins the client trace that caused it. Severity follows the + record *kind*: `audit_gap` is an error, everything else is info — the + verdict is an attribute to filter on, not a severity. +- **The server's diagnostics**, tagged `bugwarden.stream=diagnostics`, + under the same `RUST_LOG` filter stderr uses. Nearly the same events: + stderr additionally keeps whatever is emitted before the exporter starts + and everything from the export's own machinery — this module and the + HTTP stack it posts through — which is never exported, because a flush + that logs would otherwise be the reason for the next one. + +What it costs when the collector is not there: the same as a full disk. +Delivery is probed at startup (the server refuses to start if the collector +will not take a record) and watched while serving. A collector that is +down, slow or refusing marks the audit sink failing and the configured +`fail_mode` decides what happens to tool calls — `open` keeps serving and +accounts the window with `audit_gap`; `closed_all` (the http default) +refuses with the tool's usual failure text. The record still reaches the +file first, when there is one, and the exporter only afterwards. Diagnostics +stay best-effort on a separate queue: a dropped log line is counted (a +warning at 1, 2, 4, 8 … drops) and never stops the guard. A served call's +response is byte-identical with export on, off or failing. The stream is +reachable through no MCP surface either way. + +Two things to know before pointing this anywhere: + +- **The exported stream is as sensitive as the file**, because it *is* the + file: verdicts, rule names and withheld bug ids included. Its destination + is a policy decision. `suppressed_ids = false` in the audit configuration + drops the ids and keeps the counts if the collector cannot be trusted + with them. +- **`OTEL_EXPORTER_OTLP_HEADERS` is a credential**, and it is handled like + the bearer tokens: environment only — no command-line option exists, so + it never reaches `ps` — and it is never logged, never in an error, never + in a record. That one is absolute: nothing formats it, and reqwest's + byte-level connection tracing, which would dump the header, stays off. + The endpoint is a weaker promise and worth stating plainly: bugwarden + never logs it — the drop warning carries a count and one of + `queue_full`, `network`, `http_status` or `shutdown`, and nothing else — + and it never reaches the exported stream or an audit record. But at + `RUST_LOG=debug` the HTTP client underneath prints the collector's host + and port as any HTTP client does. That is left alone deliberately: it is + what you read when the collector will not answer. + +`http/protobuf` is the only transport (`OTEL_EXPORTER_OTLP_PROTOCOL`); gRPC +is deliberately absent, which is what keeps the export on the same rustls +stack the Bugzilla client uses and adds no dependency to the binary. The +logs-specific `OTEL_EXPORTER_OTLP_LOGS_*` variables override their general +counterparts, as the OTLP specification requires — so a fleet that names +only `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` gets its logs exported rather than +silently nothing. + ## Tool reference Two rules cut across the tools that name bugs. A single call may reference at diff --git a/compose.yaml b/compose.yaml index 13afb93..1cdaf82 100644 --- a/compose.yaml +++ b/compose.yaml @@ -16,6 +16,14 @@ services: # Bugzilla key in the ApiKey header instead. BUGZILLA_API_KEY_FILE: /run/secrets/bugzilla-key # BUGWARDEN_AUDIT_CONFIG: /etc/bugwarden/audit.toml + # OTLP export to the sidecar below. A configured collector is + # load-bearing: delivery is probed at startup and an outage engages + # fail_mode. Pair it with BUGWARDEN_AUDIT_CONFIG (a file, or `none` + # for collector-only); an endpoint with no file decision is a + # startup error. The service name resolves on compose's network, + # which is why the collector must not bind loopback — see the note + # on its own receiver in examples/otel-collector.yaml. + # OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318 # Deny-by-default: without a token the server refuses to start. Kept in a # file rather than the shell environment so the value stays out of argv. env_file: @@ -41,3 +49,32 @@ services: - ALL security_opt: - no-new-privileges:true + + # Sidecar collector, commented out because its destination is a policy + # decision (see examples/otel-collector.yaml): the stream carries guard + # verdicts, rule names and withheld bug ids, so it is at least as + # sensitive as the bugs it describes. + # + # Two ways records reach it, and the example config takes both: the OTLP + # endpoint above, and the filelog receiver tailing the shared audit + # volume. The tail is the tamper-evidence path — once the lines are off + # the host, the host cannot rewrite its own history. + # + # bugwarden writes the audit file 0600 and its directory 0700 as uid + # 65532, so the collector has to run as the SAME uid to read it. Do not + # loosen the file, and do not run the collector as root: that would swap + # a permission problem for a collector that can rewrite the record it + # exists to preserve. + # otel-collector: + # image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:latest + # user: "65532:65532" + # command: ["--config=/etc/otelcol/config.yaml"] + # volumes: + # - ./examples/otel-collector.yaml:/etc/otelcol/config.yaml:ro + # # The same directory bugwarden writes, read-only on this side. + # - ./audit:/var/log/bugwarden:ro + # read_only: true + # cap_drop: + # - ALL + # security_opt: + # - no-new-privileges:true diff --git a/crates/bugwarden/Cargo.toml b/crates/bugwarden/Cargo.toml index d187f06..5fb6207 100644 --- a/crates/bugwarden/Cargo.toml +++ b/crates/bugwarden/Cargo.toml @@ -45,8 +45,12 @@ rmcp = { version = "3.1", features = [ axum = { version = "0.8", default-features = false, features = ["http1", "tokio"] } # Same crate rmcp uses to parse Host authorities (`parse_allowed_authority`). http = "1" -tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "net"] } +tokio = { version = "1", features = ["macros", "rt-multi-thread", "signal", "net", "time"] } tokio-util = "0.7" +# OTLP/HTTP export (issue #31) posts protobuf with the same rustls stack +# bugwarden-core already resolves for Bugzilla, so this adds no crate to the +# lock file and no second TLS implementation. +reqwest = { version = "0.13", default-features = false, features = ["rustls"] } serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/crates/bugwarden/completions/_bugwarden b/crates/bugwarden/completions/_bugwarden index 1c77d11..2950a93 100644 --- a/crates/bugwarden/completions/_bugwarden +++ b/crates/bugwarden/completions/_bugwarden @@ -25,7 +25,7 @@ stdio\:"Stdio transport. The API key comes from \`--api-key\` / \`BUGZILLA_API_K '--api-key=[Bugzilla API key. Required for --transport stdio (no HTTP headers exist there) unless --api-key-file provides it. Environment variable BUGZILLA_API_KEY can also be used. Ignored for --transport http (clients send the key per-request via the API key header; use --api-key-file for a server-held key)]:API_KEY:_default' \ '--api-key-file=[Path to a file holding the Bugzilla API key (e.g. a container secret or systemd LoadCredential path). Mutually exclusive with --api-key. Over http this selects server-held key mode\: every request is served with this key and the per-request API key header is not consulted. An empty value counts as absent, like --api-key (so \`BUGZILLA_API_KEY_FILE=\` is an unset, not an error)]:API_KEY_FILE:_files' \ '--policy=[Path to the guard policy TOML file. Environment variable BUGWARDEN_POLICY can also be used. Without it an allow-all default policy is used]:POLICY:_files' \ -'--audit-config=[Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. Without it no audit stream is written]:AUDIT_CONFIG:_files' \ +'--audit-config=[Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. The exact value \`none\` disables the audit file (OTLP-only when an endpoint is set). Without it, and with no OTLP endpoint, no audit stream is written. An endpoint with no file decision, or \`none\` with no endpoint, is a startup error]:AUDIT_CONFIG:_files' \ '--use-auth-header[Use '\''Authorization\: Bearer'\'' header instead of the api_key query parameter (required for some Bugzilla instances). Environment variable BUGZILLA_USE_AUTH_HEADER=true can also be used]' \ '--read-only[Disables all tools which modify the state of a bug. Environment variable MCP_READ_ONLY=true can also be used. Can only tighten the guard policy, never loosen it]' \ '--insecure-no-auth[Serve the http transport without bearer authentication. Only for a trusted, isolated network\: every caller that reaches the port gets the full write scope. Command line only, with no environment variable, so no ambient value can turn authentication off. Tokens are never taken from the command line either (argv is world-readable)\: set BUGWARDEN_HTTP_TOKEN / BUGWARDEN_HTTP_READ_TOKEN in the environment]' \ diff --git a/crates/bugwarden/completions/bugwarden.fish b/crates/bugwarden/completions/bugwarden.fish index 49c5f9b..d19d8cd 100644 --- a/crates/bugwarden/completions/bugwarden.fish +++ b/crates/bugwarden/completions/bugwarden.fish @@ -8,7 +8,7 @@ complete -c bugwarden -l api-key-header -d 'HTTP header for clients to send the complete -c bugwarden -l api-key -d 'Bugzilla API key. Required for --transport stdio (no HTTP headers exist there) unless --api-key-file provides it. Environment variable BUGZILLA_API_KEY can also be used. Ignored for --transport http (clients send the key per-request via the API key header; use --api-key-file for a server-held key)' -r complete -c bugwarden -l api-key-file -d 'Path to a file holding the Bugzilla API key (e.g. a container secret or systemd LoadCredential path). Mutually exclusive with --api-key. Over http this selects server-held key mode: every request is served with this key and the per-request API key header is not consulted. An empty value counts as absent, like --api-key (so `BUGZILLA_API_KEY_FILE=` is an unset, not an error)' -r -F complete -c bugwarden -l policy -d 'Path to the guard policy TOML file. Environment variable BUGWARDEN_POLICY can also be used. Without it an allow-all default policy is used' -r -F -complete -c bugwarden -l audit-config -d 'Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. Without it no audit stream is written' -r -F +complete -c bugwarden -l audit-config -d 'Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. The exact value `none` disables the audit file (OTLP-only when an endpoint is set). Without it, and with no OTLP endpoint, no audit stream is written. An endpoint with no file decision, or `none` with no endpoint, is a startup error' -r -F complete -c bugwarden -l use-auth-header -d 'Use \'Authorization: Bearer\' header instead of the api_key query parameter (required for some Bugzilla instances). Environment variable BUGZILLA_USE_AUTH_HEADER=true can also be used' complete -c bugwarden -l read-only -d 'Disables all tools which modify the state of a bug. Environment variable MCP_READ_ONLY=true can also be used. Can only tighten the guard policy, never loosen it' complete -c bugwarden -l insecure-no-auth -d 'Serve the http transport without bearer authentication. Only for a trusted, isolated network: every caller that reaches the port gets the full write scope. Command line only, with no environment variable, so no ambient value can turn authentication off. Tokens are never taken from the command line either (argv is world-readable): set BUGWARDEN_HTTP_TOKEN / BUGWARDEN_HTTP_READ_TOKEN in the environment' diff --git a/crates/bugwarden/man/bugwarden.1 b/crates/bugwarden/man/bugwarden.1 index 63232d0..7971647 100644 --- a/crates/bugwarden/man/bugwarden.1 +++ b/crates/bugwarden/man/bugwarden.1 @@ -72,7 +72,7 @@ Disables all tools which modify the state of a bug. Environment variable MCP_REA Path to the guard policy TOML file. Environment variable BUGWARDEN_POLICY can also be used. Without it an allow\-all default policy is used .TP \fB\-\-audit\-config\fR \fI\fR -Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. Without it no audit stream is written +Path to the audit configuration TOML file (see examples/audit.toml). Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. The exact value `none` disables the audit file (OTLP\-only when an endpoint is set). Without it, and with no OTLP endpoint, no audit stream is written. An endpoint with no file decision, or `none` with no endpoint, is a startup error .TP \fB\-\-insecure\-no\-auth\fR Serve the http transport without bearer authentication. Only for a trusted, isolated network: every caller that reaches the port gets the full write scope. Command line only, with no environment variable, so no ambient value can turn authentication off. Tokens are never taken from the command line either (argv is world\-readable): set BUGWARDEN_HTTP_TOKEN / BUGWARDEN_HTTP_READ_TOKEN in the environment @@ -90,7 +90,8 @@ Clean shutdown. .B 1 Startup or runtime failure: a missing or malformed http bearer token, an unreadable policy or audit configuration, a key misconfiguration, an -unparsable \-\-allowed\-hosts list, a Bugzilla client or transport error. +unparsable \-\-allowed\-hosts list, an OTLP collector that will not take +records, a Bugzilla client or transport error. .TP .B 2 Command\-line usage error. @@ -151,6 +152,35 @@ Either token may be set alone; over http, setting neither is a startup error unless .B \-\-insecure\-no\-auth is given. +.TP +.B OTEL_EXPORTER_OTLP_ENDPOINT +Base URL of an OpenTelemetry collector (bugwarden appends /v1/logs). +Environment only. Unset or empty turns export off. Must be paired with +.B \-\-audit\-config +/ BUGWARDEN_AUDIT_CONFIG (a file, or the exact value +.BR none ). +An endpoint with no file decision is a startup error. +.TP +.B OTEL_EXPORTER_OTLP_HEADERS +Headers added to every export request, key=value separated by commas. +Credential material (I12): environment only, never logged. +.TP +.B OTEL_EXPORTER_OTLP_PROTOCOL +The one accepted value is http/protobuf; anything else is a startup +error. Not consulted while export is off. +.TP +.B OTEL_SERVICE_NAME +service.name on exported records. Defaults to bugwarden. +.TP +.B OTEL_EXPORTER_OTLP_LOGS_ENDPOINT +Logs-specific endpoint; overrides OTEL_EXPORTER_OTLP_ENDPOINT and is +used as given (write the whole URL including /v1/logs). +.TP +.B OTEL_EXPORTER_OTLP_LOGS_HEADERS +Logs-specific headers; overrides OTEL_EXPORTER_OTLP_HEADERS. Same secrecy. +.TP +.B OTEL_EXPORTER_OTLP_LOGS_PROTOCOL +Logs-specific protocol; overrides OTEL_EXPORTER_OTLP_PROTOCOL. .SH FILES .TP .I /etc/bugwarden/policy.toml @@ -170,7 +200,9 @@ Without .B \-\-audit\-config or .B BUGWARDEN_AUDIT_CONFIG -no audit stream is written. +and with no OTLP endpoint, no audit stream is written. The exact value +.B none +disables the file so a collector can be the only sink. .SH EXAMPLES Serve a local MCP client over stdio, the server reading the Bugzilla API key from a file: diff --git a/crates/bugwarden/src/audit.rs b/crates/bugwarden/src/audit.rs index a4dbb71..fd3bad8 100644 --- a/crates/bugwarden/src/audit.rs +++ b/crates/bugwarden/src/audit.rs @@ -5,11 +5,22 @@ //! never named. The audit stream is the other half of that bargain — the //! operator's own record of what was asked and what the guard decided. Its //! records carry exactly the facts the client must never see (guard -//! verdicts, matched rule names, suppressed bug ids), so the stream is -//! written only to an operator-controlled local file: it is never exposed -//! through any MCP surface, and it is never mixed into the diagnostic -//! stderr stream. Diagnostics about the sink itself (a full disk, a failed -//! rotation) go to `tracing` as usual, but carry no event content. +//! verdicts, matched rule names, suppressed bug ids), so the stream goes +//! only to the sinks the operator named: it is never exposed through any +//! MCP surface, and it is never mixed into the diagnostic stderr stream. +//! Diagnostics about the sink itself (a full disk, a failed rotation) go +//! to `tracing` as usual, but carry no event content. +//! +//! A deployment CHOOSES its sinks (revised 2026-08-18, issue #31; +//! [`select_sinks`]): the JSONL file, an OTLP collector ([`AuditExport`], +//! `crate::otel`), both, or none at all. Every configured sink is +//! load-bearing — a failed export puts the sink into failure exactly as a +//! failed write does, and the operator's [`FailMode`] decides what the +//! server then does about tool calls. With both configured the record +//! reaches the file first and the exporter afterwards, so the file never +//! lacks a record the collector has. With neither configured the server +//! serves with no audit trail and no audit gate — a deployment choice, +//! made only by leaving BOTH knobs unset. //! //! # File format //! @@ -140,10 +151,13 @@ fn default_suppressed_ids() -> bool { #[derive(Debug, Clone, PartialEq, Eq, Deserialize)] #[serde(deny_unknown_fields)] pub struct AuditConfig { - /// The JSONL audit file. Required. The parent directory is created - /// (mode 0700) if missing; the file itself is created mode 0600 and - /// only ever appended to. - pub path: PathBuf, + /// The JSONL audit file. Still REQUIRED in a configuration file — + /// this is `Option` only so a deployment that has no configuration + /// file can construct a fileless sink ([`AuditConfig::fileless`]), + /// and a TOML document that omits `path` is a load error as before. + /// The parent directory is created (mode 0700) if missing; the file + /// itself is created mode 0600 and only ever appended to. + pub path: Option, /// `sync_data()` after every record. The default `false` still /// survives a killed process — `write(2)` puts the record in the page /// cache before [`AuditSink::record`] returns — but only `true` @@ -178,6 +192,25 @@ pub struct AuditConfig { } impl AuditConfig { + /// The configuration of a deployment that keeps no audit file, for + /// `BUGWARDEN_AUDIT_CONFIG=none`. + /// + /// There is no document to carry the other knobs, so they take their + /// defaults and [`AuditConfig::fail_mode`] is left for the caller to + /// resolve. Such a deployment MUST have another sink; `main` enforces + /// that before it gets here. + #[must_use] + pub fn fileless() -> AuditConfig { + AuditConfig { + path: None, + fsync: false, + fail_mode: None, + rotate_max_bytes: default_rotate_max_bytes(), + rotate_keep: default_rotate_keep(), + suppressed_ids: default_suppressed_ids(), + } + } + /// Strict parse + validation of an audit configuration document. /// /// Rejects unknown keys everywhere (`deny_unknown_fields`), then @@ -187,6 +220,12 @@ impl AuditConfig { pub fn from_toml_str(s: &str) -> anyhow::Result { let cfg: AuditConfig = toml::from_str(s).context("failed to parse audit configuration TOML")?; + // `path` is `Option` only for the fileless constructor; a DOCUMENT + // without one is a load error as it always was — the audit file is + // turned off by BUGWARDEN_AUDIT_CONFIG=none, never by omission. + if cfg.path.is_none() { + anyhow::bail!("audit configuration names no `path`"); + } cfg.validate()?; Ok(cfg) } @@ -222,6 +261,80 @@ impl AuditConfig { } } +// --------------------------------------------------------------------------- +// Sink selection +// --------------------------------------------------------------------------- + +/// The `--audit-config` / `BUGWARDEN_AUDIT_CONFIG` value that disables +/// the audit FILE by name. Compared by exact bytes, like a rule name: a +/// real file that happens to be called `none` is reachable as `./none`. +pub const AUDIT_CONFIG_NONE: &str = "none"; + +/// Which audit sinks this deployment runs (issue #31, revised 2026-08-18). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SinkSelection { + /// No sink at all: the server serves normally with NO audit trail and + /// no audit gate. Expressible only by leaving both knobs unset. + NoAudit, + /// The JSONL file alone — every pre-#31 deployment, unchanged. + FileOnly, + /// File and OTLP: the write precedes the export, and either sink + /// failing gates serving under the one [`FailMode`]. + Both, + /// OTLP alone, on a fileless sink (`BUGWARDEN_AUDIT_CONFIG=none`). + OtlpOnly, +} + +/// Resolve the two knobs — the audit-config path and whether an OTLP +/// endpoint is configured — into the deployment's sink selection. +/// +/// The rule that shapes the matrix: turning the FILE off is an explicit +/// act (`BUGWARDEN_AUDIT_CONFIG=none`), never an inference from absence. +/// A typo that unsets one variable must fail the start, not silently +/// reshape the audit trail — so the two cells where absence and an +/// explicit choice could be confused are startup errors: +/// +/// - An OTLP endpoint with NO audit-config value could mean "OTLP-only" +/// or "I mistyped the file configuration"; the server refuses to guess. +/// - `none` with no OTLP endpoint would leave zero sinks on an explicit +/// request for one; running without an audit trail is legitimate but is +/// asked for by leaving both knobs unset, not by this. +/// +/// # Errors +/// +/// The two ambiguous cells above, each naming the variables and the two +/// ways to resolve it. +pub fn select_sinks( + audit_config: Option<&Path>, + otlp_configured: bool, +) -> anyhow::Result { + let file = match audit_config { + None => None, + Some(path) if path.as_os_str() == AUDIT_CONFIG_NONE => Some(false), + Some(_) => Some(true), + }; + match (file, otlp_configured) { + (None, false) => Ok(SinkSelection::NoAudit), + (Some(true), false) => Ok(SinkSelection::FileOnly), + (Some(true), true) => Ok(SinkSelection::Both), + (Some(false), true) => Ok(SinkSelection::OtlpOnly), + (Some(false), false) => anyhow::bail!( + "--audit-config/BUGWARDEN_AUDIT_CONFIG is `{AUDIT_CONFIG_NONE}` but no OTLP \ + endpoint is configured, which would leave no audit sink at all; set \ + OTEL_EXPORTER_OTLP_ENDPOINT for OTLP-only auditing, or unset \ + BUGWARDEN_AUDIT_CONFIG entirely if this deployment is meant to run \ + without an audit trail" + ), + (None, true) => anyhow::bail!( + "OTEL_EXPORTER_OTLP_ENDPOINT is set, so audit records are exported and \ + their delivery gates serving — but no audit file is configured; point \ + --audit-config/BUGWARDEN_AUDIT_CONFIG at an audit configuration to run \ + file and OTLP together, or set it to `{AUDIT_CONFIG_NONE}` for OTLP-only \ + auditing" + ), + } +} + // --------------------------------------------------------------------------- // Event schema (v1) // --------------------------------------------------------------------------- @@ -657,12 +770,26 @@ pub enum AuditError { /// Rotating the audit file failed. #[error("audit rotation failed")] Rotation(#[source] std::io::Error), + /// The export sink would not take the record — its queue is full, or + /// it has shut down. Content-free like its siblings, and deliberately + /// carrying no transport detail: the endpoint may not reach a + /// diagnostic (I12). + #[error("audit export refused the record")] + Export, } impl AuditError { fn gap_reason(&self) -> GapReason { match self { - AuditError::Serialize(_) | AuditError::Write(_) => GapReason::WriteError, + // `Export` reports as a write error on purpose: schema v1's + // vocabulary has no spelling for a delivery failure, and + // adding one would fork every reader of a v1 corpus over a + // record whose shape is unchanged. The DIAGNOSTIC says which + // sink failed; the record says only that records were lost. + // #34 owns the v2 vocabulary. + AuditError::Serialize(_) | AuditError::Write(_) | AuditError::Export => { + GapReason::WriteError + } AuditError::Rotation(_) => GapReason::RotationError, } } @@ -672,9 +799,75 @@ impl AuditError { // Sink // --------------------------------------------------------------------------- +/// A second, LOAD-BEARING destination for records, beside (or instead of) +/// the file. +/// +/// Revised 2026-08-18 (#31). Export was a best-effort copy the sink +/// ignored; it is now a sink in its own right. Every configured sink is +/// load-bearing: a record this one will not deliver puts the whole sink +/// into failure ([`AuditSink::failing`]) exactly as a failed file write +/// does, and the operator's [`FailMode`] decides what the server then does +/// about tool calls. `crate::otel` is the one implementation; the trait +/// exists so this module keeps no knowledge of OTLP. +/// +/// Ordering is unchanged and still matters: when a file is configured, the +/// record reaches it FIRST and the exporter only afterwards, so the file +/// never lacks a record the collector has. +/// +/// Obligations on an implementor: +/// +/// - **Never block.** Every call happens with the sink's write lock held, +/// so a blocking exporter would serialize behind every audit write and +/// stall the request path. [`AuditExport::accept`] takes custody and +/// returns; delivery is reported later through +/// [`AuditExport::delivery_failing`] and [`AuditExport::take_lost`]. +/// - **Never drop an audit record silently.** A record that cannot be +/// accepted must be REFUSED (so the call is gated) rather than dropped, +/// and one accepted but not delivered must be counted in +/// [`AuditExport::take_lost`] so it reaches an `audit_gap`. +/// - **`Debug` must be content-free.** [`AuditSink`] derives `Debug` and +/// would print an exporter through it, so an exporter holding +/// configuration (an endpoint, credentials) must hide it (I12). +pub trait AuditExport: Send + Sync + std::fmt::Debug { + /// Take custody of one record, or refuse it. + /// + /// `event` is the record, `line` the exact bytes the file carries for + /// it — without the terminating newline and without the leading one a + /// torn-line repair may have prefixed. Exporting `line` verbatim is + /// what makes the exported payload byte-equal to the file's (I12). + /// + /// # Errors + /// + /// [`ExportRefused`] when custody cannot be taken — a full queue, a + /// shut-down pipeline. The sink turns that into [`AuditError::Export`], + /// which the request path treats exactly like a failed write. + fn accept(&self, event: &AuditEvent, line: &[u8]) -> Result<(), ExportRefused>; + + /// Whether delivery is currently known not to work. + /// + /// True from the first failed attempt until one succeeds. This is what + /// keeps the gate closed across the whole outage rather than only at + /// the instants a record happens to be refused. + fn delivery_failing(&self) -> bool; + + /// Take, and reset, the count of accepted records that were never + /// delivered. The sink folds it into its own loss accounting, so the + /// `audit_gap` a reader sees covers both sinks. + fn take_lost(&self) -> u64; +} + +/// Why [`AuditExport::accept`] refused custody. +/// +/// The reason is not distinguished on the wire: a full queue and a +/// shut-down pipeline are the same failure as far as the fail-mode gate +/// is concerned. The type exists so the refusal is not `Result<(), ()>`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ExportRefused; + #[derive(Debug)] struct SinkState { - file: File, + /// The open live file, or `None` for a fileless (OTLP-only) sink. + file: Option, /// Current size of the live file, maintained across writes so /// rotation needs no `stat` per record. size: u64, @@ -711,6 +904,9 @@ struct SinkState { pub struct AuditSink { cfg: AuditConfig, state: Mutex, + /// Load-bearing exporter attached by [`AuditSink::with_export`]. + /// See [`AuditExport`]. + export: Option>, /// Test-only fault injection: when set, the write step fails without /// touching the file. Lets the gap-marker path be exercised without /// weakening the production API. @@ -746,37 +942,47 @@ impl AuditSink { // A hand-constructed config could bypass load-time validation, so // re-validate here; the check is free. cfg.validate()?; - if let Ok(meta) = std::fs::symlink_metadata(&cfg.path) { - if meta.file_type().is_symlink() { - anyhow::bail!( - "audit file {} is a symlink; refusing to follow it — point `path` \ - at a regular file in an operator-owned directory", - cfg.path.display() - ); - } - } - if let Some(parent) = cfg.path.parent() { - if !parent.as_os_str().is_empty() { - let mut builder = std::fs::DirBuilder::new(); - builder.recursive(true); - #[cfg(unix)] - { - use std::os::unix::fs::DirBuilderExt as _; - builder.mode(0o700); + // A fileless sink touches no filesystem at all: no symlink probe, + // no directory creation, no open. That is what lets a container + // run OTLP-only with nothing writable mounted. + let (file, size) = match &cfg.path { + None => (None, 0), + Some(path) => { + if let Ok(meta) = std::fs::symlink_metadata(path) { + if meta.file_type().is_symlink() { + anyhow::bail!( + "audit file {} is a symlink; refusing to follow it — point \ + `path` at a regular file in an operator-owned directory", + path.display() + ); + } } - builder.create(parent).with_context(|| { - format!("failed to create audit directory {}", parent.display()) - })?; + if let Some(parent) = path.parent() { + if !parent.as_os_str().is_empty() { + let mut builder = std::fs::DirBuilder::new(); + builder.recursive(true); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt as _; + builder.mode(0o700); + } + builder.create(parent).with_context(|| { + format!("failed to create audit directory {}", parent.display()) + })?; + } + } + let file = open_live_file(path) + .with_context(|| format!("failed to open audit file {}", path.display()))?; + let size = file + .metadata() + .with_context(|| format!("failed to stat audit file {}", path.display()))? + .len(); + (Some(file), size) } - } - let file = open_live_file(&cfg.path) - .with_context(|| format!("failed to open audit file {}", cfg.path.display()))?; - let size = file - .metadata() - .with_context(|| format!("failed to stat audit file {}", cfg.path.display()))? - .len(); + }; Ok(AuditSink { cfg, + export: None, state: Mutex::new(SinkState { file, size, @@ -795,6 +1001,20 @@ impl AuditSink { }) } + /// Attach the load-bearing exporter that receives every record this + /// sink persists; see [`AuditExport`]. A fileless sink MUST have one — + /// without it records would vanish silently — and `main`'s sink + /// selection is what guarantees that pairing. + /// + /// Consuming, so the exporter can only be attached while the sink is + /// still being built at startup — a running sink's export side never + /// changes, and the request path cannot reach it. + #[must_use] + pub fn with_export(mut self, export: std::sync::Arc) -> AuditSink { + self.export = Some(export); + self + } + /// Persist one record; returns its assigned `seq`. /// /// Blocks until the record is written (and synced, under `fsync`). If @@ -814,6 +1034,10 @@ impl AuditSink { /// over-report the loss. Over-reporting, never under-reporting, is /// the chosen direction. pub fn record(&self, kind: AuditEventKind, session: SessionInfo) -> Result { + debug_assert!( + self.cfg.path.is_some() || self.export.is_some(), + "a fileless audit sink must have an exporter; records would vanish" + ); // A poisoned mutex means a writer panicked; the size/seq state is // still usable (worst case an early rotation), so keep going — // dropping audit coverage over a bookkeeping wobble would be the @@ -822,6 +1046,12 @@ impl AuditSink { .state .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); + // Records the exporter accepted and then failed to deliver are + // losses of this sink like any other, so they join the same + // counter and surface through the same `audit_gap`. Folded before + // the gap check below, so an export outage is reported in the very + // next record rather than one later. + self.absorb_export_losses(&mut state); if state.dropped > 0 { let gap = AuditEventKind::AuditGap(AuditGapEvent { dropped: state.dropped, @@ -872,7 +1102,8 @@ impl AuditSink { // end of the file; a leading newline terminates it so the stream // self-heals (worst case one empty line, which parsers skip — // see the module docs). - if state.torn { + let healed = state.torn; + if healed { line.push(b'\n'); } serde_json::to_writer(&mut line, &event).map_err(AuditError::Serialize)?; @@ -900,32 +1131,57 @@ impl AuditSink { { // Write roughly half the line, then fail: a deterministic // stand-in for ENOSPC tearing a line mid-write. - state - .file - .write_all(&line[..line.len() / 2]) - .map_err(AuditError::Write)?; + if let Some(file) = state.file.as_mut() { + file.write_all(&line[..line.len() / 2]) + .map_err(AuditError::Write)?; + } state.torn = true; return Err(AuditError::Write(std::io::Error::other( "injected partial write failure (test)", ))); } - state.file.write_all(&line).map_err(|e| { - // An unknown number of bytes made it out: assume a torn line. - state.torn = true; - AuditError::Write(e) - })?; - if self.cfg.fsync { - state.file.sync_data().map_err(|e| { - // The line is complete in the page cache but may not have - // reached the disk; treat it as torn so the next record - // re-terminates whatever survives. + if let Some(file) = state.file.as_mut() { + file.write_all(&line).map_err(|e| { + // An unknown number of bytes made it out: assume a torn + // line. state.torn = true; AuditError::Write(e) })?; + if self.cfg.fsync { + file.sync_data().map_err(|e| { + // The line is complete in the page cache but may not + // have reached the disk; treat it as torn so the next + // record re-terminates whatever survives. + state.torn = true; + AuditError::Write(e) + })?; + } + state.size += len; } state.torn = false; - state.size += len; + // The file has the record and its `seq` is spent, whatever the + // export does next: a `seq` reused after a rejected export would + // put two different records on one number in the file. state.seq = seq; + if let Some(export) = &self.export { + // AFTER the write, never before: the file, when there is one, + // is authoritative, and a record it never took must not exist + // anywhere else. The slice strips the repair newline this + // record may have been prefixed with and the one terminating + // it, so what the exporter gets is byte-for-byte the record + // line the file holds. Still under the state lock, so export + // order equals `seq` order — accepting only queues. + let start = usize::from(healed); + // Load-bearing since 2026-08-18 (#31): a record the exporter + // cannot even accept is a record that will not be delivered, + // and saying so here is what puts the call in front of the + // same fail-mode gate a failed file write does. The file, if + // any, keeps the record — the loss accounting over-reports + // rather than under-reports, as it does under `fsync`. + export + .accept(&event, &line[start..line.len() - 1]) + .map_err(|ExportRefused| AuditError::Export)?; + } Ok(seq) } @@ -941,7 +1197,11 @@ impl AuditSink { { return Err(std::io::Error::other("injected rotation failure (test)")); } - let live = &self.cfg.path; + // Unreachable for a fileless sink: `write_event` only rotates when + // it holds a file, and `size` never leaves 0 without one. + let Some(live) = &self.cfg.path else { + return Ok(()); + }; // The oldest kept slot would shift past the cap: delete it. match std::fs::remove_file(rotated_path(live, self.cfg.rotate_keep)) { Ok(()) => {} @@ -963,7 +1223,7 @@ impl AuditSink { Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} Err(e) => return Err(e), } - state.file = open_live_file(live)?; + state.file = Some(open_live_file(live)?); state.size = 0; #[cfg(unix)] if let Some(parent) = live.parent() { @@ -989,23 +1249,54 @@ impl AuditSink { tracing::error!( error = ?err, dropped = state.dropped, - path = %self.cfg.path.display(), + path = self.cfg.path.as_ref().map(|p| p.display().to_string()), "audit record could not be persisted; records are being dropped" ); } } /// Whether the sink is currently in failure: at least one record has - /// been dropped since the last successful write. The request path + /// been lost since the last success on ANY configured destination, or + /// export delivery is known not to be working. The request path /// consults this before dispatching a tool call, so a sink that is /// already down can hold back further unaudited work under the closed /// fail modes instead of discovering the outage one record at a time. + /// + /// The second condition is what makes an export outage as visible as a + /// full disk. Loss counters clear as soon as their `audit_gap` is + /// written, and for the exporter "written" means "accepted for + /// delivery" — so a sink judged only by its counters would report + /// healthy the instant it queued the gap record, and flap once per + /// batch for the whole outage. Delivery health does not clear until a + /// request actually succeeds. pub fn failing(&self) -> bool { - self.state + let mut state = self + .state .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) - .dropped - > 0 + .unwrap_or_else(std::sync::PoisonError::into_inner); + // Also here, not only in `record`: the pre-dispatch gate asks this + // question far more often than it writes, and an outage that only + // became visible between two records must close the gate at the + // first ask. + self.absorb_export_losses(&mut state); + state.dropped > 0 + || self + .export + .as_ref() + .is_some_and(|export| export.delivery_failing()) + } + + /// Move any undelivered-record count out of the exporter and into this + /// sink's own loss accounting. Called with the state lock held. + fn absorb_export_losses(&self, state: &mut SinkState) { + let Some(export) = &self.export else { + return; + }; + let lost = export.take_lost(); + if lost > 0 { + state.dropped = state.dropped.saturating_add(lost); + state.gap_reason = GapReason::WriteError; + } } /// Whether records may carry suppressed bug ids @@ -1423,7 +1714,7 @@ mod tests { fn test_cfg(path: PathBuf) -> AuditConfig { AuditConfig { - path, + path: Some(path), fsync: false, fail_mode: None, rotate_max_bytes: 0, @@ -1432,6 +1723,12 @@ mod tests { } } + /// The live file of a config built by [`test_cfg`], which always + /// names one. + fn live_path(cfg: &AuditConfig) -> &Path { + cfg.path.as_deref().expect("test config names a file") + } + /// Parse every line of an audit file. Empty lines are skipped, as /// the module docs require of parsers: a recovered write outage may /// leave one behind. @@ -1448,8 +1745,8 @@ mod tests { /// limit is a hard bound (a single oversized record is the one /// documented exception, tested separately). fn assert_files_within_rotate_limit(cfg: &AuditConfig) { - let mut paths = vec![cfg.path.clone()]; - paths.extend((1..=cfg.rotate_keep).map(|n| rotated_path(&cfg.path, n))); + let mut paths = vec![live_path(cfg).to_path_buf()]; + paths.extend((1..=cfg.rotate_keep).map(|n| rotated_path(live_path(cfg), n))); for p in paths { if let Ok(meta) = std::fs::metadata(&p) { assert!( @@ -1728,7 +2025,10 @@ mod tests { #[test] fn config_minimal_file_applies_defaults() { let cfg = AuditConfig::from_toml_str("path = \"/var/log/bugwarden/audit.jsonl\"").unwrap(); - assert_eq!(cfg.path, PathBuf::from("/var/log/bugwarden/audit.jsonl")); + assert_eq!( + cfg.path.as_deref(), + Some(Path::new("/var/log/bugwarden/audit.jsonl")) + ); assert!(!cfg.fsync); assert_eq!(cfg.fail_mode, None); assert_eq!(cfg.rotate_max_bytes, 64 * 1024 * 1024); @@ -1854,7 +2154,7 @@ mod tests { second.record(sample_initialize(), session_stdio()).unwrap(), 1 ); - let seqs: Vec = read_events(&cfg.path).iter().map(|e| e.seq).collect(); + let seqs: Vec = read_events(live_path(&cfg)).iter().map(|e| e.seq).collect(); assert_eq!(seqs, vec![1, 2, 1]); } @@ -1887,13 +2187,13 @@ mod tests { let mut seqs: Vec = Vec::new(); let mut rotated = 0; for n in (1..=cfg.rotate_keep).rev() { - let p = rotated_path(&cfg.path, n); + let p = rotated_path(live_path(&cfg), n); if p.exists() { rotated += 1; seqs.extend(read_events(&p).iter().map(|e| e.seq)); } } - seqs.extend(read_events(&cfg.path).iter().map(|e| e.seq)); + seqs.extend(read_events(live_path(&cfg)).iter().map(|e| e.seq)); assert!(rotated >= 2, "expected multiple rotations, saw {rotated}"); assert!( rotated <= cfg.rotate_keep, @@ -1908,7 +2208,10 @@ mod tests { #[cfg(unix)] { use std::os::unix::fs::PermissionsExt as _; - let mode = std::fs::metadata(&cfg.path).unwrap().permissions().mode(); + let mode = std::fs::metadata(live_path(&cfg)) + .unwrap() + .permissions() + .mode(); assert_eq!(mode & 0o777, 0o600, "recreated live file must be 0600"); } } @@ -1923,15 +2226,15 @@ mod tests { for _ in 0..8 { sink.record(sample_initialize(), session_stdio()).unwrap(); } - assert!(rotated_path(&cfg.path, 1).exists()); + assert!(rotated_path(live_path(&cfg), 1).exists()); assert!( - !rotated_path(&cfg.path, 2).exists(), + !rotated_path(live_path(&cfg), 2).exists(), "rotation must not keep more than rotate_keep files" ); // What survives still parses and stays in order (deletion loses // the oldest records — that is the operator's configured trade). - let mut events = read_events(&rotated_path(&cfg.path, 1)); - events.extend(read_events(&cfg.path)); + let mut events = read_events(&rotated_path(live_path(&cfg), 1)); + events.extend(read_events(live_path(&cfg))); let seqs: Vec = events.iter().map(|e| e.seq).collect(); assert!( seqs.len() < 8, @@ -1962,19 +2265,19 @@ mod tests { // The oversized record went into the empty live file as-is — // the one documented exception to the size bound. assert!( - std::fs::metadata(&cfg.path).unwrap().len() > cfg.rotate_max_bytes, + std::fs::metadata(live_path(&cfg)).unwrap().len() > cfg.rotate_max_bytes, "a single oversized record must be written despite the limit" ); - assert!(!rotated_path(&cfg.path, 1).exists()); + assert!(!rotated_path(live_path(&cfg), 1).exists()); // The next record must rotate the oversized live file out first. assert_eq!( sink.record(sample_initialize(), session_stdio()).unwrap(), 2 ); - let rotated = read_events(&rotated_path(&cfg.path, 1)); + let rotated = read_events(&rotated_path(live_path(&cfg), 1)); assert_eq!(rotated.len(), 1, "rotated file holds the first record"); assert_eq!(rotated[0].seq, 1); - let live = read_events(&cfg.path); + let live = read_events(live_path(&cfg)); assert_eq!(live.len(), 1, "live file holds only the new record"); assert_eq!(live[0].seq, 2); } @@ -1993,13 +2296,13 @@ mod tests { let second = AuditSink::open(cfg.clone()).unwrap(); second.record(sample_initialize(), session_stdio()).unwrap(); assert!( - rotated_path(&cfg.path, 1).exists(), + rotated_path(live_path(&cfg), 1).exists(), "reopen must count existing bytes toward the rotation limit" ); - let live = read_events(&cfg.path); + let live = read_events(live_path(&cfg)); assert_eq!(live.len(), 1, "live file holds only the new record"); assert_eq!(live[0].seq, 1, "a fresh sink assigns seq from 1"); - assert_eq!(read_events(&rotated_path(&cfg.path, 1)).len(), 1); + assert_eq!(read_events(&rotated_path(live_path(&cfg), 1)).len(), 1); } #[test] @@ -2017,7 +2320,7 @@ mod tests { }); } }); - let seqs: Vec = read_events(&cfg.path).iter().map(|e| e.seq).collect(); + let seqs: Vec = read_events(live_path(&cfg)).iter().map(|e| e.seq).collect(); // Unique, contiguous from 1, and file order equals seq order — // all three in one assertion. assert_eq!(seqs, (1..=100).collect::>()); @@ -2046,7 +2349,7 @@ mod tests { sink.record(sample_initialize(), session_stdio()).unwrap(), 3 ); - let events = read_events(&cfg.path); + let events = read_events(live_path(&cfg)); assert_eq!(events.len(), 3); assert_eq!(events[1].seq, 2); match &events[1].kind { @@ -2089,14 +2392,14 @@ mod tests { let mut raw = String::new(); let mut events = Vec::new(); for n in (1..=cfg.rotate_keep).rev() { - let p = rotated_path(&cfg.path, n); + let p = rotated_path(live_path(&cfg), n); if p.exists() { raw.push_str(&std::fs::read_to_string(&p).unwrap()); events.extend(read_events(&p)); } } - raw.push_str(&std::fs::read_to_string(&cfg.path).unwrap()); - events.extend(read_events(&cfg.path)); + raw.push_str(&std::fs::read_to_string(live_path(&cfg)).unwrap()); + events.extend(read_events(live_path(&cfg))); let seqs: Vec = events.iter().map(|e| e.seq).collect(); assert_eq!(seqs, vec![1, 2, 3]); match &events[1].kind { @@ -2127,7 +2430,7 @@ mod tests { .unwrap_err(); assert!(matches!(err, AuditError::Write(_))); sink.set_fail_writes_partial(false); - let raw = std::fs::read_to_string(&cfg.path).unwrap(); + let raw = std::fs::read_to_string(live_path(&cfg)).unwrap(); assert!( !raw.ends_with('\n'), "the partial write must have left a torn, unterminated line" @@ -2138,7 +2441,7 @@ mod tests { sink.record(sample_initialize(), session_stdio()).unwrap(), 3 ); - let raw = std::fs::read_to_string(&cfg.path).unwrap(); + let raw = std::fs::read_to_string(live_path(&cfg)).unwrap(); let mut torn_lines = 0; let mut events: Vec = Vec::new(); for line in raw.lines().filter(|l| !l.is_empty()) { @@ -2165,7 +2468,7 @@ mod tests { let cfg = test_cfg(dir.path().join("audit.jsonl")); let sink = AuditSink::open(cfg.clone()).unwrap(); sink.record(sample_initialize(), session_stdio()).unwrap(); - let events = read_events(&cfg.path); + let events = read_events(live_path(&cfg)); let ts = &events[0].ts; // ^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$, spelled out // positionally ('d' = ASCII digit) to avoid a regex dependency. @@ -2195,7 +2498,7 @@ mod tests { sink.record(sample_tool_call(), session_http()).unwrap(); sink.record(sample_initialize(), session_stdio()).unwrap(); sink.record(sample_gap(), session_stdio()).unwrap(); - let contents = std::fs::read_to_string(&cfg.path).unwrap(); + let contents = std::fs::read_to_string(live_path(&cfg)).unwrap(); assert_eq!(contents.lines().count(), 3); assert!( !contents.contains(CANARY), @@ -2440,6 +2743,260 @@ mod tests { ); } + // ---------- sink selection (issue #31, revised 2026-08-18) ---------- + + #[test] + fn select_sinks_resolves_the_four_deployments() { + assert_eq!(select_sinks(None, false).unwrap(), SinkSelection::NoAudit); + assert_eq!( + select_sinks(Some(Path::new("/etc/bugwarden/audit.toml")), false).unwrap(), + SinkSelection::FileOnly + ); + assert_eq!( + select_sinks(Some(Path::new("/etc/bugwarden/audit.toml")), true).unwrap(), + SinkSelection::Both + ); + assert_eq!( + select_sinks(Some(Path::new(AUDIT_CONFIG_NONE)), true).unwrap(), + SinkSelection::OtlpOnly + ); + } + + #[test] + fn select_sinks_refuses_the_two_ambiguous_spellings() { + // `none` with no OTLP endpoint: an explicit request for a sink + // that leaves zero of them. + let err = format!( + "{}", + select_sinks(Some(Path::new("none")), false).unwrap_err() + ); + assert!( + err.contains("no audit sink") && err.contains("OTEL_EXPORTER_OTLP_ENDPOINT"), + "the refusal must explain both ways out: {err}" + ); + // An endpoint with no file decision: OTLP-only must be said, not + // inferred, or a typo'd BUGWARDEN_AUDIT_CONFIG silently drops the + // file from a deployment that meant to run both. + let err = format!("{}", select_sinks(None, true).unwrap_err()); + assert!( + err.contains("BUGWARDEN_AUDIT_CONFIG") && err.contains("`none`"), + "the refusal must name the explicit spellings: {err}" + ); + } + + #[test] + fn the_none_sentinel_is_exact_bytes() { + // Anything else is a path: `None` and `./none` are files an + // operator can really have, and over-matching would disable + // auditing on a spelling nobody wrote. + assert_eq!( + select_sinks(Some(Path::new("None")), true).unwrap(), + SinkSelection::Both + ); + assert_eq!( + select_sinks(Some(Path::new("./none")), true).unwrap(), + SinkSelection::Both + ); + } + + // ---------- the exporter as a load-bearing sink ---------- + + /// A scriptable exporter for sink-side tests. + #[derive(Debug, Default)] + struct StubExport { + refuse: std::sync::atomic::AtomicBool, + failing: std::sync::atomic::AtomicBool, + lost: std::sync::atomic::AtomicU64, + accepted: Mutex>>, + } + + impl AuditExport for StubExport { + fn accept(&self, _event: &AuditEvent, line: &[u8]) -> Result<(), ExportRefused> { + if self.refuse.load(std::sync::atomic::Ordering::Relaxed) { + return Err(ExportRefused); + } + self.accepted.lock().expect("stub lock").push(line.to_vec()); + Ok(()) + } + + fn delivery_failing(&self) -> bool { + self.failing.load(std::sync::atomic::Ordering::Relaxed) + } + + fn take_lost(&self) -> u64 { + self.lost.swap(0, std::sync::atomic::Ordering::Relaxed) + } + } + + #[test] + fn a_fileless_sink_records_through_the_exporter_alone() { + let cfg = AuditConfig::fileless(); + assert!(cfg.path.is_none(), "fileless names no file"); + let export = std::sync::Arc::new(StubExport::default()); + let sink = AuditSink::open(cfg).unwrap().with_export(export.clone()); + assert_eq!( + sink.record(sample_initialize(), session_stdio()).unwrap(), + 1 + ); + assert_eq!(sink.record(sample_tool_call(), session_http()).unwrap(), 2); + let accepted = export.accepted.lock().expect("stub lock").clone(); + assert_eq!(accepted.len(), 2, "every record reaches the exporter"); + // The handed-over line is the record, parseable and in seq order — + // the same bytes a file would have carried. + let events: Vec = accepted + .iter() + .map(|line| serde_json::from_slice(line).expect("a parseable record line")) + .collect(); + assert_eq!(events.iter().map(|e| e.seq).collect::>(), vec![1, 2]); + assert!(!sink.failing()); + } + + #[test] + fn an_export_refusal_fails_the_record_and_the_file_still_keeps_it() { + let dir = tempfile::tempdir().unwrap(); + let cfg = test_cfg(dir.path().join("audit.jsonl")); + let export = std::sync::Arc::new(StubExport::default()); + let sink = AuditSink::open(cfg.clone()) + .unwrap() + .with_export(export.clone()); + export + .refuse + .store(true, std::sync::atomic::Ordering::Relaxed); + let err = sink + .record(sample_initialize(), session_stdio()) + .unwrap_err(); + assert!( + matches!(err, AuditError::Export), + "a refused hand-off is an export failure: {err:?}" + ); + // The FILE write preceded the refusal and stands, its seq spent: + // the loss accounting over-reports rather than under-reports, + // exactly as a failed fsync does. + let events = read_events(live_path(&cfg)); + assert_eq!(events.len(), 1); + assert_eq!(events[0].seq, 1); + assert!(sink.failing(), "a refused record puts the sink in failure"); + // Recovery: the next record is preceded by the gap accounting it. + export + .refuse + .store(false, std::sync::atomic::Ordering::Relaxed); + assert_eq!( + sink.record(sample_initialize(), session_stdio()).unwrap(), + 3 + ); + let events = read_events(live_path(&cfg)); + assert_eq!(events.len(), 3); + match &events[1].kind { + AuditEventKind::AuditGap(gap) => { + assert_eq!(gap.dropped, 1); + assert_eq!(gap.reason, GapReason::WriteError); + } + other => panic!("expected audit_gap, got {other:?}"), + } + } + + #[test] + fn export_delivery_failure_alone_puts_the_sink_in_failure() { + // The gate condition: delivery health, not only a counter — a + // counter clears the moment its gap is queued, and a gate read + // off it alone would flap once per batch for the whole outage. + let export = std::sync::Arc::new(StubExport::default()); + let sink = AuditSink::open(AuditConfig::fileless()) + .unwrap() + .with_export(export.clone()); + assert!(!sink.failing()); + export + .failing + .store(true, std::sync::atomic::Ordering::Relaxed); + assert!( + sink.failing(), + "failing delivery must gate even with nothing dropped" + ); + export + .failing + .store(false, std::sync::atomic::Ordering::Relaxed); + assert!(!sink.failing(), "recovered delivery must clear the gate"); + } + + #[test] + fn undelivered_export_losses_surface_as_an_audit_gap() { + let dir = tempfile::tempdir().unwrap(); + let cfg = test_cfg(dir.path().join("audit.jsonl")); + let export = std::sync::Arc::new(StubExport::default()); + let sink = AuditSink::open(cfg.clone()) + .unwrap() + .with_export(export.clone()); + // Three records accepted earlier were never delivered. + export.lost.store(3, std::sync::atomic::Ordering::Relaxed); + assert!(sink.failing(), "known losses put the sink in failure"); + assert_eq!( + sink.record(sample_initialize(), session_stdio()).unwrap(), + 2 + ); + let events = read_events(live_path(&cfg)); + assert_eq!(events.len(), 2); + match &events[0].kind { + AuditEventKind::AuditGap(gap) => { + assert_eq!(gap.dropped, 3, "the gap carries the loss count"); + assert_eq!(gap.reason, GapReason::WriteError); + } + other => panic!("expected audit_gap, got {other:?}"), + } + // The gap record itself is exported too: the collector sees its + // own outage accounted in the stream it lost records from. + let accepted = export.accepted.lock().expect("stub lock").clone(); + assert_eq!(accepted.len(), 2, "gap and record both reach the exporter"); + } + + /// An exporter that proves the file write PRECEDES the hand-off: at + /// accept time the line must already be on disk. + #[derive(Debug)] + struct FileCheckingExport { + path: PathBuf, + checked: std::sync::atomic::AtomicU64, + } + + impl AuditExport for FileCheckingExport { + fn accept(&self, _event: &AuditEvent, line: &[u8]) -> Result<(), ExportRefused> { + let contents = std::fs::read(&self.path).expect("the audit file exists at accept"); + assert!( + contents.windows(line.len()).any(|window| window == line), + "the file must already hold the record when the exporter sees it" + ); + self.checked + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + Ok(()) + } + + fn delivery_failing(&self) -> bool { + false + } + + fn take_lost(&self) -> u64 { + 0 + } + } + + #[test] + fn with_both_sinks_the_file_write_precedes_the_export() { + let dir = tempfile::tempdir().unwrap(); + let cfg = test_cfg(dir.path().join("audit.jsonl")); + let export = std::sync::Arc::new(FileCheckingExport { + path: live_path(&cfg).to_path_buf(), + checked: std::sync::atomic::AtomicU64::new(0), + }); + let sink = AuditSink::open(cfg.clone()) + .unwrap() + .with_export(export.clone()); + sink.record(sample_initialize(), session_stdio()).unwrap(); + sink.record(sample_tool_call(), session_http()).unwrap(); + assert_eq!( + export.checked.load(std::sync::atomic::Ordering::Relaxed), + 2, + "the ordering assertion must actually have run" + ); + } + #[test] fn policy_hash_of_is_fixed_width_lowercase_hex() { // The vector above pins one exact value; this pins the encoding diff --git a/crates/bugwarden/src/bin/bugwarden-gen.rs b/crates/bugwarden/src/bin/bugwarden-gen.rs index 49b9503..cff3a83 100644 --- a/crates/bugwarden/src/bin/bugwarden-gen.rs +++ b/crates/bugwarden/src/bin/bugwarden-gen.rs @@ -142,6 +142,7 @@ fn render_environment(cmd: &clap::Command, buf: &mut Vec) -> std::io::Result // environment-only, because argv is world-readable — so the loop above // cannot see them. buf.write_all(HTTP_TOKEN_ENV.as_bytes())?; + buf.write_all(OTLP_ENV.as_bytes())?; Ok(()) } @@ -162,6 +163,37 @@ unless is given. "; +const OTLP_ENV: &str = r".TP +.B OTEL_EXPORTER_OTLP_ENDPOINT +Base URL of an OpenTelemetry collector (bugwarden appends /v1/logs). +Environment only. Unset or empty turns export off. Must be paired with +.B \-\-audit\-config +/ BUGWARDEN_AUDIT_CONFIG (a file, or the exact value +.BR none ). +An endpoint with no file decision is a startup error. +.TP +.B OTEL_EXPORTER_OTLP_HEADERS +Headers added to every export request, key=value separated by commas. +Credential material (I12): environment only, never logged. +.TP +.B OTEL_EXPORTER_OTLP_PROTOCOL +The one accepted value is http/protobuf; anything else is a startup +error. Not consulted while export is off. +.TP +.B OTEL_SERVICE_NAME +service.name on exported records. Defaults to bugwarden. +.TP +.B OTEL_EXPORTER_OTLP_LOGS_ENDPOINT +Logs-specific endpoint; overrides OTEL_EXPORTER_OTLP_ENDPOINT and is +used as given (write the whole URL including /v1/logs). +.TP +.B OTEL_EXPORTER_OTLP_LOGS_HEADERS +Logs-specific headers; overrides OTEL_EXPORTER_OTLP_HEADERS. Same secrecy. +.TP +.B OTEL_EXPORTER_OTLP_LOGS_PROTOCOL +Logs-specific protocol; overrides OTEL_EXPORTER_OTLP_PROTOCOL. +"; + const EXIT_STATUS: &str = r".SH EXIT STATUS .TP .B 0 @@ -170,7 +202,8 @@ Clean shutdown. .B 1 Startup or runtime failure: a missing or malformed http bearer token, an unreadable policy or audit configuration, a key misconfiguration, an -unparsable \-\-allowed\-hosts list, a Bugzilla client or transport error. +unparsable \-\-allowed\-hosts list, an OTLP collector that will not take +records, a Bugzilla client or transport error. .TP .B 2 Command\-line usage error. @@ -195,7 +228,9 @@ Without .B \-\-audit\-config or .B BUGWARDEN_AUDIT_CONFIG -no audit stream is written. +and with no OTLP endpoint, no audit stream is written. The exact value +.B none +disables the file so a collector can be the only sink. "; const EXAMPLES: &str = r#".SH EXAMPLES diff --git a/crates/bugwarden/src/config.rs b/crates/bugwarden/src/config.rs index a9a9573..bf5740a 100644 --- a/crates/bugwarden/src/config.rs +++ b/crates/bugwarden/src/config.rs @@ -109,8 +109,11 @@ pub struct Cli { pub policy: Option, /// Path to the audit configuration TOML file (see examples/audit.toml). - /// Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. - /// Without it no audit stream is written. + /// Environment variable BUGWARDEN_AUDIT_CONFIG can also be used. The + /// exact value `none` disables the audit file (OTLP-only when an + /// endpoint is set). Without it, and with no OTLP endpoint, no audit + /// stream is written. An endpoint with no file decision, or `none` + /// with no endpoint, is a startup error. #[arg(long, env = "BUGWARDEN_AUDIT_CONFIG", value_hint = clap::ValueHint::FilePath)] pub audit_config: Option, diff --git a/crates/bugwarden/src/lib.rs b/crates/bugwarden/src/lib.rs index 4f53c73..fcd4022 100644 --- a/crates/bugwarden/src/lib.rs +++ b/crates/bugwarden/src/lib.rs @@ -13,6 +13,9 @@ pub mod audit; pub mod config; /// Bearer authentication for the streamable-HTTP transport. pub mod http_auth; +/// OTLP export of the audit stream (load-bearing) and of the server's +/// own diagnostics (best-effort). +pub mod otel; pub mod server; #[cfg(test)] diff --git a/crates/bugwarden/src/main.rs b/crates/bugwarden/src/main.rs index 56479e7..cd7b5e5 100644 --- a/crates/bugwarden/src/main.rs +++ b/crates/bugwarden/src/main.rs @@ -5,13 +5,15 @@ //! live in the `bugwarden` library crate (`config`, `server`) so integration //! tests can drive the tools without a process boundary. -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; use anyhow::Context; use bugwarden::audit::{ - policy_hash_of, AuditConfig, AuditSink, AuditState, FailMode, TransportKind, + policy_hash_of, select_sinks, AuditConfig, AuditSink, AuditState, FailMode, SinkSelection, + TransportKind, }; use bugwarden::http_auth::{self, HttpEnv}; +use bugwarden::otel::{self, OtelEnv, Pipeline}; use bugwarden::{config, server}; use bugwarden_core::{guard::Guard, policy::Policy}; use clap::Parser; @@ -22,6 +24,8 @@ use rmcp::{ }, ServiceExt, }; +use tracing_subscriber::layer::SubscriberExt as _; +use tracing_subscriber::util::SubscriberInitExt as _; use tracing_subscriber::EnvFilter; use config::{Cli, Transport}; @@ -30,14 +34,33 @@ use config::{Cli, Transport}; async fn main() -> anyhow::Result<()> { let cli = Cli::parse(); + // OTLP export configuration, resolved before anything is installed: + // parsing the environment starts no task and opens no socket, but a + // protocol this build cannot speak has to be a startup error rather + // than a surprise once records exist. Unset or empty + // OTEL_EXPORTER_OTLP_ENDPOINT means the whole feature stays off. + let otel_config = otel::resolve(&OtelEnv::from_env())?; + // Filled in below, once the audit sink has opened; the diagnostics + // layer reads it and does nothing while it is empty. + let otel_slot: Arc>> = Arc::new(OnceLock::new()); + // Tracing always goes to stderr: stdout belongs to the stdio transport. - tracing_subscriber::fmt() - .with_env_filter( - EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")), - ) - .with_writer(std::io::stderr) - .with_ansi(false) - .init(); + // The OTLP layer, when export is on, sits beside the stderr one under + // the same filter, so both carry the same events. + let registry = tracing_subscriber::registry() + .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info"))) + .with( + tracing_subscriber::fmt::layer() + .with_writer(std::io::stderr) + .with_ansi(false), + ); + if otel_config.is_some() { + registry + .with(Pipeline::diagnostics_layer(otel_slot.clone())) + .init(); + } else { + registry.init(); + } // The http bearer gate, resolved before anything else runs: over http // the port is the access boundary, and a half-configured credential must @@ -53,6 +76,14 @@ async fn main() -> anyhow::Result<()> { auth.log_startup_mode(); } + // Which audit sinks this deployment runs (issue #31, revised + // 2026-08-18): file, OTLP, both, or none. Resolved here — pure + // configuration, no file touched — so the two ambiguous spellings + // (an endpoint with no file decision, `none` with no endpoint) refuse + // startup before any network or filesystem work. The sinks themselves + // open after the preflight, below, exactly as before. + let sinks = select_sinks(cli.audit_config.as_deref(), otel_config.is_some())?; + // Guard policy comes ONLY from the TOML file given at startup (I1); // without one the built-in default policy applies. let mut policy = match &cli.policy { @@ -81,12 +112,53 @@ async fn main() -> anyhow::Result<()> { // file (the same ordering rationale as key custody resolution above). server.preflight().await?; - // Audit stream, when configured. The fail mode falls back to the - // transport-derived default; the policy digest ties every record to - // the policy document in force. - let audit = match &cfg.audit_config { - Some(path) => { - let audit_cfg = AuditConfig::load(path)?; + // OTLP export, started AFTER the identity preflight and BEFORE the + // audit file is created: a collector that will not take records must + // refuse to start without leaving a file behind, the same ordering + // the preflight already keeps. Off entirely when no endpoint is + // configured: no task, no thread, no layer. + let otel = match otel_config { + Some(otel_cfg) => Some(Arc::new(Pipeline::start(otel_cfg)?)), + None => None, + }; + if let Some(pipeline) = &otel { + // Fills the slot the diagnostics layer installed above reads. + let _ = otel_slot.set(pipeline.clone()); + // The delivery probe, mirroring the identity preflight: audit + // records are load-bearing on this collector, so a deployment + // that cannot deliver them refuses to start — a startup failure + // now beats a wave of fail-mode refusals under load. Bounded + // retry inside, because a collector starting alongside us is + // allowed to lose the race by a few seconds. + pipeline.probe().await?; + tracing::info!( + "OTLP export enabled: audit records are exported to the configured \ + collector and gate serving when delivery fails; diagnostics are \ + copied best-effort" + ); + } + + // Audit sinks, per the selection above. The fail mode falls back to + // the transport-derived default — an OTLP-only sink has no document + // to say otherwise, so it always derives — and the policy digest ties + // every record to the policy document in force. + let audit_sink = match sinks { + SinkSelection::NoAudit => None, + SinkSelection::FileOnly | SinkSelection::Both | SinkSelection::OtlpOnly => { + let audit_cfg = match sinks { + SinkSelection::FileOnly | SinkSelection::Both => { + match cfg.audit_config.as_deref() { + Some(path) => AuditConfig::load(path)?, + // Unreachable: `select_sinks` returns a file-bearing + // selection only for a real path. Refuse rather than + // panic, like the http bearer gate below. + None => anyhow::bail!( + "internal error: a file-bearing sink selection has no path" + ), + } + } + _ => AuditConfig::fileless(), + }; let transport = match cfg.transport { Transport::Stdio => TransportKind::Stdio, Transport::Http => TransportKind::Http, @@ -111,14 +183,23 @@ async fn main() -> anyhow::Result<()> { None => None, }; let sink = AuditSink::open(audit_cfg).context("failed to open the audit sink")?; - Some(Arc::new(AuditState::new(sink, fail_mode, policy_hash))) + Some((sink, fail_mode, policy_hash)) } - None => None, }; + + let audit = audit_sink.map(|(sink, fail_mode, policy_hash)| { + let sink = match &otel { + Some(pipeline) => sink.with_export(pipeline.audit_exporter()), + None => sink, + }; + Arc::new(AuditState::new(sink, fail_mode, policy_hash)) + }); if audit.is_none() && cfg.transport == Transport::Http { tracing::warn!( "auditing is OFF: remote tool calls over http will leave no audit \ - record — pass --audit-config / BUGWARDEN_AUDIT_CONFIG to enable it" + record — pass --audit-config / BUGWARDEN_AUDIT_CONFIG for a file, \ + or set OTEL_EXPORTER_OTLP_ENDPOINT with BUGWARDEN_AUDIT_CONFIG=none \ + for a collector-only trail" ); } @@ -132,88 +213,118 @@ async fn main() -> anyhow::Result<()> { let server = server.with_scope_enforcement(http_auth.as_ref().is_some_and(|auth| !auth.is_insecure())); - match cfg.transport { - Transport::Stdio => { - // Two stages: an unused stdio container sits in `serve` - // (handshake). After initialize it sits in `waiting`. - // Handlers register on the first poll of `shutdown`, which - // is this `select!` immediately after the startup line. - let shutdown = shutdown_signal(); - tokio::pin!(shutdown); - tracing::info!("Starting Bugzilla MCP server on stdio"); - let service = tokio::select! { - result = server.serve(stdio()) => { - result.inspect_err(|e| { - tracing::error!("serving error: {:?}", e); - })? - } - () = &mut shutdown => { - tracing::info!("received shutdown signal"); - // serve() is already blocked in tokio::io::stdin()'s - // uncancellable read. Returning from main drops the - // runtime onto that blocking thread. - std::process::exit(0); - } - }; - let cancel = service.cancellation_token(); - tokio::select! { - result = service.waiting() => { - result?; + // Cloned into the serve future so a stdio SIGTERM can flush before + // `process::exit`. The original stays here for the HTTP / peer-close + // path, which returns from the future instead of exiting. + let otel_on_signal = otel.clone(); + // The result is held rather than propagated so the OTLP flush below + // runs on every exit path, a transport error included. + let served: anyhow::Result<()> = async move { + match cfg.transport { + Transport::Stdio => { + // Two stages: an unused stdio container sits in `serve` + // (handshake). After initialize it sits in `waiting`. + // Handlers register on the first poll of `shutdown`, which + // is this `select!` immediately after the startup line. + let shutdown = shutdown_signal(); + tokio::pin!(shutdown); + tracing::info!("Starting Bugzilla MCP server on stdio"); + let service = tokio::select! { + result = server.serve(stdio()) => { + result.inspect_err(|e| { + tracing::error!("serving error: {:?}", e); + })? + } + () = &mut shutdown => { + tracing::info!("received shutdown signal"); + // serve() is already blocked in tokio::io::stdin()'s + // uncancellable read. Returning from main drops the + // runtime onto that blocking thread. Flush first: + // process::exit would otherwise skip the tail of a + // load-bearing OTLP-only sink. + flush_otel_and_exit(otel_on_signal.as_deref()).await; + } + }; + let cancel = service.cancellation_token(); + tokio::select! { + result = service.waiting() => { + result?; + } + () = shutdown => { + tracing::info!("received shutdown signal"); + cancel.cancel(); + flush_otel_and_exit(otel_on_signal.as_deref()).await; + } } - () = shutdown => { + } + Transport::Http => { + let ct = tokio_util::sync::CancellationToken::new(); + + // Derived from the server's own guard policy (the POST body cap + // follows `global.max_attachment_bytes`, issue #52), so it is + // built while `server` can still be borrowed. + let config = server + .http_server_config()? + .with_cancellation_token(ct.child_token()); + let service = StreamableHttpService::new( + move || Ok(server.clone()), + LocalSessionManager::default().into(), + config, + ); + // `resolve_for` returns the gate for every http start, so the + // bail below is unreachable — it exists so a future refactor + // that lost the gate fails to serve rather than serving open. + let Some(auth) = http_auth else { + anyhow::bail!("internal error: the http transport has no resolved bearer gate"); + }; + let router = http_auth::guard_router( + axum::Router::new().nest_service("/mcp", service), + auth, + ); + let addr = format!("{}:{}", cfg.host, cfg.port); + tracing::info!("Starting Bugzilla MCP server on {addr}"); + let tcp_listener = tokio::net::TcpListener::bind(&addr) + .await + .with_context(|| format!("failed to bind {addr}"))?; + // Connect-info makes the remote peer address available in the + // request extensions, where the audit session info reads it. + axum::serve( + tcp_listener, + router.into_make_service_with_connect_info::(), + ) + .with_graceful_shutdown(async move { + shutdown_signal().await; tracing::info!("received shutdown signal"); - cancel.cancel(); - // Same blocking stdin read as the handshake arm. - std::process::exit(0); - } + // Tear the live streamable-HTTP transport down with the + // listener: axum's graceful shutdown alone waits for + // in-flight connections, and an open MCP session is one. + ct.cancel(); + }) + .await?; } } - Transport::Http => { - let ct = tokio_util::sync::CancellationToken::new(); - - // Derived from the server's own guard policy (the POST body cap - // follows `global.max_attachment_bytes`, issue #52), so it is - // built while `server` can still be borrowed. - let config = server - .http_server_config()? - .with_cancellation_token(ct.child_token()); - let service = StreamableHttpService::new( - move || Ok(server.clone()), - LocalSessionManager::default().into(), - config, - ); - // `resolve_for` returns the gate for every http start, so the - // bail below is unreachable — it exists so a future refactor - // that lost the gate fails to serve rather than serving open. - let Some(auth) = http_auth else { - anyhow::bail!("internal error: the http transport has no resolved bearer gate"); - }; - let router = - http_auth::guard_router(axum::Router::new().nest_service("/mcp", service), auth); - let addr = format!("{}:{}", cfg.host, cfg.port); - tracing::info!("Starting Bugzilla MCP server on {addr}"); - let tcp_listener = tokio::net::TcpListener::bind(&addr) - .await - .with_context(|| format!("failed to bind {addr}"))?; - // Connect-info makes the remote peer address available in the - // request extensions, where the audit session info reads it. - axum::serve( - tcp_listener, - router.into_make_service_with_connect_info::(), - ) - .with_graceful_shutdown(async move { - shutdown_signal().await; - tracing::info!("received shutdown signal"); - // Tear the live streamable-HTTP transport down with the - // listener: axum's graceful shutdown alone waits for - // in-flight connections, and an open MCP session is one. - ct.cancel(); - }) - .await?; - } + Ok(()) + } + .await; + + // Best-effort, bounded flush of whatever is still queued. On http this + // runs after graceful shutdown, i.e. after the SIGINT that cancelled + // it; on stdio after the peer closed the session. + if let Some(pipeline) = &otel { + pipeline.shutdown().await; } - Ok(()) + served +} + +/// Bounded OTLP flush, then `_exit`. Used on the stdio signal arms: those +/// cannot return from `main` (rmcp's stdin read is uncancellable) and +/// must not skip a load-bearing collector either. +async fn flush_otel_and_exit(otel: Option<&Pipeline>) -> ! { + if let Some(pipeline) = otel { + pipeline.shutdown().await; + } + std::process::exit(0); } /// Wait until the process should stop serving. diff --git a/crates/bugwarden/src/otel.rs b/crates/bugwarden/src/otel.rs new file mode 100644 index 0000000..2786912 --- /dev/null +++ b/crates/bugwarden/src/otel.rs @@ -0,0 +1,2228 @@ +//! OTLP export of the audit stream and of the server's own diagnostics +//! (issue #31). +//! +//! Revised 2026-08-18: a configured collector is a LOAD-BEARING audit +//! sink, not a best-effort copy. Delivery is proven at startup +//! ([`Pipeline::probe`], bounded retry, refusing to serve on failure) and +//! watched while serving: a delivery failure or a refused record marks +//! the audit sink failing ([`AuditExport::delivery_failing`]), which feeds +//! the same [`FailMode`] gate a failed file write does, until a delivery +//! succeeds again. When a file is also configured, the record still +//! reaches it FIRST and the exporter only afterwards. +//! +//! Only the DIAGNOSTICS stream stays best-effort: a dropped log line is +//! counted and never halts the guard. Neither stream changes what a +//! client sees on a served call (I15) — the gate refusals are the +//! audit machinery's own, uniform per tool. +//! +//! # What is exported +//! +//! Two streams, tagged apart by the `bugwarden.stream` attribute: +//! +//! - `audit` — one OTel log record per [`AuditEvent`] the sink persisted. +//! The record body is the audit line VERBATIM — when a file is +//! configured, the same bytes it carries minus the newline, so the +//! exported payload and the file record are byte-equal (I12: nothing +//! extra is exported, and nothing is exported that the file does not +//! hold); a fileless (OTLP-only) sink exports the line it would have +//! written. +//! - `diagnostics` — the server's ordinary `tracing` output, the same +//! events the stderr layer formats, under the same `RUST_LOG` filter. +//! +//! # Transport +//! +//! OTLP/HTTP with protobuf payloads, posted with the workspace's existing +//! reqwest/rustls stack. The protobuf encoder below is hand-written +//! against the OTLP logs schema; the alternative — the OpenTelemetry SDK — +//! was measured and rejected, see `docs/DESIGN.md`. +//! +//! # Secrets (I12) +//! +//! [`OTEL_EXPORTER_OTLP_HEADERS`] carries whatever credential the +//! collector wants, so it is treated exactly like the http bearer tokens: +//! read from the environment only, never a command-line option, and no +//! type that holds it derives `Debug`. The resolved endpoint is weaker: +//! this crate never writes it to a log line, an error, or an audit +//! record, but at `RUST_LOG=debug` the HTTP stack may print the +//! authority. That is why the drop diagnostic carries a count and a +//! closed-vocabulary reason and nothing else, and why a failed request's +//! `reqwest::Error` — which would carry the URL — is discarded rather +//! than logged. +//! +//! [`AuditEvent`]: crate::audit::AuditEvent +//! [`AuditExport::delivery_failing`]: crate::audit::AuditExport::delivery_failing +//! [`FailMode`]: crate::audit::FailMode +//! [`OTEL_EXPORTER_OTLP_HEADERS`]: HEADERS_VAR + +use std::fmt; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use tokio::sync::{mpsc, oneshot}; +use tokio_util::sync::CancellationToken; +use tracing_subscriber::layer::Context as LayerContext; +use tracing_subscriber::Layer; + +use crate::audit::{AuditEvent, AuditEventKind, AuditExport, ExportRefused}; + +/// Collector base URL; [`LOGS_PATH`] is appended to it. Unset or empty — +/// with [`LOGS_ENDPOINT_VAR`] unset or empty too — turns the whole feature +/// off. +pub const ENDPOINT_VAR: &str = "OTEL_EXPORTER_OTLP_ENDPOINT"; + +/// Comma-separated `key=value` headers added to every export request. +/// Secret material (I12). +pub const HEADERS_VAR: &str = "OTEL_EXPORTER_OTLP_HEADERS"; + +/// OTLP transport selector; this build speaks [`PROTOCOL_HTTP_PROTOBUF`] +/// and rejects every other value at startup. +pub const PROTOCOL_VAR: &str = "OTEL_EXPORTER_OTLP_PROTOCOL"; + +/// `service.name` on the exported resource; defaults to `bugwarden`. +pub const SERVICE_NAME_VAR: &str = "OTEL_SERVICE_NAME"; + +/// Logs-specific endpoint. Per the OTLP specification it OVERRIDES +/// [`ENDPOINT_VAR`] and is used as given — the signal path is NOT appended, +/// because the operator wrote the whole URL. +pub const LOGS_ENDPOINT_VAR: &str = "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"; + +/// Logs-specific headers; overrides [`HEADERS_VAR`]. Secret material (I12). +pub const LOGS_HEADERS_VAR: &str = "OTEL_EXPORTER_OTLP_LOGS_HEADERS"; + +/// Logs-specific protocol; overrides [`PROTOCOL_VAR`]. +pub const LOGS_PROTOCOL_VAR: &str = "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL"; + +/// Every environment variable this module reads. +/// +/// These are read by the module, not by `Cli`, so clap's own +/// `get_env` sweep cannot see them. `tests/binary_user_agent.rs` holds its +/// environment scrub list to this list as well, so a variable added here +/// cannot go on reaching a spawned test binary from the developer's shell. +pub const ENV_VARS: [&str; 7] = [ + ENDPOINT_VAR, + HEADERS_VAR, + PROTOCOL_VAR, + SERVICE_NAME_VAR, + LOGS_ENDPOINT_VAR, + LOGS_HEADERS_VAR, + LOGS_PROTOCOL_VAR, +]; + +/// The one OTLP transport this build speaks. +pub const PROTOCOL_HTTP_PROTOBUF: &str = "http/protobuf"; + +/// `service.name` when [`SERVICE_NAME_VAR`] says nothing. +const DEFAULT_SERVICE_NAME: &str = "bugwarden"; + +/// Signal path appended to the configured base endpoint, per the OTLP +/// specification for `OTEL_EXPORTER_OTLP_ENDPOINT`. +const LOGS_PATH: &str = "/v1/logs"; + +/// Records the queue holds before it starts dropping. Bounded on purpose: +/// an unreachable collector must cost a bounded amount of memory and a +/// counter, never unbounded growth behind a socket nobody is reading. +const QUEUE_CAPACITY: usize = 2048; + +/// Most records in one export request. +const MAX_BATCH: usize = 512; + +/// How long the exporter waits for a batch to fill before sending what it +/// has. +const BATCH_INTERVAL: Duration = Duration::from_millis(500); + +/// Bound on a single export request, so a collector that accepts a +/// connection and then stalls cannot pin the exporter task forever. +const EXPORT_TIMEOUT: Duration = Duration::from_secs(10); + +/// Bound on the best-effort flush at shutdown. Long enough for a live +/// collector to take the tail of the queue, short enough that a dead one +/// does not hold the process open. +pub const SHUTDOWN_FLUSH_TIMEOUT: Duration = Duration::from_secs(5); + +/// Startup probe attempts before the deployment refuses to start +/// ([`Pipeline::probe`]). Bounded retry, not one shot: a collector and a +/// server that start together race, and losing that race by half a second +/// is not a misconfiguration. +const PROBE_ATTEMPTS: u32 = 5; + +/// Base backoff between probe attempts; attempt `n` waits `n` times this, +/// so the five attempts span about five seconds of sleep (each attempt +/// itself bounded by [`EXPORT_TIMEOUT`]). +const PROBE_BACKOFF: Duration = Duration::from_millis(500); + +/// This module's own tracing target — the first entry of +/// [`NEVER_EXPORTED_TARGETS`], named once so the probe record and the +/// skip list cannot drift apart. +const SELF_TARGET: &str = "bugwarden::otel"; + +/// Target prefixes whose events are never exported. +/// +/// Exporting these would make the export its own input. This module's own +/// drop warning is the obvious case, but the expensive one is the HTTP +/// stack underneath: at `RUST_LOG=debug` a single flush makes +/// `hyper_util`'s connection pool log "pooling idle connection for +/// " and `reqwest::connect` log "starting new connection", each +/// of which would become a record in the NEXT batch, whose flush logs +/// again — a loop that sustains itself at one export per batch interval +/// forever, on an idle server, and puts the collector authority on the +/// wire as a side effect. +/// +/// Matched as prefixes against the event target, which for the log-crate +/// events `tracing-log` forwards is the emitting module path +/// (`hyper_util::client::legacy::pool`, `reqwest::connect`). `"hyper"` +/// therefore covers `hyper_util` as well. +/// +/// The cost is real and accepted: these targets also carry the BUGZILLA +/// client's HTTP diagnostics, and those stop being exported too. They +/// still reach stderr, which is where a debugging operator reads them, and +/// no filter that can tell the two clients apart exists at the layer — +/// both are the same crates on the same targets. +const NEVER_EXPORTED_TARGETS: [&str; 6] = [ + SELF_TARGET, + "reqwest", + // Covers `hyper_util` too. + "hyper", + "rustls", + "h2", + "tower", +]; + +/// Whether an event's target is one the export must never carry. +fn never_exported(target: &str) -> bool { + NEVER_EXPORTED_TARGETS + .iter() + .any(|prefix| target.starts_with(prefix)) +} + +/// Instrumentation scope name on every exported record. +const SCOPE_NAME: &str = "bugwarden"; + +// --------------------------------------------------------------------------- +// Configuration +// --------------------------------------------------------------------------- + +/// What the four OTLP variables held when the process started. +/// +/// A plain struct rather than a direct read of `std::env`, so resolution is +/// a function of its argument and a test states its own world instead of +/// mutating the process. No `Debug`: [`OtelEnv::headers`] is secret (I12). +#[derive(Default, Clone)] +pub struct OtelEnv { + /// [`ENDPOINT_VAR`], if it held anything. + pub endpoint: Option, + /// [`HEADERS_VAR`], if it held anything. Secret. + pub headers: Option, + /// [`PROTOCOL_VAR`], if it held anything. + pub protocol: Option, + /// [`SERVICE_NAME_VAR`], if it held anything. + pub service_name: Option, + /// [`LOGS_ENDPOINT_VAR`], if it held anything. Overrides `endpoint`. + pub logs_endpoint: Option, + /// [`LOGS_HEADERS_VAR`], if it held anything. Overrides `headers`. + /// Secret. + pub logs_headers: Option, + /// [`LOGS_PROTOCOL_VAR`], if it held anything. Overrides `protocol`. + pub logs_protocol: Option, +} + +impl OtelEnv { + /// Read the four variables out of the process environment. + /// + /// A variable set to the empty string counts as one that was never set, + /// the same "cleared" idiom unit files and container specs use for + /// `BUGZILLA_API_KEY_FILE` and `MCP_ALLOWED_HOSTS`. For + /// [`ENDPOINT_VAR`] that idiom is the off switch. + #[must_use] + pub fn from_env() -> Self { + Self { + endpoint: env_value(ENDPOINT_VAR), + headers: env_value(HEADERS_VAR), + protocol: env_value(PROTOCOL_VAR), + service_name: env_value(SERVICE_NAME_VAR), + logs_endpoint: env_value(LOGS_ENDPOINT_VAR), + logs_headers: env_value(LOGS_HEADERS_VAR), + logs_protocol: env_value(LOGS_PROTOCOL_VAR), + } + } +} + +/// One environment variable, with the empty string read as absence. +fn env_value(var: &str) -> Option { + match std::env::var(var) { + Ok(value) if !value.is_empty() => Some(value), + _ => None, + } +} + +/// A resolved, usable export configuration. +/// +/// No `Debug` and no accessor for the endpoint or the headers: the only +/// code that may see either is the exporter task that puts them on the +/// wire (I12). +#[derive(Clone)] +pub struct ExportConfig { + /// Full URL of the logs endpoint, base plus [`LOGS_PATH`]. + logs_url: String, + /// Headers added to every request. Secret. + headers: Vec<(String, String)>, + /// `service.name` for the exported resource. + service_name: String, + /// Which endpoint variable actually won. Named in probe refusals + /// (I12: the name, never the URL). + endpoint_var: &'static str, + /// Which headers variable actually won. Named in probe refusals. + headers_var: &'static str, +} + +impl ExportConfig { + /// The `service.name` records are exported under. Safe to log — it is + /// operator-chosen labelling, not credential material. + #[must_use] + pub fn service_name(&self) -> &str { + &self.service_name + } + + /// The resolved logs URL. Crate-visible for this module's own tests; + /// nothing in the request path may print it (I12). + #[cfg(test)] + pub(crate) fn logs_url(&self) -> &str { + &self.logs_url + } + + /// The resolved headers. Crate-visible for this module's own tests. + #[cfg(test)] + pub(crate) fn headers(&self) -> &[(String, String)] { + &self.headers + } +} + +/// Resolve the environment into an export configuration, or into `None` +/// when export is off. +/// +/// Off is the default and the whole feature: with no endpoint set — from +/// either [`ENDPOINT_VAR`] or [`LOGS_ENDPOINT_VAR`], and an emptied +/// variable counts as unset — this returns `None`, no exporter task is +/// started, no diagnostics layer is installed, and the process behaves +/// exactly as a build without this module. That is also why the protocol +/// is validated only once an endpoint exists: a fleet-wide +/// `OTEL_EXPORTER_OTLP_*` environment must not refuse to start a +/// deployment that exports nothing. +/// +/// The three logs-specific variables override their general counterparts, +/// as the OTLP specification requires. [`LOGS_ENDPOINT_VAR`] is used +/// exactly as given — [`LOGS_PATH`] is appended only to [`ENDPOINT_VAR`], +/// because the signal-specific form is the operator's whole URL. +/// +/// # Errors +/// +/// - A protocol naming anything but [`PROTOCOL_HTTP_PROTOBUF`]. +/// - An endpoint that is not an `http://` or `https://` URL. +/// - A header entry that is not a usable `key=value` HTTP header. +/// +/// Every message names the variable that carried the offending value — +/// the specific one where it was the specific one that lost — and, for a +/// header list, the position of the offending entry, never a value: a +/// mispasted credential is exactly what lands in the wrong position +/// (I12). +pub fn resolve(env: &OtelEnv) -> anyhow::Result> { + // The logs-specific variable wins where it is set, and is used as the + // operator wrote it — the signal path is appended only to the general + // one. Honouring it is not optional decoration: a fleet that sets only + // `_LOGS_ENDPOINT` is a fleet that expects logs to be exported, and + // reading just the general variable would leave export silently off. + let (endpoint_var, endpoint, append_path) = match logs_override(env.logs_endpoint.as_deref()) { + Some(endpoint) => (LOGS_ENDPOINT_VAR, endpoint, false), + None => ( + ENDPOINT_VAR, + env.endpoint.as_deref().unwrap_or_default().trim(), + true, + ), + }; + if endpoint.is_empty() { + return Ok(None); + } + let (protocol_var, protocol) = match logs_override(env.logs_protocol.as_deref()) { + Some(protocol) => (LOGS_PROTOCOL_VAR, Some(protocol)), + None => (PROTOCOL_VAR, env.protocol.as_deref().map(str::trim)), + }; + match protocol { + None | Some(PROTOCOL_HTTP_PROTOBUF) => {} + Some(_) => anyhow::bail!( + "{protocol_var} selects an OTLP transport this build does not speak; \ + only \"{PROTOCOL_HTTP_PROTOBUF}\" is supported" + ), + } + if !(endpoint.starts_with("http://") || endpoint.starts_with("https://")) { + anyhow::bail!("{endpoint_var} must be an http:// or https:// URL"); + } + let (headers_var, raw_headers) = match logs_override(env.logs_headers.as_deref()) { + Some(raw) => (LOGS_HEADERS_VAR, Some(raw)), + None => (HEADERS_VAR, env.headers.as_deref()), + }; + let headers = match raw_headers { + Some(raw) => parse_headers(headers_var, raw)?, + None => Vec::new(), + }; + let service_name = match env.service_name.as_deref().map(str::trim) { + Some(name) if !name.is_empty() => name.to_owned(), + _ => DEFAULT_SERVICE_NAME.to_owned(), + }; + let logs_url = if append_path { + format!("{}{LOGS_PATH}", endpoint.trim_end_matches('/')) + } else { + endpoint.to_owned() + }; + Ok(Some(ExportConfig { + logs_url, + headers, + service_name, + endpoint_var, + headers_var, + })) +} + +/// A signal-specific variable's value, if it holds one. An emptied +/// variable is an unset one here as everywhere, so it falls back to the +/// general variable rather than turning the export off on its own. +fn logs_override(value: Option<&str>) -> Option<&str> { + match value.map(str::trim) { + Some(value) if !value.is_empty() => Some(value), + _ => None, + } +} + +/// Parse `key=value,key=value` into headers, naming `var` in any refusal. +/// +/// Empty entries are skipped, so a trailing or doubled comma is not an +/// error — `a=1,` and `a=1,,b=2` parse as the pairs they obviously mean. +/// +/// Values are taken VERBATIM. The OTLP specification describes the list as +/// percent-encoded; decoding it here would rewrite any credential +/// containing a `%`, and silently mangling a secret is worse than not +/// implementing an encoding nobody's collector requires. The deviation is +/// deliberate and documented in DESIGN.md. +fn parse_headers(var: &str, raw: &str) -> anyhow::Result> { + let mut headers = Vec::new(); + for (index, entry) in raw.split(',').enumerate() { + let position = index + 1; + let entry = entry.trim(); + if entry.is_empty() { + continue; + } + let Some((key, value)) = entry.split_once('=') else { + anyhow::bail!("{var} entry {position} is not `key=value`"); + }; + let key = key.trim(); + let value = value.trim(); + if key.is_empty() || !key.bytes().all(is_header_token) { + anyhow::bail!("{var} entry {position} has no usable header name"); + } + if value.is_empty() + || !value + .bytes() + .all(|b| b == b'\t' || (0x20..=0x7e).contains(&b)) + { + anyhow::bail!("{var} entry {position} has no usable header value"); + } + headers.push((key.to_owned(), value.to_owned())); + } + Ok(headers) +} + +/// RFC 9110 `token` characters, the alphabet of a header field name. +fn is_header_token(b: u8) -> bool { + b.is_ascii_alphanumeric() || b"!#$%&'*+-.^_`|~".contains(&b) +} + +// --------------------------------------------------------------------------- +// Records +// --------------------------------------------------------------------------- + +/// OTLP severity numbers, at the base of each severity range. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Severity { + Trace = 1, + Debug = 5, + Info = 9, + Warn = 13, + Error = 17, +} + +impl Severity { + /// The `severity_text` accompanying the number. + fn text(self) -> &'static str { + match self { + Severity::Trace => "TRACE", + Severity::Debug => "DEBUG", + Severity::Info => "INFO", + Severity::Warn => "WARN", + Severity::Error => "ERROR", + } + } + + fn of_level(level: &tracing::Level) -> Severity { + match *level { + tracing::Level::TRACE => Severity::Trace, + tracing::Level::DEBUG => Severity::Debug, + tracing::Level::INFO => Severity::Info, + tracing::Level::WARN => Severity::Warn, + tracing::Level::ERROR => Severity::Error, + } + } +} + +/// An attribute value; the two shapes the exported streams need. +#[derive(Debug, Clone, PartialEq, Eq)] +enum AttrValue { + Str(String), + Int(i64), +} + +/// One OTel log record, queued for export. +#[derive(Debug, Clone, PartialEq, Eq)] +struct LogEntry { + time_unix_nano: u64, + severity: Severity, + body: String, + attrs: Vec<(&'static str, AttrValue)>, + trace: Option<([u8; 16], [u8; 8])>, +} + +/// Wall clock as OTLP wants it. +fn now_unix_nano() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |d| u64::try_from(d.as_nanos()).unwrap_or(u64::MAX)) +} + +/// Build the log record for one persisted audit event. +/// +/// `line` is the record as the file holds it, without the terminating +/// newline; it becomes the record body unchanged (I12: the exported +/// payload carries exactly what the file carries). +fn audit_entry(event: &AuditEvent, line: &[u8]) -> LogEntry { + let mut attrs: Vec<(&'static str, AttrValue)> = Vec::with_capacity(8); + attrs.push(("bugwarden.stream", AttrValue::Str("audit".to_owned()))); + let (kind, severity) = match &event.kind { + AuditEventKind::ToolCall(_) => ("tool_call", Severity::Info), + AuditEventKind::Initialize(_) => ("initialize", Severity::Info), + // The one kind that reports a loss of the record stream itself. + AuditEventKind::AuditGap(_) => ("audit_gap", Severity::Error), + }; + attrs.push(("bugwarden.event", AttrValue::Str(kind.to_owned()))); + attrs.push(( + "bugwarden.seq", + AttrValue::Int(i64::try_from(event.seq).unwrap_or(i64::MAX)), + )); + attrs.push(( + "bugwarden.transport", + AttrValue::Str( + match event.session.transport { + crate::audit::TransportKind::Stdio => "stdio", + crate::audit::TransportKind::Http => "http", + } + .to_owned(), + ), + )); + if let Some(id) = &event.session.id { + attrs.push(("bugwarden.session.id", AttrValue::Str(id.clone()))); + } + let mut trace = None; + if let AuditEventKind::ToolCall(call) = &event.kind { + attrs.push(("bugwarden.tool", AttrValue::Str(call.request.tool.clone()))); + if let Some(guard) = &call.guard { + attrs.push(( + "bugwarden.verdict", + AttrValue::Str(verdict_name(guard.verdict).to_owned()), + )); + if let Some(rule) = &guard.rule { + attrs.push(("bugwarden.rule", AttrValue::Str(rule.clone()))); + } + } + // Correlation ids the client claimed; unauthenticated, exactly as + // the audit record documents them. Anything that does not decode + // to the two fixed widths is left off rather than exported wrong. + if let Some(ctx) = &call.trace { + if let (Some(t), Some(s)) = + (hex_bytes::<16>(&ctx.trace_id), hex_bytes::<8>(&ctx.span_id)) + { + trace = Some((t, s)); + } + } + } + LogEntry { + time_unix_nano: now_unix_nano(), + severity, + body: String::from_utf8_lossy(line).into_owned(), + attrs, + trace, + } +} + +/// Wire spelling of a verdict, matching the audit schema's own. +fn verdict_name(verdict: crate::audit::Verdict) -> &'static str { + match verdict { + crate::audit::Verdict::Served => "served", + crate::audit::Verdict::ServedFiltered => "served_filtered", + crate::audit::Verdict::Denied => "denied", + crate::audit::Verdict::Refused => "refused", + } +} + +/// Decode exactly `N` bytes of lowercase or uppercase hex, or nothing. +fn hex_bytes(value: &str) -> Option<[u8; N]> { + let bytes = value.as_bytes(); + if bytes.len() != N * 2 { + return None; + } + let mut out = [0u8; N]; + for (i, slot) in out.iter_mut().enumerate() { + let hi = (bytes[i * 2] as char).to_digit(16)?; + let lo = (bytes[i * 2 + 1] as char).to_digit(16)?; + *slot = u8::try_from(hi * 16 + lo).ok()?; + } + Some(out) +} + +// --------------------------------------------------------------------------- +// Protobuf encoding (OTLP logs) +// --------------------------------------------------------------------------- + +/// Minimal protobuf writer for the handful of OTLP logs messages this +/// module emits. Field numbers come from +/// `opentelemetry/proto/logs/v1/logs.proto` and +/// `opentelemetry/proto/collector/logs/v1/logs_service.proto`. +mod wire { + /// Length-delimited wire type. + pub(super) const LEN: u32 = 2; + /// 64-bit fixed wire type. + pub(super) const I64: u32 = 1; + /// Varint wire type. + pub(super) const VARINT: u32 = 0; + + pub(super) fn put_varint(buf: &mut Vec, mut value: u64) { + loop { + let byte = u8::try_from(value & 0x7f).unwrap_or(0); + value >>= 7; + if value == 0 { + buf.push(byte); + return; + } + buf.push(byte | 0x80); + } + } + + pub(super) fn put_tag(buf: &mut Vec, field: u32, wire_type: u32) { + put_varint(buf, u64::from(field << 3 | wire_type)); + } + + pub(super) fn put_bytes(buf: &mut Vec, field: u32, value: &[u8]) { + put_tag(buf, field, LEN); + put_varint(buf, value.len() as u64); + buf.extend_from_slice(value); + } + + pub(super) fn put_str(buf: &mut Vec, field: u32, value: &str) { + put_bytes(buf, field, value.as_bytes()); + } + + pub(super) fn put_varint_field(buf: &mut Vec, field: u32, value: u64) { + put_tag(buf, field, VARINT); + put_varint(buf, value); + } + + pub(super) fn put_fixed64(buf: &mut Vec, field: u32, value: u64) { + put_tag(buf, field, I64); + buf.extend_from_slice(&value.to_le_bytes()); + } +} + +/// `AnyValue { string_value = 1 }`. +fn encode_any_string(value: &str) -> Vec { + let mut buf = Vec::with_capacity(value.len() + 8); + wire::put_str(&mut buf, 1, value); + buf +} + +/// `AnyValue { int_value = 3 }`. +fn encode_any_int(value: i64) -> Vec { + let mut buf = Vec::with_capacity(12); + wire::put_varint_field(&mut buf, 3, value as u64); + buf +} + +/// `KeyValue { key = 1, value = 2 }`. +fn encode_kv(key: &str, value: &AttrValue) -> Vec { + let encoded = match value { + AttrValue::Str(s) => encode_any_string(s), + AttrValue::Int(i) => encode_any_int(*i), + }; + let mut buf = Vec::with_capacity(key.len() + encoded.len() + 8); + wire::put_str(&mut buf, 1, key); + wire::put_bytes(&mut buf, 2, &encoded); + buf +} + +/// `LogRecord`. +fn encode_log_record(entry: &LogEntry) -> Vec { + let mut buf = Vec::with_capacity(entry.body.len() + 128); + wire::put_fixed64(&mut buf, 1, entry.time_unix_nano); + wire::put_varint_field(&mut buf, 2, entry.severity as u64); + wire::put_str(&mut buf, 3, entry.severity.text()); + wire::put_bytes(&mut buf, 5, &encode_any_string(&entry.body)); + for (key, value) in &entry.attrs { + wire::put_bytes(&mut buf, 6, &encode_kv(key, value)); + } + if let Some((trace_id, span_id)) = &entry.trace { + wire::put_bytes(&mut buf, 9, trace_id); + wire::put_bytes(&mut buf, 10, span_id); + } + wire::put_fixed64(&mut buf, 11, entry.time_unix_nano); + buf +} + +/// A whole `ExportLogsServiceRequest` for one batch. +fn encode_request(service_name: &str, entries: &[LogEntry]) -> Vec { + let mut scope = Vec::new(); + { + let mut inner = Vec::new(); + wire::put_str(&mut inner, 1, SCOPE_NAME); + wire::put_str(&mut inner, 2, env!("CARGO_PKG_VERSION")); + wire::put_bytes(&mut scope, 1, &inner); + } + for entry in entries { + wire::put_bytes(&mut scope, 2, &encode_log_record(entry)); + } + + let mut resource = Vec::new(); + wire::put_bytes( + &mut resource, + 1, + &encode_kv("service.name", &AttrValue::Str(service_name.to_owned())), + ); + + let mut resource_logs = Vec::new(); + wire::put_bytes(&mut resource_logs, 1, &resource); + wire::put_bytes(&mut resource_logs, 2, &scope); + + let mut request = Vec::new(); + wire::put_bytes(&mut request, 1, &resource_logs); + request +} + +// --------------------------------------------------------------------------- +// The exporter +// --------------------------------------------------------------------------- + +/// Why records were dropped. A closed vocabulary, for the same reason the +/// audit schema's [`GapReason`] is one: a free-text reason built from a +/// transport error is how an endpoint gets into a log line (I12). +/// +/// [`GapReason`]: crate::audit::GapReason +const REASON_QUEUE_FULL: &str = "queue_full"; +const REASON_NETWORK: &str = "network"; +const REASON_HTTP_STATUS: &str = "http_status"; +/// The exporter has shut down and the queue is closed; a record offered +/// after that is lost for a different reason than a full queue, and saying +/// so keeps the vocabulary honest. +const REASON_SHUTDOWN: &str = "shutdown"; + +/// The running export pipeline: a bounded queue, one task draining it, and +/// the drop accounting. +/// +/// Created only when [`resolve`] returned a configuration. `main` starts +/// it after the identity preflight and probes it BEFORE the audit file +/// is created, so a collector that will not take records refuses without +/// leaving a file behind. +pub struct Pipeline { + /// Audit records. Load-bearing: a record this queue will not take is + /// REFUSED, never dropped, so the refusal reaches the fail-mode gate. + audit_tx: mpsc::Sender, + /// Diagnostics. Best-effort and deliberately a separate queue: a log + /// storm must not fill the audit queue and take the server down, and + /// a dropped diagnostic must not stop the guard. + diag_tx: mpsc::Sender, + /// The client the drain task and the startup probe share. + client: reqwest::Client, + cfg: ExportConfig, + /// DIAGNOSTICS dropped. Audit records are never counted here — they + /// are never dropped in the first place. + dropped: Arc, + /// Next drop total that earns a log line; doubles each time, so the + /// diagnostic appears at 1, 2, 4, 8 … drops and not once per record. + log_threshold: Arc, + /// Audit records accepted and then not delivered, awaiting the + /// `audit_gap` that reports them. Read and reset by the sink. + lost: Arc, + /// Whether delivery is known to work: false from a failed request + /// until one succeeds. This, not the counter, is what holds the gate + /// closed for a whole outage. + healthy: Arc, + shutdown: CancellationToken, + /// Taken once, by [`Pipeline::shutdown`]. + finished: Mutex>>, +} + +impl fmt::Debug for Pipeline { + /// Hand-written and deliberately content-free: the pipeline owns the + /// endpoint and the export headers, and it is reachable from + /// [`AuditSink`]'s derived `Debug` (I12). + /// + /// [`AuditSink`]: crate::audit::AuditSink + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Pipeline") + .field("dropped", &self.dropped.load(Ordering::Relaxed)) + .field("healthy", &self.healthy.load(Ordering::Relaxed)) + .finish_non_exhaustive() + } +} + +impl Pipeline { + /// Start the exporter: spawn the drain task and return its handle. + /// + /// Must be called from within a Tokio runtime. Nothing is delivered + /// yet and delivery is not yet known to work — [`Pipeline::probe`] + /// decides that, before the server serves anything. + /// + /// # Errors + /// + /// The HTTP client could not be built. Reported here rather than + /// swallowed in the task, because an exporter that can never deliver + /// must refuse startup, not run. + pub fn start(cfg: ExportConfig) -> anyhow::Result { + let client = reqwest::Client::builder() + .timeout(EXPORT_TIMEOUT) + // A 3xx would forward the audit body and any non-Authorization + // collector credential to a host the operator did not name. + .redirect(reqwest::redirect::Policy::none()) + .build() + // The error is dropped rather than reported: a reqwest builder + // error can name the proxy URL it choked on (I12). + .map_err(|_| anyhow::anyhow!("the OTLP export HTTP client could not be built"))?; + let (audit_tx, audit_rx) = mpsc::channel(QUEUE_CAPACITY); + let (diag_tx, diag_rx) = mpsc::channel(QUEUE_CAPACITY); + let dropped = Arc::new(AtomicU64::new(0)); + let log_threshold = Arc::new(AtomicU64::new(1)); + let lost = Arc::new(AtomicU64::new(0)); + let healthy = Arc::new(AtomicBool::new(true)); + let shutdown = CancellationToken::new(); + let (done_tx, done_rx) = oneshot::channel(); + let task = ExportTask { + cfg: cfg.clone(), + client: client.clone(), + audit_rx, + diag_rx, + dropped: dropped.clone(), + log_threshold: log_threshold.clone(), + lost: lost.clone(), + healthy: healthy.clone(), + shutdown: shutdown.clone(), + done: done_tx, + }; + tokio::spawn(task.run()); + Ok(Pipeline { + audit_tx, + diag_tx, + client, + cfg, + dropped, + log_threshold, + lost, + healthy, + shutdown, + finished: Mutex::new(Some(done_rx)), + }) + } + + /// Prove the collector takes records, before the server serves any. + /// + /// The same philosophy as the identity preflight: a dependency this + /// deployment cannot run without is checked while a startup failure + /// is still a startup failure, rather than discovered as a wave of + /// refusals under load. One real diagnostics record is posted — not + /// an empty request, which some collectors accept without looking, and + /// not an audit record, which would mean inventing an event kind the + /// schema does not have — so this exercises the whole path: DNS, TCP, + /// TLS, the headers, the protocol and the collector's own acceptance. + /// + /// Bounded retry, because a collector and a server that start together + /// race, and losing that race by half a second is not a + /// misconfiguration. + /// + /// # Errors + /// + /// Every attempt failed. The message names no endpoint and no header + /// (I12) — the operator knows what they configured; what they need to + /// be told is that it does not answer. + pub async fn probe(&self) -> anyhow::Result<()> { + let entry = LogEntry { + time_unix_nano: now_unix_nano(), + severity: Severity::Info, + body: format!( + "bugwarden {} starting: audit records are exported to this collector \ + and the server refuses to serve while they cannot be delivered", + env!("CARGO_PKG_VERSION") + ), + attrs: vec![ + ("bugwarden.stream", AttrValue::Str("diagnostics".to_owned())), + ("log.target", AttrValue::Str(SELF_TARGET.to_owned())), + ], + trace: None, + }; + let batch = std::slice::from_ref(&entry); + let mut attempt = 1; + loop { + if post_batch(&self.client, &self.cfg, batch).await.is_ok() { + self.healthy.store(true, Ordering::Relaxed); + return Ok(()); + } + if attempt >= PROBE_ATTEMPTS { + self.healthy.store(false, Ordering::Relaxed); + anyhow::bail!( + "the OTLP collector did not accept the startup record after \ + {PROBE_ATTEMPTS} attempts; audit records are exported to it and \ + this deployment refuses to serve without them — check that the \ + endpoint in {} is reachable and that any credential \ + in {} is the one it expects", + self.cfg.endpoint_var, + self.cfg.headers_var, + ); + } + tokio::time::sleep(PROBE_BACKOFF * attempt).await; + attempt += 1; + } + } + + /// How many DIAGNOSTICS records this pipeline has dropped. Audit + /// records never appear here: they are refused or delivered, and a + /// refusal is the sink's to account for. + #[must_use] + pub fn dropped(&self) -> u64 { + self.dropped.load(Ordering::Relaxed) + } + + /// Whether delivery is currently known to work. False from a failed + /// request until one succeeds. + #[must_use] + pub fn healthy(&self) -> bool { + self.healthy.load(Ordering::Relaxed) + } + + /// This pipeline as one of the audit sink's destinations. + #[must_use] + pub fn audit_exporter(self: &Arc) -> Arc { + self.clone() + } + + /// A tracing layer feeding this pipeline the server's diagnostics. + #[must_use] + pub fn diagnostics_layer(slot: Arc>>) -> DiagnosticsLayer { + DiagnosticsLayer { slot } + } + + /// Stop accepting records, flush what is queued, and return. + /// + /// Bounded by [`SHUTDOWN_FLUSH_TIMEOUT`]: a collector that has gone + /// away costs the timeout once, not the process. Anything still + /// queued when that expires is lost — see the module docs on what a + /// fileless deployment does and does not durably keep. + pub async fn shutdown(&self) { + self.shutdown.cancel(); + let finished = self + .finished + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .take(); + if let Some(rx) = finished { + let _ = tokio::time::timeout(SHUTDOWN_FLUSH_TIMEOUT, rx).await; + } + } + + /// Queue one diagnostic, or account for the drop. Never blocks. + fn emit(&self, entry: LogEntry) { + let reason = match self.diag_tx.try_send(entry) { + Ok(()) => return, + Err(mpsc::error::TrySendError::Full(_)) => REASON_QUEUE_FULL, + Err(mpsc::error::TrySendError::Closed(_)) => REASON_SHUTDOWN, + }; + note_drops(&self.dropped, &self.log_threshold, 1, reason); + } +} + +impl AuditExport for Pipeline { + fn accept(&self, event: &AuditEvent, line: &[u8]) -> Result<(), ExportRefused> { + match self.audit_tx.try_send(audit_entry(event, line)) { + Ok(()) => Ok(()), + Err(_) => { + // Refused, not dropped. The sink turns this into a failed + // record and the fail mode decides what the caller sees; + // marking delivery unhealthy keeps the gate shut until a + // request actually succeeds, rather than reopening it the + // moment one queue slot frees up. + self.healthy.store(false, Ordering::Relaxed); + Err(ExportRefused) + } + } + } + + fn delivery_failing(&self) -> bool { + !self.healthy.load(Ordering::Relaxed) + } + + fn take_lost(&self) -> u64 { + self.lost.swap(0, Ordering::Relaxed) + } +} + +/// Add `count` drops to the running total and log if the total crossed a +/// power of two. +/// +/// The line carries the count and a closed-vocabulary reason and nothing +/// else — no endpoint, no header, no transport error (I12). +fn note_drops(dropped: &AtomicU64, threshold: &AtomicU64, count: u64, reason: &'static str) { + if count == 0 { + return; + } + let total = dropped.fetch_add(count, Ordering::Relaxed) + count; + let current = threshold.load(Ordering::Relaxed); + if total < current { + return; + } + let next = if total.is_power_of_two() { + total.saturating_mul(2) + } else { + total.checked_next_power_of_two().unwrap_or(u64::MAX) + }; + // A lost race means another reporter is logging this crossing; one + // line per crossing is the point. + if threshold + .compare_exchange(current, next, Ordering::Relaxed, Ordering::Relaxed) + .is_ok() + { + tracing::warn!( + dropped = total, + reason, + "otlp export is dropping diagnostic records" + ); + } +} + +/// Post one batch of entries: the one request path, shared by the drain +/// task and the startup probe so the probe proves exactly what delivery +/// uses — the client, the URL, the headers and the encoding. +/// +/// # Errors +/// +/// The closed-vocabulary reason. The transport error itself is +/// deliberately dropped rather than returned or logged: it carries the +/// request URL, and the endpoint may not reach a log line (I12, the same +/// rule `.without_url()` exists for). +async fn post_batch( + client: &reqwest::Client, + cfg: &ExportConfig, + batch: &[LogEntry], +) -> Result<(), &'static str> { + let body = encode_request(&cfg.service_name, batch); + let mut request = client + .post(&cfg.logs_url) + .header("content-type", "application/x-protobuf") + .body(body); + for (name, value) in &cfg.headers { + request = request.header(name.as_str(), value.as_str()); + } + match request.send().await { + Ok(response) if response.status().is_success() => Ok(()), + Ok(_) => Err(REASON_HTTP_STATUS), + Err(_) => Err(REASON_NETWORK), + } +} + +/// Record one delivery attempt's outcome in the health flag the fail-mode +/// gate reads, logging TRANSITIONS only: one line per outage, one per +/// recovery, never one per batch. Audit success opens the latch; either +/// stream's failure closes it; a successful diagnostics flush is ignored. +fn note_delivery(healthy: &AtomicBool, ok: bool) { + let was = healthy.swap(ok, Ordering::Relaxed); + if was && !ok { + tracing::warn!( + "otlp delivery is failing; the audit fail mode now gates tool calls \ + until a delivery succeeds" + ); + } else if !was && ok { + tracing::info!("otlp delivery recovered"); + } +} + +/// The drain task's own state. +struct ExportTask { + cfg: ExportConfig, + /// Built by [`Pipeline::start`] and shared with the probe. + client: reqwest::Client, + /// Audit records: refused at the queue rather than dropped, and + /// counted in `lost` when a send fails after acceptance. + audit_rx: mpsc::Receiver, + /// Diagnostics: dropped and counted when the queue is full or a send + /// fails. + diag_rx: mpsc::Receiver, + dropped: Arc, + log_threshold: Arc, + /// Audit records accepted and then not delivered; drained into the + /// sink's gap accounting through [`AuditExport::take_lost`]. + lost: Arc, + /// Whether the last delivery attempt worked; read by the fail-mode + /// gate through [`AuditExport::delivery_failing`]. + healthy: Arc, + shutdown: CancellationToken, + done: oneshot::Sender<()>, +} + +impl ExportTask { + async fn run(mut self) { + let mut audit_batch: Vec = Vec::with_capacity(MAX_BATCH); + let mut diag_batch: Vec = Vec::with_capacity(MAX_BATCH); + loop { + let deadline = tokio::time::Instant::now() + BATCH_INTERVAL; + let mut stopping = false; + loop { + tokio::select! { + biased; + () = self.shutdown.cancelled() => { stopping = true; break; } + entry = self.audit_rx.recv() => match entry { + Some(entry) => { + audit_batch.push(entry); + if audit_batch.len() >= MAX_BATCH { + break; + } + } + None => { stopping = true; break; } + }, + entry = self.diag_rx.recv() => match entry { + Some(entry) => { + diag_batch.push(entry); + if diag_batch.len() >= MAX_BATCH { + break; + } + } + None => { stopping = true; break; } + }, + () = tokio::time::sleep_until(deadline) => break, + } + } + // Audit first, always: when both sinks run, export order is + // seq order, and a diagnostic about an outage must not + // overtake the records of the calls it describes. + self.flush_audit(&mut audit_batch).await; + self.flush_diag(&mut diag_batch).await; + if stopping { + // Bounded tail: whatever is already queued, in batches, + // bounded by EXPORT_TIMEOUT per request and by the + // caller's SHUTDOWN_FLUSH_TIMEOUT overall. Anything this + // does not deliver is gone — in a fileless deployment, + // gone entirely. + self.audit_rx.close(); + self.diag_rx.close(); + while let Ok(entry) = self.audit_rx.try_recv() { + audit_batch.push(entry); + if audit_batch.len() >= MAX_BATCH { + self.flush_audit(&mut audit_batch).await; + } + } + while let Ok(entry) = self.diag_rx.try_recv() { + diag_batch.push(entry); + if diag_batch.len() >= MAX_BATCH { + self.flush_diag(&mut diag_batch).await; + } + } + self.flush_audit(&mut audit_batch).await; + self.flush_diag(&mut diag_batch).await; + break; + } + } + let _ = self.done.send(()); + } + + /// Post one audit batch. A failed batch is LOST, never retried (a + /// retry queue is another unbounded buffer in front of a collector + /// that is already not answering) — but never silently: the count + /// reaches the sink's `audit_gap` accounting via `lost`, and the + /// failure flips the health flag that holds the fail-mode gate + /// closed until a delivery succeeds. + async fn flush_audit(&self, batch: &mut Vec) { + if batch.is_empty() { + return; + } + match post_batch(&self.client, &self.cfg, batch).await { + Ok(()) => note_delivery(&self.healthy, true), + Err(_) => { + self.lost.fetch_add(batch.len() as u64, Ordering::Relaxed); + note_delivery(&self.healthy, false); + } + } + batch.clear(); + } + + /// Post one diagnostics batch. A failed batch costs the batch and a + /// counted warning — best-effort. A failure still marks delivery + /// unhealthy (the same collector will refuse audit records too); a + /// success does not clear the latch. Only a successful AUDIT flush + /// proves the load-bearing sink works again. + async fn flush_diag(&self, batch: &mut Vec) { + if batch.is_empty() { + return; + } + match post_batch(&self.client, &self.cfg, batch).await { + Ok(()) => {} + Err(reason) => { + note_drops( + &self.dropped, + &self.log_threshold, + batch.len() as u64, + reason, + ); + note_delivery(&self.healthy, false); + } + } + batch.clear(); + } +} + +// --------------------------------------------------------------------------- +// Diagnostics layer +// --------------------------------------------------------------------------- + +/// The tracing layer that copies the server's diagnostics to OTLP. +/// +/// It holds a slot rather than a pipeline: the subscriber is installed +/// before anything else runs, while the pipeline may only start after the +/// audit sink has opened. Until the slot is filled the layer does nothing +/// but read one atomic per event, and if export is off it is never +/// installed at all. +pub struct DiagnosticsLayer { + slot: Arc>>, +} + +/// Collects an event's fields into the body text, message first, the way +/// the stderr formatter renders them. +/// +/// It also picks the `log.target` field out rather than rendering it. The +/// `log` crate's records reach a `tracing` subscriber through +/// `tracing-log`'s bridge with their metadata target set to the literal +/// `"log"` and their real one demoted to that field, so the field is the +/// only place the emitting module is legible — and [`never_exported`] has +/// to read it, or every `reqwest` and `hyper` line sails past a check that +/// only ever sees `"log"`. Its `log.module_path`/`log.file`/`log.line` +/// siblings are dropped for the same reason the stderr formatter drops +/// them: they are bridge bookkeeping, not what the event said. +struct BodyVisitor { + message: String, + fields: String, + /// The `log.target` field's value, for a bridged `log` record. + log_target: Option, +} + +/// Fields `tracing-log` adds to a bridged record, which are metadata about +/// the bridge rather than part of the event. +fn is_log_bridge_field(name: &str) -> bool { + matches!( + name, + "log.target" | "log.module_path" | "log.file" | "log.line" + ) +} + +impl tracing::field::Visit for BodyVisitor { + fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn fmt::Debug) { + use std::fmt::Write as _; + if field.name() == "log.target" { + self.log_target = Some(format!("{value:?}").trim_matches('"').to_owned()); + return; + } + if is_log_bridge_field(field.name()) { + return; + } + if field.name() == "message" { + let _ = write!(self.message, "{value:?}"); + } else { + let _ = write!( + self.fields, + "{}{}={value:?}", + if self.fields.is_empty() { "" } else { " " }, + field.name() + ); + } + } + + fn record_str(&mut self, field: &tracing::field::Field, value: &str) { + use std::fmt::Write as _; + if field.name() == "log.target" { + self.log_target = Some(value.to_owned()); + return; + } + if is_log_bridge_field(field.name()) { + return; + } + if field.name() == "message" { + self.message.push_str(value); + } else { + let _ = write!( + self.fields, + "{}{}={value}", + if self.fields.is_empty() { "" } else { " " }, + field.name() + ); + } + } +} + +impl Layer for DiagnosticsLayer { + fn on_event(&self, event: &tracing::Event<'_>, _ctx: LayerContext<'_, S>) { + let meta = event.metadata(); + // The export must never carry what the export itself emits; see + // NEVER_EXPORTED_TARGETS for why the HTTP stack is in that set and + // not only this module. + if never_exported(meta.target()) { + return; + } + let Some(pipeline) = self.slot.get() else { + return; + }; + let mut visitor = BodyVisitor { + message: String::new(), + fields: String::new(), + log_target: None, + }; + event.record(&mut visitor); + // Second gate, and the load-bearing one: a bridged `log` record + // only reveals its real target here, in a field, so the cheap + // metadata check above cannot see the export's own HTTP stack — + // reqwest and hyper log through `log`, not through `tracing`. + let target = visitor + .log_target + .as_deref() + .unwrap_or_else(|| meta.target()); + if never_exported(target) { + return; + } + let body = if visitor.fields.is_empty() { + visitor.message + } else if visitor.message.is_empty() { + visitor.fields + } else { + format!("{} {}", visitor.message, visitor.fields) + }; + pipeline.emit(LogEntry { + time_unix_nano: now_unix_nano(), + severity: Severity::of_level(meta.level()), + body, + attrs: vec![ + ("bugwarden.stream", AttrValue::Str("diagnostics".to_owned())), + ("log.target", AttrValue::Str(target.to_owned())), + ], + trace: None, + }); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::audit::{ + AuditEvent, AuditEventKind, ClientInfo, GuardInfo, InitializeEvent, OutcomeClass, + OutcomeInfo, RequestInfo, SessionInfo, ToolCallEvent, TraceContext, TransportKind, Verdict, + }; + use crate::testlog::{assert_logged, capture_logs}; + + fn env(endpoint: &str) -> OtelEnv { + OtelEnv { + endpoint: Some(endpoint.to_owned()), + ..OtelEnv::default() + } + } + + // -- configuration ------------------------------------------------------ + + #[test] + fn unset_endpoint_turns_export_off() { + let resolved = resolve(&OtelEnv::default()).expect("an empty environment resolves"); + assert!( + resolved.is_none(), + "no endpoint must mean no export configuration at all" + ); + } + + #[test] + fn empty_endpoint_turns_export_off() { + // The "cleared variable" idiom of unit files and container specs. + let resolved = resolve(&OtelEnv { + endpoint: Some(String::new()), + ..OtelEnv::default() + }) + .expect("an emptied endpoint resolves"); + assert!(resolved.is_none(), "an emptied endpoint must read as unset"); + let resolved = resolve(&OtelEnv { + endpoint: Some(" ".to_owned()), + ..OtelEnv::default() + }) + .expect("a blank endpoint resolves"); + assert!(resolved.is_none(), "a blank endpoint must read as unset"); + } + + #[test] + fn an_endpoint_gains_the_logs_path_and_the_default_service_name() { + let cfg = resolve(&env("http://collector.example:4318")) + .expect("resolves") + .expect("export is on"); + assert_eq!(cfg.logs_url(), "http://collector.example:4318/v1/logs"); + assert_eq!(cfg.service_name(), "bugwarden"); + // A trailing slash must not produce a doubled separator. + let cfg = resolve(&env("http://collector.example:4318/")) + .expect("resolves") + .expect("export is on"); + assert_eq!(cfg.logs_url(), "http://collector.example:4318/v1/logs"); + } + + #[test] + fn service_name_is_taken_from_the_environment_when_set() { + let cfg = resolve(&OtelEnv { + service_name: Some(" bugwarden-edge ".to_owned()), + ..env("http://c:4318") + }) + .expect("resolves") + .expect("export is on"); + assert_eq!(cfg.service_name(), "bugwarden-edge"); + } + + #[test] + fn the_only_accepted_protocol_is_http_protobuf() { + for accepted in [None, Some("http/protobuf".to_owned())] { + let cfg = resolve(&OtelEnv { + protocol: accepted.clone(), + ..env("http://c:4318") + }) + .expect("http/protobuf and an unset protocol resolve"); + assert!(cfg.is_some(), "{accepted:?} must leave export on"); + } + for rejected in ["grpc", "http/json", "HTTP/PROTOBUF", ""] { + let err = resolve(&OtelEnv { + protocol: Some(rejected.to_owned()), + ..env("http://c:4318") + }) + .err() + .expect("only http/protobuf is spoken"); + let text = format!("{err}"); + assert!( + text.contains(PROTOCOL_VAR) && text.contains(PROTOCOL_HTTP_PROTOBUF), + "the error must name the variable and the accepted value: {text}" + ); + } + } + + #[test] + fn a_bad_protocol_without_an_endpoint_is_not_an_error() { + // Export is off, so nothing about the transport is decided. A + // fleet-wide OTEL_* environment must not refuse to start a + // deployment that exports nothing (issue #31: unset endpoint means + // zero new behaviour). + let resolved = resolve(&OtelEnv { + protocol: Some("grpc".to_owned()), + ..OtelEnv::default() + }) + .expect("no endpoint decides everything"); + assert!(resolved.is_none()); + } + + #[test] + fn an_endpoint_must_be_an_http_url() { + let err = resolve(&env("collector.example:4318")) + .err() + .expect("a bare authority is not a URL"); + assert!(format!("{err}").contains(ENDPOINT_VAR)); + assert!(resolve(&env("https://c:4318")) + .expect("https resolves") + .is_some()); + } + + #[test] + fn headers_parse_into_pairs_and_are_taken_verbatim() { + let cfg = resolve(&OtelEnv { + headers: Some(" authorization=Bearer abc%20def , x-tenant=acme ".to_owned()), + ..env("http://c:4318") + }) + .expect("resolves") + .expect("export is on"); + assert_eq!( + cfg.headers(), + [ + ("authorization".to_owned(), "Bearer abc%20def".to_owned()), + ("x-tenant".to_owned(), "acme".to_owned()), + ], + "values are used as given; percent-encoding is not decoded" + ); + + // A trailing or doubled comma is the shape a generated env file + // produces; it names no header and is not an error. + let cfg = resolve(&OtelEnv { + headers: Some("a=1,,b=2,".to_owned()), + ..env("http://c:4318") + }) + .expect("resolves") + .expect("export is on"); + assert_eq!( + cfg.headers(), + [ + ("a".to_owned(), "1".to_owned()), + ("b".to_owned(), "2".to_owned()), + ] + ); + } + + #[test] + fn the_logs_specific_variables_override_the_general_ones() { + // The OTLP specification makes the signal-specific form win, and + // its endpoint is used AS GIVEN — no signal path is appended, + // because the operator wrote the whole URL. + let cfg = resolve(&OtelEnv { + logs_endpoint: Some("http://logs.example:4318/otlp/v1/logs".to_owned()), + logs_headers: Some("x-logs=1".to_owned()), + headers: Some("x-general=1".to_owned()), + ..env("http://general.example:4318") + }) + .expect("resolves") + .expect("export is on"); + assert_eq!(cfg.logs_url(), "http://logs.example:4318/otlp/v1/logs"); + assert_eq!(cfg.headers(), [("x-logs".to_owned(), "1".to_owned())]); + + // Set ALONE it still turns export on: a fleet naming only the + // logs endpoint expects logs, and reading just the general + // variable would leave the export silently off. + let cfg = resolve(&OtelEnv { + logs_endpoint: Some("http://logs.example:4318/v1/logs".to_owned()), + ..OtelEnv::default() + }) + .expect("resolves") + .expect("a logs endpoint alone means export is on"); + assert_eq!(cfg.logs_url(), "http://logs.example:4318/v1/logs"); + + // Emptied, it is unset, and the general one is used again. + let cfg = resolve(&OtelEnv { + logs_endpoint: Some(String::new()), + ..env("http://general.example:4318") + }) + .expect("resolves") + .expect("export is on"); + assert_eq!(cfg.logs_url(), "http://general.example:4318/v1/logs"); + + // A refusal names the variable that actually carried the value. + let err = resolve(&OtelEnv { + logs_protocol: Some("grpc".to_owned()), + protocol: Some(PROTOCOL_HTTP_PROTOBUF.to_owned()), + ..env("http://c:4318") + }) + .err() + .expect("grpc must be refused wherever it came from"); + assert!( + format!("{err}").contains(LOGS_PROTOCOL_VAR), + "the error must name the losing variable, not its general twin: {err}" + ); + } + + #[test] + fn a_malformed_header_entry_is_refused_without_echoing_it() { + // The classic paste accident: the whole credential where a + // `key=value` belongs. The error must not repeat it (I12). + let secret = "Bearer super-secret-token"; + let err = resolve(&OtelEnv { + headers: Some(format!("x-a=1,{secret}")), + ..env("http://c:4318") + }) + .err() + .expect("an entry without `=` is refused"); + let text = format!("{err}"); + assert!( + text.contains(HEADERS_VAR) && text.contains('2'), + "the error must name the variable and the position: {text}" + ); + assert!( + !text.contains("super-secret-token"), + "the error must never echo header material: {text}" + ); + } + + #[test] + fn an_unusable_header_name_or_value_is_refused() { + // A name with internal whitespace, an empty name or value, an + // entry with no separator at all, and — the one that matters — + // a value carrying a newline, which is header injection. + for bad in ["bad name=1", "x=", "=value", "no-equals-sign", "x=va\nlue"] { + let err = resolve(&OtelEnv { + headers: Some(bad.to_owned()), + ..env("http://c:4318") + }) + .err() + .unwrap_or_else(|| panic!("{bad:?} must be refused")); + assert!(format!("{err}").contains(HEADERS_VAR)); + } + } + + // -- record shaping ----------------------------------------------------- + + fn session() -> SessionInfo { + SessionInfo { + id: Some("sess-1".to_owned()), + transport: TransportKind::Http, + remote: Some("192.0.2.7:52611".to_owned()), + } + } + + fn tool_call(trace: Option) -> AuditEventKind { + AuditEventKind::ToolCall(Box::new(ToolCallEvent { + client: ClientInfo { + name: Some("agent".to_owned()), + version: None, + principal: None, + }, + trace, + request: RequestInfo { + tool: "bug_info".to_owned(), + id: Some("3".to_owned()), + params: std::collections::BTreeMap::new(), + }, + guard: Some(GuardInfo { + verdict: Verdict::Denied, + rule: Some("embargo".to_owned()), + policy_hash: None, + suppressed_count: 1, + suppressed_ids: vec![7], + redacted_fields: Vec::new(), + scan: None, + }), + upstream: None, + outcome: OutcomeInfo { + class: OutcomeClass::Ok, + duration_ms: 3, + }, + })) + } + + fn event(kind: AuditEventKind) -> AuditEvent { + AuditEvent { + v: crate::audit::SCHEMA_VERSION, + ts: "2026-08-18T00:00:00.000Z".to_owned(), + seq: 42, + session: session(), + kind, + } + } + + fn attr<'a>(entry: &'a LogEntry, key: &str) -> Option<&'a AttrValue> { + entry.attrs.iter().find(|(k, _)| *k == key).map(|(_, v)| v) + } + + #[test] + fn an_audit_entry_carries_the_line_verbatim_as_its_body() { + let line = br#"{"v":1,"seq":42}"#; + let entry = audit_entry(&event(tool_call(None)), line); + assert_eq!( + entry.body.as_bytes(), + line, + "the body must be the file's bytes, unchanged" + ); + } + + #[test] + fn an_audit_entry_carries_the_documented_attributes() { + let entry = audit_entry(&event(tool_call(None)), b"{}"); + assert_eq!( + attr(&entry, "bugwarden.stream"), + Some(&AttrValue::Str("audit".to_owned())) + ); + assert_eq!( + attr(&entry, "bugwarden.event"), + Some(&AttrValue::Str("tool_call".to_owned())) + ); + assert_eq!(attr(&entry, "bugwarden.seq"), Some(&AttrValue::Int(42))); + assert_eq!( + attr(&entry, "bugwarden.transport"), + Some(&AttrValue::Str("http".to_owned())) + ); + assert_eq!( + attr(&entry, "bugwarden.session.id"), + Some(&AttrValue::Str("sess-1".to_owned())) + ); + assert_eq!( + attr(&entry, "bugwarden.tool"), + Some(&AttrValue::Str("bug_info".to_owned())) + ); + assert_eq!( + attr(&entry, "bugwarden.verdict"), + Some(&AttrValue::Str("denied".to_owned())) + ); + assert_eq!( + attr(&entry, "bugwarden.rule"), + Some(&AttrValue::Str("embargo".to_owned())) + ); + } + + #[test] + fn severity_follows_the_event_kind() { + assert_eq!( + audit_entry(&event(tool_call(None)), b"{}").severity, + Severity::Info + ); + assert_eq!( + audit_entry( + &event(AuditEventKind::Initialize(InitializeEvent { + client: ClientInfo { + name: None, + version: None, + principal: None + }, + protocol_version: None, + })), + b"{}" + ) + .severity, + Severity::Info + ); + assert_eq!( + audit_entry( + &event(AuditEventKind::AuditGap(crate::audit::AuditGapEvent { + dropped: 2, + reason: crate::audit::GapReason::WriteError, + })), + b"{}" + ) + .severity, + Severity::Error, + "a gap in the record stream is an error, not an info line" + ); + } + + #[test] + fn trace_ids_reach_the_record_as_raw_bytes() { + let entry = audit_entry( + &event(tool_call(Some(TraceContext { + trace_id: "4bf92f3577b34da6a3ce929d0e0e4736".to_owned(), + span_id: "00f067aa0ba902b7".to_owned(), + }))), + b"{}", + ); + let (trace_id, span_id) = entry.trace.expect("a valid traceparent must be exported"); + assert_eq!( + trace_id, + [ + 0x4b, 0xf9, 0x2f, 0x35, 0x77, 0xb3, 0x4d, 0xa6, 0xa3, 0xce, 0x92, 0x9d, 0x0e, 0x0e, + 0x47, 0x36 + ] + ); + assert_eq!(span_id, [0x00, 0xf0, 0x67, 0xaa, 0x0b, 0xa9, 0x02, 0xb7]); + } + + #[test] + fn a_record_without_trace_context_exports_no_ids() { + assert!(audit_entry(&event(tool_call(None)), b"{}").trace.is_none()); + } + + #[test] + fn hex_of_the_wrong_width_or_alphabet_decodes_to_nothing() { + assert!(hex_bytes::<8>("00f067aa0ba902b").is_none()); + assert!(hex_bytes::<8>("00f067aa0ba902b77").is_none()); + assert!(hex_bytes::<8>("00f067aa0ba902bz").is_none()); + assert!(hex_bytes::<8>("").is_none()); + } + + // -- protobuf encoding -------------------------------------------------- + + /// Walk a protobuf message, returning `(field, wire_type, payload)` + /// triples. Length-delimited payloads come back as byte slices; the + /// fixed and varint ones as their raw bytes. + fn fields(buf: &[u8]) -> Vec<(u32, u32, Vec)> { + let mut out = Vec::new(); + let mut i = 0usize; + while i < buf.len() { + let (tag, used) = read_varint(&buf[i..]).expect("a tag"); + i += used; + let field = u32::try_from(tag >> 3).expect("field number"); + let wire_type = u32::try_from(tag & 7).expect("wire type"); + match wire_type { + 0 => { + let (value, used) = read_varint(&buf[i..]).expect("a varint"); + out.push((field, wire_type, value.to_le_bytes().to_vec())); + i += used; + } + 1 => { + out.push((field, wire_type, buf[i..i + 8].to_vec())); + i += 8; + } + 2 => { + let (len, used) = read_varint(&buf[i..]).expect("a length"); + i += used; + let len = usize::try_from(len).expect("a sane length"); + out.push((field, wire_type, buf[i..i + len].to_vec())); + i += len; + } + 5 => { + out.push((field, wire_type, buf[i..i + 4].to_vec())); + i += 4; + } + other => panic!("unexpected wire type {other}"), + } + } + out + } + + fn read_varint(buf: &[u8]) -> Option<(u64, usize)> { + let mut value = 0u64; + for (i, byte) in buf.iter().enumerate().take(10) { + value |= u64::from(byte & 0x7f) << (7 * i); + if byte & 0x80 == 0 { + return Some((value, i + 1)); + } + } + None + } + + fn only(buf: &[u8], field: u32) -> Vec { + fields(buf) + .into_iter() + .find(|(f, _, _)| *f == field) + .unwrap_or_else(|| panic!("field {field} must be present")) + .2 + } + + fn all(buf: &[u8], field: u32) -> Vec> { + fields(buf) + .into_iter() + .filter(|(f, _, _)| *f == field) + .map(|(_, _, v)| v) + .collect() + } + + #[test] + fn varints_encode_the_way_protobuf_reads_them() { + for value in [0u64, 1, 127, 128, 300, 16_383, 16_384, u64::MAX] { + let mut buf = Vec::new(); + wire::put_varint(&mut buf, value); + assert_eq!(read_varint(&buf), Some((value, buf.len())), "{value}"); + } + // The canonical two-byte example from the protobuf encoding docs. + let mut buf = Vec::new(); + wire::put_varint(&mut buf, 300); + assert_eq!(buf, [0xac, 0x02]); + } + + #[test] + fn a_request_nests_resource_scope_and_records_as_otlp_expects() { + let entry = audit_entry(&event(tool_call(None)), b"{\"seq\":42}"); + let request = encode_request("bugwarden", std::slice::from_ref(&entry)); + + let resource_logs = only(&request, 1); + let resource = only(&resource_logs, 1); + let service_kv = only(&resource, 1); + assert_eq!(only(&service_kv, 1), b"service.name"); + assert_eq!(only(&only(&service_kv, 2), 1), b"bugwarden"); + + let scope_logs = only(&resource_logs, 2); + let scope = only(&scope_logs, 1); + assert_eq!(only(&scope, 1), SCOPE_NAME.as_bytes()); + assert_eq!(only(&scope, 2), env!("CARGO_PKG_VERSION").as_bytes()); + + let records = all(&scope_logs, 2); + assert_eq!(records.len(), 1); + let record = &records[0]; + // severity_number is a varint; the helper hands back little-endian + // bytes of the decoded value. + assert_eq!(only(record, 2)[0], Severity::Info as u8); + assert_eq!(only(record, 3), b"INFO"); + assert_eq!(only(&only(record, 5), 1), b"{\"seq\":42}"); + assert!( + all(record, 6).len() >= 7, + "every documented attribute must be encoded" + ); + // The two timestamps, by FIELD NUMBER: `time_unix_nano` is 1 and + // `observed_time_unix_nano` is 11, and field 4 is reserved in the + // schema. Pinning the numbers is what stops a stamp from being + // written where no reader looks for it. + let stamp = entry.time_unix_nano.to_le_bytes().to_vec(); + assert_eq!(only(record, 1), stamp, "time_unix_nano is field 1"); + assert_eq!( + only(record, 11), + stamp, + "observed_time_unix_nano is field 11" + ); + assert!( + !fields(record).iter().any(|(f, _, _)| *f == 4), + "field 4 is reserved in the OTLP LogRecord and must carry nothing" + ); + } + + #[test] + fn every_severity_keeps_its_otlp_number_and_text() { + // The numbers are the base of each OTLP severity range and a + // consumer filters on them, so they are wire contract rather than + // an internal enum: pinned here, and the ERROR one again through + // the encoder, since only INFO rides the request test above. + for (severity, number, text) in [ + (Severity::Trace, 1u8, "TRACE"), + (Severity::Debug, 5, "DEBUG"), + (Severity::Info, 9, "INFO"), + (Severity::Warn, 13, "WARN"), + (Severity::Error, 17, "ERROR"), + ] { + assert_eq!(severity as u8, number, "{text} is OTLP severity {number}"); + assert_eq!(severity.text(), text); + } + + let gap = audit_entry( + &event(AuditEventKind::AuditGap(crate::audit::AuditGapEvent { + dropped: 2, + reason: crate::audit::GapReason::WriteError, + })), + b"{}", + ); + let record = all(&only(&only(&request_of(&gap), 1), 2), 2).remove(0); + assert_eq!(only(&record, 2)[0], 17, "an audit_gap is ERROR on the wire"); + assert_eq!(only(&record, 3), b"ERROR"); + } + + #[test] + fn a_batch_becomes_one_request_with_one_record_each() { + let entries: Vec = (0..3) + .map(|_| audit_entry(&event(tool_call(None)), b"{}")) + .collect(); + let request = encode_request("bugwarden", &entries); + let scope_logs = only(&only(&request, 1), 2); + assert_eq!(all(&scope_logs, 2).len(), 3); + } + + #[test] + fn trace_ids_are_encoded_as_the_fixed_width_byte_fields() { + let entry = audit_entry( + &event(tool_call(Some(TraceContext { + trace_id: "4bf92f3577b34da6a3ce929d0e0e4736".to_owned(), + span_id: "00f067aa0ba902b7".to_owned(), + }))), + b"{}", + ); + let record = all(&only(&only(&request_of(&entry), 1), 2), 2).remove(0); + assert_eq!(only(&record, 9).len(), 16); + assert_eq!(only(&record, 10).len(), 8); + assert_eq!(only(&record, 9)[0], 0x4b); + assert_eq!(only(&record, 10)[1], 0xf0); + } + + fn request_of(entry: &LogEntry) -> Vec { + encode_request("bugwarden", std::slice::from_ref(entry)) + } + + // -- drop accounting ---------------------------------------------------- + + #[test] + fn drops_are_logged_at_powers_of_two_only() { + let dropped = AtomicU64::new(0); + let threshold = AtomicU64::new(1); + let (_, logs) = capture_logs(|| { + for _ in 0..16 { + note_drops(&dropped, &threshold, 1, REASON_QUEUE_FULL); + } + }); + assert_eq!(dropped.load(Ordering::Relaxed), 16); + let lines = logs + .as_str() + .matches("otlp export is dropping diagnostic records") + .count(); + assert_eq!( + lines, + 5, + "one line per crossing of 1, 2, 4, 8, 16 — not one per record: {}", + logs.as_str() + ); + assert_logged(&logs, "dropped=16"); + } + + #[test] + fn a_batch_sized_drop_logs_once_and_moves_the_threshold_past_it() { + let dropped = AtomicU64::new(0); + let threshold = AtomicU64::new(1); + let (_, logs) = capture_logs(|| { + note_drops(&dropped, &threshold, 300, REASON_NETWORK); + note_drops(&dropped, &threshold, 1, REASON_NETWORK); + }); + assert_eq!( + logs.as_str() + .matches("otlp export is dropping diagnostic records") + .count(), + 1, + "a 300-record batch crosses nine powers of two (1..256) and still logs once: {}", + logs.as_str() + ); + assert_eq!(threshold.load(Ordering::Relaxed), 512); + } + + #[test] + fn the_drop_line_names_no_endpoint_and_no_header() { + let dropped = AtomicU64::new(0); + let threshold = AtomicU64::new(1); + let (_, logs) = capture_logs(|| { + note_drops(&dropped, &threshold, 1, REASON_NETWORK); + }); + assert_logged(&logs, "reason=\"network\""); + for forbidden in ["http://", "https://", "authorization", "Bearer", "4318"] { + logs.assert_not_contains(forbidden); + } + } + + // -- pipeline ----------------------------------------------------------- + + #[tokio::test] + async fn an_unreachable_collector_loses_audit_records_loudly_not_as_drops() { + // An accepted-then-undelivered audit record is a LOSS the sink + // must be able to account for (take_lost -> audit_gap), never a + // silent drop; the diagnostics counter stays at zero for it, and + // delivery is marked failing so the fail-mode gate closes. + let cfg = resolve(&env("http://127.0.0.1:1/")) + .expect("resolves") + .expect("export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + assert!( + !pipeline.delivery_failing(), + "before any attempt, delivery is not known to be failing" + ); + pipeline + .accept(&event(tool_call(None)), b"{}") + .expect("an open queue takes custody"); + // The batch interval plus a margin: enough for one failed post. + tokio::time::sleep(BATCH_INTERVAL * 3).await; + assert!( + pipeline.delivery_failing(), + "a failed delivery must mark the pipeline failing" + ); + assert_eq!( + pipeline.dropped(), + 0, + "audit records never ride the diagnostics drop counter" + ); + assert_eq!( + pipeline.take_lost(), + 1, + "the undelivered record must be surfaced for gap accounting" + ); + assert_eq!(pipeline.take_lost(), 0, "take_lost drains the count"); + pipeline.shutdown().await; + } + + #[tokio::test] + async fn an_unreachable_collector_counts_diagnostics_as_drops() { + let cfg = resolve(&env("http://127.0.0.1:1/")) + .expect("resolves") + .expect("export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + pipeline.emit(LogEntry { + time_unix_nano: now_unix_nano(), + severity: Severity::Info, + body: "diagnostic".to_owned(), + attrs: Vec::new(), + trace: None, + }); + tokio::time::sleep(BATCH_INTERVAL * 3).await; + pipeline.shutdown().await; + assert!( + pipeline.dropped() >= 1, + "an unreachable collector must count the diagnostics it lost" + ); + assert_eq!( + pipeline.take_lost(), + 0, + "a dropped diagnostic is not an audit loss" + ); + } + + #[tokio::test] + async fn a_record_offered_after_shutdown_is_refused_not_dropped() { + // The audit queue REFUSES what it cannot take: the sink turns the + // refusal into a failed record and the fail mode decides what the + // caller sees. Only diagnostics are dropped-and-counted. + let cfg = resolve(&env("http://127.0.0.1:1/")) + .expect("resolves") + .expect("export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + pipeline.shutdown().await; + let before = pipeline.dropped(); + assert!( + pipeline.accept(&event(tool_call(None)), b"{}").is_err(), + "a shut-down pipeline must refuse custody" + ); + assert!( + pipeline.delivery_failing(), + "a refused record marks delivery failing" + ); + assert_eq!( + pipeline.dropped(), + before, + "a refused audit record must not be counted as a drop" + ); + } + + #[tokio::test] + async fn a_diagnostic_offered_after_shutdown_is_a_shutdown_drop() { + // The queue is closed, not full, and saying "queue_full" would + // send an operator looking for a volume problem that is not + // there. The line is lost either way — this is about the + // vocabulary being true. + let cfg = resolve(&env("http://127.0.0.1:1/")) + .expect("resolves") + .expect("export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + pipeline.shutdown().await; + let before = pipeline.dropped(); + let (_, logs) = capture_logs(|| { + pipeline.emit(LogEntry { + time_unix_nano: now_unix_nano(), + severity: Severity::Info, + body: "late diagnostic".to_owned(), + attrs: Vec::new(), + trace: None, + }); + }); + assert_eq!( + pipeline.dropped(), + before + 1, + "a diagnostic offered to a shut-down pipeline is still counted" + ); + assert_logged(&logs, "reason=\"shutdown\""); + } + + #[test] + fn delivery_notes_log_transitions_only() { + // One line per outage and one per recovery: a collector that is + // down for an hour must not log once per batch interval. + let healthy = AtomicBool::new(true); + let (_, logs) = capture_logs(|| { + note_delivery(&healthy, false); + note_delivery(&healthy, false); + note_delivery(&healthy, false); + }); + assert_eq!( + logs.as_str().matches("otlp delivery is failing").count(), + 1, + "repeated failures log once: {}", + logs.as_str() + ); + assert!(!healthy.load(Ordering::Relaxed)); + let (_, logs) = capture_logs(|| { + note_delivery(&healthy, true); + note_delivery(&healthy, true); + }); + assert_eq!( + logs.as_str().matches("otlp delivery recovered").count(), + 1, + "repeated successes log once: {}", + logs.as_str() + ); + assert!(healthy.load(Ordering::Relaxed)); + } + + #[tokio::test] + async fn the_startup_probe_refuses_a_dead_collector_without_naming_it() { + // The refusal names the variables the operator has to check and + // never the endpoint they hold (I12) — port 1 would be the tell. + let cfg = resolve(&env("http://127.0.0.1:1/")) + .expect("resolves") + .expect("export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + let err = pipeline + .probe() + .await + .expect_err("a dead collector must refuse startup"); + let message = format!("{err}"); + assert!( + message.contains(ENDPOINT_VAR) && message.contains(HEADERS_VAR), + "the refusal must point at the configuration: {message}" + ); + assert!( + !message.contains(LOGS_ENDPOINT_VAR) && !message.contains(LOGS_HEADERS_VAR), + "a general-variable config must not blame the logs-specific pair: {message}" + ); + assert!( + !message.contains("127.0.0.1") && !message.contains(":1"), + "the refusal must not carry the endpoint: {message}" + ); + assert!( + pipeline.delivery_failing(), + "a failed probe leaves delivery marked failing" + ); + pipeline.shutdown().await; + } + + #[tokio::test] + async fn the_startup_probe_names_the_logs_specific_variables_when_those_won() { + let cfg = resolve(&OtelEnv { + logs_endpoint: Some("http://127.0.0.1:1/v1/logs".to_owned()), + logs_headers: Some("authorization=Bearer x".to_owned()), + ..OtelEnv::default() + }) + .expect("resolves") + .expect("a logs-specific endpoint turns export on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + let message = format!( + "{}", + pipeline + .probe() + .await + .expect_err("a dead collector must refuse startup") + ); + assert!( + message.contains(LOGS_ENDPOINT_VAR) && message.contains(LOGS_HEADERS_VAR), + "the refusal must name the variables that actually won: {message}" + ); + assert!( + !message.contains("127.0.0.1") && !message.contains("Bearer"), + "the refusal must not carry the endpoint or the credential: {message}" + ); + pipeline.shutdown().await; + } + + #[tokio::test] + async fn shutdown_is_idempotent_and_bounded() { + let cfg = resolve(&env("http://127.0.0.1:1/")) + .expect("resolves") + .expect("export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + pipeline.shutdown().await; + pipeline.shutdown().await; + } + + #[test] + fn the_pipeline_debug_carries_no_configuration() { + // Reachable from AuditSink's derived Debug, so it must hold + // nothing that came out of the environment (I12). + let rendered = { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("a runtime"); + let _guard = runtime.enter(); + let cfg = resolve(&OtelEnv { + headers: Some("authorization=Bearer super-secret-token".to_owned()), + ..env("http://collector.example:4318") + }) + .expect("resolves") + .expect("export is on"); + format!( + "{:?}", + Pipeline::start(cfg).expect("the pipeline must start") + ) + }; + for forbidden in [ + "collector.example", + "super-secret-token", + "authorization", + "4318", + ] { + assert!( + !rendered.contains(forbidden), + "Pipeline's Debug must not carry {forbidden}: {rendered}" + ); + } + } +} diff --git a/crates/bugwarden/src/server.rs b/crates/bugwarden/src/server.rs index a26878b..44cc3d8 100644 --- a/crates/bugwarden/src/server.rs +++ b/crates/bugwarden/src/server.rs @@ -4725,7 +4725,7 @@ mod tests { ) -> (Arc, std::path::PathBuf) { let path = dir.join("audit.jsonl"); let sink = AuditSink::open(AuditConfig { - path: path.clone(), + path: Some(path.clone()), fsync: false, fail_mode: None, rotate_max_bytes: 0, diff --git a/crates/bugwarden/tests/audit_wiremock.rs b/crates/bugwarden/tests/audit_wiremock.rs index 905d91d..b2d24ad 100644 --- a/crates/bugwarden/tests/audit_wiremock.rs +++ b/crates/bugwarden/tests/audit_wiremock.rs @@ -55,7 +55,7 @@ async fn audited_client_with( let dir = tempfile::tempdir().expect("tempdir"); let audit_path = dir.path().join("audit.jsonl"); let sink = AuditSink::open(AuditConfig { - path: audit_path.clone(), + path: Some(audit_path.clone()), fsync: false, fail_mode: None, rotate_max_bytes: 0, diff --git a/crates/bugwarden/tests/binary_shutdown.rs b/crates/bugwarden/tests/binary_shutdown.rs index 6824462..dc20bf6 100644 --- a/crates/bugwarden/tests/binary_shutdown.rs +++ b/crates/bugwarden/tests/binary_shutdown.rs @@ -56,6 +56,13 @@ const AMBIENT_VARS: &[&str] = &[ "MCP_READ_ONLY", "MCP_API_KEY_HEADER", "RUST_LOG", + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_EXPORTER_OTLP_HEADERS", + "OTEL_EXPORTER_OTLP_PROTOCOL", + "OTEL_SERVICE_NAME", + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT", + "OTEL_EXPORTER_OTLP_LOGS_HEADERS", + "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL", ]; /// The scrub list is only as good as its coverage of `Cli`. @@ -79,6 +86,15 @@ fn the_scrub_list_covers_every_environment_fallback() { ] { assert!(AMBIENT_VARS.contains(&var), "{var} must be scrubbed"); } + let unscrubbed_otel: Vec<&str> = bugwarden::otel::ENV_VARS + .iter() + .copied() + .filter(|var| !AMBIENT_VARS.contains(var)) + .collect(); + assert!( + unscrubbed_otel.is_empty(), + "these OTLP variables reach the spawned binary: {unscrubbed_otel:?}" + ); } /// A port to hand the child, chosen by binding and releasing an ephemeral diff --git a/crates/bugwarden/tests/binary_user_agent.rs b/crates/bugwarden/tests/binary_user_agent.rs index 1dddde6..e0bd415 100644 --- a/crates/bugwarden/tests/binary_user_agent.rs +++ b/crates/bugwarden/tests/binary_user_agent.rs @@ -21,12 +21,14 @@ use wiremock::{Mock, MockServer, ResponseTemplate}; const REPLY_TIMEOUT: Duration = Duration::from_secs(20); /// The ambient environment must not reach the child: every one of these is -/// read by `Cli` (`RUST_LOG` only muddies the captured stderr). Scrubbed as +/// read by `Cli`, or — the `OTEL_*` four — by `bugwarden::otel` outside +/// clap entirely (`RUST_LOG` only muddies the captured stderr). Scrubbed as /// one set rather than per test — the http-only knobs are inert for a stdio /// run today, and pruning them is how the list falls behind `Cli` again. /// `the_scrub_list_covers_every_environment_fallback` holds it to every -/// `env`-backed flag, so adding one cannot quietly leave a hole here. -const SCRUBBED_ENV: [&str; 13] = [ +/// `env`-backed flag AND to every variable the otel module names, so adding +/// one in either population cannot quietly leave a hole here. +const SCRUBBED_ENV: [&str; 20] = [ "BUGZILLA_SERVER", "BUGZILLA_API_KEY", "BUGZILLA_API_KEY_FILE", @@ -40,6 +42,19 @@ const SCRUBBED_ENV: [&str; 13] = [ "MCP_API_KEY_HEADER", "MCP_READ_ONLY", "RUST_LOG", + // Read by the otel module, not by `Cli`: a developer exporting to a + // collector would otherwise have every spawned child export too, and + // an unreachable one would slow this test down for no reason. + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_EXPORTER_OTLP_HEADERS", + "OTEL_EXPORTER_OTLP_PROTOCOL", + "OTEL_SERVICE_NAME", + // The signal-specific trio, which the OTLP spec makes override the + // three above — so leaving them ambient would override the test's own + // world, not merely add to it. + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT", + "OTEL_EXPORTER_OTLP_LOGS_HEADERS", + "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL", ]; /// The scrub list above is only as good as its coverage of `Cli`, and a @@ -63,6 +78,17 @@ fn the_scrub_list_covers_every_environment_fallback() { unscrubbed.is_empty(), "these environment fallbacks reach the spawned binary: {unscrubbed:?}" ); + + // The second population: variables read outside clap, which the sweep + // above cannot see because no `Arg` carries them. + let unscrubbed: Vec<&str> = bugwarden::otel::ENV_VARS + .into_iter() + .filter(|var| !SCRUBBED_ENV.contains(var)) + .collect(); + assert!( + unscrubbed.is_empty(), + "these otel variables reach the spawned binary: {unscrubbed:?}" + ); } /// The identity this build must present. Spelled out rather than read from diff --git a/crates/bugwarden/tests/env_config.rs b/crates/bugwarden/tests/env_config.rs index e9c4690..4b590cd 100644 --- a/crates/bugwarden/tests/env_config.rs +++ b/crates/bugwarden/tests/env_config.rs @@ -9,17 +9,34 @@ //! - dropping `env = "MCP_ALLOWED_HOSTS"` or `env = "BUGZILLA_USE_AUTH_HEADER"`; //! - keeping the empty entry `MCP_ALLOWED_HOSTS=` produces, which would turn //! Host validation ON with nothing matchable and refuse every request; -//! - splitting `MCP_ALLOWED_HOSTS=a b.example` on whitespace into two hosts; -//! - letting the environment override the command line. +//! - not splitting `MCP_ALLOWED_HOSTS=a b.example` on whitespace (a space is part of the authority); +//! - letting the environment override the command line; +//! - reading an OTLP variable from anywhere but the process environment, or +//! letting an emptied `OTEL_EXPORTER_OTLP_ENDPOINT` leave export on; +//! - dropping `env = "BUGWARDEN_AUDIT_CONFIG"`, widening the exact-bytes +//! `none` sentinel, or letting mere absence of the variable select +//! OTLP-only auditing (the file may only be disabled explicitly). +use std::path::Path; + +use bugwarden::audit::{select_sinks, SinkSelection}; use bugwarden::config::Cli; +use bugwarden::otel::{self, OtelEnv}; use clap::error::ErrorKind; use clap::Parser as _; -const VARS: [&str; 3] = [ +const VARS: [&str; 11] = [ "MCP_ALLOWED_HOSTS", "BUGZILLA_USE_AUTH_HEADER", "MCP_READ_ONLY", + "BUGWARDEN_AUDIT_CONFIG", + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_EXPORTER_OTLP_HEADERS", + "OTEL_EXPORTER_OTLP_PROTOCOL", + "OTEL_SERVICE_NAME", + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT", + "OTEL_EXPORTER_OTLP_LOGS_HEADERS", + "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL", ]; fn clear() { @@ -128,5 +145,121 @@ fn every_flag_is_settable_from_the_environment() { ); } + // The OTLP export knobs are read by `bugwarden::otel`, never by clap, + // so nothing above would notice if `OtelEnv::from_env` stopped reading + // one. They live in this test because it owns process-environment + // mutation for the whole crate. + clear(); + assert!( + otel::resolve(&OtelEnv::from_env()) + .expect("an environment with no endpoint resolves") + .is_none(), + "with no OTEL_EXPORTER_OTLP_ENDPOINT the export must be off" + ); + + std::env::set_var( + "OTEL_EXPORTER_OTLP_ENDPOINT", + "http://collector.example:4318", + ); + std::env::set_var("OTEL_SERVICE_NAME", "bugwarden-edge"); + let cfg = otel::resolve(&OtelEnv::from_env()) + .expect("an endpoint resolves") + .expect("an endpoint means export is on"); + assert_eq!( + cfg.service_name(), + "bugwarden-edge", + "OTEL_SERVICE_NAME must reach the exported resource" + ); + + // The set-but-empty "unset" idiom again, and here it is the off switch + // for the whole feature. + std::env::set_var("OTEL_EXPORTER_OTLP_ENDPOINT", ""); + assert!( + otel::resolve(&OtelEnv::from_env()) + .expect("an emptied endpoint resolves") + .is_none(), + "OTEL_EXPORTER_OTLP_ENDPOINT= must read as unset, leaving export off" + ); + + // A protocol this build cannot speak is a startup error — but only + // once an endpoint makes the transport matter. + std::env::set_var("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc"); + assert!( + otel::resolve(&OtelEnv::from_env()) + .expect("no endpoint decides everything") + .is_none(), + "with export off the protocol is not consulted" + ); + std::env::set_var( + "OTEL_EXPORTER_OTLP_ENDPOINT", + "http://collector.example:4318", + ); + let err = otel::resolve(&OtelEnv::from_env()) + .err() + .expect("grpc must be refused"); + assert!( + format!("{err}").contains("OTEL_EXPORTER_OTLP_PROTOCOL"), + "the refusal must name the variable: {err}" + ); + + // Sink selection (issue #31, revised 2026-08-18). The variable reaches + // the selection, and the literal `none` — exact bytes — is the ONLY + // spelling that disables the audit file; absence beside an OTLP + // endpoint refuses startup rather than silently going fileless. + clear(); + std::env::set_var("BUGWARDEN_AUDIT_CONFIG", "/etc/bugwarden/audit.toml"); + let cli = parse(&[]).expect("a config path parses"); + assert_eq!( + select_sinks(cli.audit_config.as_deref(), false).expect("file only"), + SinkSelection::FileOnly, + "BUGWARDEN_AUDIT_CONFIG must reach the sink selection" + ); + assert_eq!( + select_sinks(cli.audit_config.as_deref(), true).expect("both sinks"), + SinkSelection::Both + ); + std::env::set_var("BUGWARDEN_AUDIT_CONFIG", "none"); + let cli = parse(&[]).expect("the sentinel parses"); + assert_eq!( + select_sinks(cli.audit_config.as_deref(), true).expect("otlp only"), + SinkSelection::OtlpOnly, + "BUGWARDEN_AUDIT_CONFIG=none must select the fileless sink" + ); + assert!( + select_sinks(cli.audit_config.as_deref(), false).is_err(), + "`none` with no OTLP endpoint must refuse startup, never run sinkless" + ); + // The command line wins over the environment, as everywhere. + let cli = parse(&["--audit-config", "/from/cli.toml"]).expect("the flag parses"); + assert_eq!( + cli.audit_config.as_deref(), + Some(Path::new("/from/cli.toml")) + ); + // Absence is not `none`: with an endpoint configured the server + // demands an explicit file decision, and with none it audits nothing. + clear(); + let cli = parse(&[]).expect("no variable parses"); + assert!( + select_sinks(cli.audit_config.as_deref(), true).is_err(), + "an OTLP endpoint without a file decision must refuse startup" + ); + assert_eq!( + select_sinks(cli.audit_config.as_deref(), false).expect("no audit"), + SinkSelection::NoAudit + ); + + // The logs-specific endpoint alone turns export on, and is used as + // given: a fleet that names only this variable expects logs exported, + // and reading only the general one would leave it silently off. + clear(); + std::env::set_var( + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT", + "http://collector.example:4318/otlp/v1/logs", + ); + let cfg = otel::resolve(&OtelEnv::from_env()) + .expect("a logs endpoint resolves") + .expect("a logs endpoint alone means export is on"); + assert_eq!(cfg.service_name(), "bugwarden"); + clear(); } diff --git a/crates/bugwarden/tests/http_auth_wiremock.rs b/crates/bugwarden/tests/http_auth_wiremock.rs index 5ee8ad3..3e9741a 100644 --- a/crates/bugwarden/tests/http_auth_wiremock.rs +++ b/crates/bugwarden/tests/http_auth_wiremock.rs @@ -75,6 +75,13 @@ const AMBIENT_VARS: &[&str] = &[ "MCP_READ_ONLY", "MCP_API_KEY_HEADER", "RUST_LOG", + "OTEL_EXPORTER_OTLP_ENDPOINT", + "OTEL_EXPORTER_OTLP_HEADERS", + "OTEL_EXPORTER_OTLP_PROTOCOL", + "OTEL_SERVICE_NAME", + "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT", + "OTEL_EXPORTER_OTLP_LOGS_HEADERS", + "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL", ]; /// The scrub list is only as good as its coverage of `Cli`; a flag added @@ -101,6 +108,15 @@ fn the_scrub_list_covers_every_environment_fallback() { ] { assert!(AMBIENT_VARS.contains(&var), "{var} must be scrubbed"); } + let unscrubbed_otel: Vec<&str> = bugwarden::otel::ENV_VARS + .iter() + .copied() + .filter(|var| !AMBIENT_VARS.contains(var)) + .collect(); + assert!( + unscrubbed_otel.is_empty(), + "these OTLP variables reach the spawned binary: {unscrubbed_otel:?}" + ); } // ---------- wire-level harness ---------- diff --git a/crates/bugwarden/tests/http_transport_wiremock.rs b/crates/bugwarden/tests/http_transport_wiremock.rs index 663bdbe..4c5b6a3 100644 --- a/crates/bugwarden/tests/http_transport_wiremock.rs +++ b/crates/bugwarden/tests/http_transport_wiremock.rs @@ -547,7 +547,7 @@ async fn a_handshake_free_call_is_refused_and_never_names_a_client() { let dir = tempfile::tempdir().expect("audit temp dir"); let audit_path = dir.path().join("audit.jsonl"); let sink = AuditSink::open(AuditConfig { - path: audit_path.clone(), + path: Some(audit_path.clone()), fsync: false, fail_mode: None, rotate_max_bytes: 0, @@ -700,7 +700,7 @@ async fn traceparent_over_http_lands_in_the_audit_record() { let dir = tempfile::tempdir().expect("audit temp dir"); let audit_path = dir.path().join("audit.jsonl"); let sink = AuditSink::open(AuditConfig { - path: audit_path.clone(), + path: Some(audit_path.clone()), fsync: false, fail_mode: None, rotate_max_bytes: 0, diff --git a/crates/bugwarden/tests/otel_diagnostics.rs b/crates/bugwarden/tests/otel_diagnostics.rs new file mode 100644 index 0000000..81089d4 --- /dev/null +++ b/crates/bugwarden/tests/otel_diagnostics.rs @@ -0,0 +1,342 @@ +//! The diagnostics half of the OTLP export (issue #31). +//! +//! ONE test on purpose, in a test binary of its own: the layer under test +//! only works through a process-wide `tracing` subscriber, and a process +//! has exactly one. Put new diagnostics cases inside this test rather than +//! beside it, and keep this file free of anything else. +//! +//! The subscriber here runs at `debug`, not `info`, and that is the point +//! rather than an accident: the export's own HTTP stack only becomes +//! talkative at debug, and the feedback loop it can start is invisible at +//! any lower level. +//! +//! Coverage contract (each of these mutations must fail this test): +//! - dropping the OTLP layer from the subscriber, or never filling its +//! pipeline slot; +//! - exporting an event's message without its fields; +//! - forwarding this module's OWN diagnostics, which would make a failing +//! exporter its own source of records to export; +//! - narrowing the skip back to this module's target, which lets the +//! exporter's `reqwest`/`hyper` events be exported and turns one flush +//! into an endless self-feeding one. + +use std::sync::{Arc, Mutex, OnceLock}; +use std::time::Duration; + +use bugwarden::otel::{OtelEnv, Pipeline}; +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tracing_subscriber::layer::SubscriberExt as _; +use tracing_subscriber::util::SubscriberInitExt as _; +use tracing_subscriber::EnvFilter; + +/// Bodies posted to the collector, newest last. +type Posted = Arc>>>; + +/// A collector that answers OTLP posts and LOGS NOTHING ITSELF. +/// +/// Deliberately hand-rolled rather than a `MockServer`: this test asserts +/// that an idle server stops exporting, and wiremock runs in this process +/// and logs a line per request through the `log` crate, which would appear +/// in the export as a record and feed exactly the loop under test. A real +/// deployment's collector is another process and contributes no events +/// here, so a socket that answers and says nothing is the faithful stand-in. +async fn silent_collector() -> (String, Posted) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") + .await + .expect("a free port"); + let addr = listener.local_addr().expect("an address"); + let posted: Posted = Arc::new(Mutex::new(Vec::new())); + let sink = posted.clone(); + tokio::spawn(async move { + while let Ok((mut stream, _)) = listener.accept().await { + let sink = sink.clone(); + tokio::spawn(async move { + let mut buf: Vec = Vec::new(); + let mut chunk = [0u8; 8192]; + // Keep-alive: reqwest reuses the connection, so one task + // serves however many requests arrive on it. + loop { + let head_end = loop { + if let Some(at) = buf.windows(4).position(|w| w == b"\r\n\r\n") { + break at + 4; + } + match stream.read(&mut chunk).await { + Ok(0) | Err(_) => return, + Ok(n) => buf.extend_from_slice(&chunk[..n]), + } + }; + let head = String::from_utf8_lossy(&buf[..head_end]).to_ascii_lowercase(); + let len = head + .split("content-length:") + .nth(1) + .and_then(|rest| rest.split("\r\n").next()) + .and_then(|value| value.trim().parse::().ok()) + .unwrap_or(0); + while buf.len() < head_end + len { + match stream.read(&mut chunk).await { + Ok(0) | Err(_) => return, + Ok(n) => buf.extend_from_slice(&chunk[..n]), + } + } + sink.lock() + .expect("the posted-bodies lock") + .push(buf[head_end..head_end + len].to_vec()); + if stream + .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\n\r\n") + .await + .is_err() + { + return; + } + buf.drain(..head_end + len); + } + }); + } + }); + (format!("http://{addr}"), posted) +} + +fn read_varint(buf: &[u8]) -> Option<(u64, usize)> { + let mut value = 0u64; + for (i, byte) in buf.iter().enumerate().take(10) { + value |= u64::from(byte & 0x7f) << (7 * i); + if byte & 0x80 == 0 { + return Some((value, i + 1)); + } + } + None +} + +fn fields(buf: &[u8]) -> Vec<(u32, u32, Vec)> { + let mut out = Vec::new(); + let mut i = 0usize; + while i < buf.len() { + let (tag, used) = read_varint(&buf[i..]).expect("a field tag"); + i += used; + let field = u32::try_from(tag >> 3).expect("a field number"); + let wire_type = u32::try_from(tag & 7).expect("a wire type"); + match wire_type { + 0 => { + let (_, used) = read_varint(&buf[i..]).expect("a varint"); + out.push((field, wire_type, buf[i..i + used].to_vec())); + i += used; + } + 1 => { + out.push((field, wire_type, buf[i..i + 8].to_vec())); + i += 8; + } + 2 => { + let (len, used) = read_varint(&buf[i..]).expect("a length"); + i += used; + let len = usize::try_from(len).expect("a sane length"); + out.push((field, wire_type, buf[i..i + len].to_vec())); + i += len; + } + 5 => { + out.push((field, wire_type, buf[i..i + 4].to_vec())); + i += 4; + } + other => panic!("unexpected protobuf wire type {other}"), + } + } + out +} + +fn first(buf: &[u8], field: u32) -> Option> { + fields(buf) + .into_iter() + .find(|(f, _, _)| *f == field) + .map(|(_, _, v)| v) +} + +fn every(buf: &[u8], field: u32) -> Vec> { + fields(buf) + .into_iter() + .filter(|(f, _, _)| *f == field) + .map(|(_, _, v)| v) + .collect() +} + +/// `(body, attributes)` of every log record in an +/// `ExportLogsServiceRequest`. +fn decode(payload: &[u8]) -> Vec<(String, Vec<(String, String)>)> { + let mut out = Vec::new(); + for resource_logs in every(payload, 1) { + for scope_logs in every(&resource_logs, 2) { + for record in every(&scope_logs, 2) { + let body = String::from_utf8( + first(&first(&record, 5).expect("a body"), 1).expect("a string body"), + ) + .expect("a utf-8 body"); + let attrs = every(&record, 6) + .into_iter() + .map(|kv| { + let key = String::from_utf8(first(&kv, 1).expect("a key")).expect("utf-8"); + let value = first(&kv, 2) + .and_then(|any| first(&any, 1)) + .map(|s| String::from_utf8(s).expect("utf-8")) + .unwrap_or_default(); + (key, value) + }) + .collect(); + out.push((body, attrs)); + } + } + } + out +} + +#[tokio::test] +async fn the_servers_diagnostics_reach_the_collector_but_the_exporters_own_do_not() { + let (endpoint, posted) = silent_collector().await; + + // The subscriber under test: the OTLP layer beside the filter, exactly + // the shape `main` installs when export is on. + let slot: Arc>> = Arc::new(OnceLock::new()); + // `init()` and not `set_global_default`, because it is what `main` + // calls and because it also installs the `log` bridge: reqwest and + // hyper log through `log`, so without the bridge their events would + // never reach the layer and the assertions below would hold + // vacuously. + tracing_subscriber::registry() + // debug, so the exporter's own HTTP stack is loud enough to feed + // itself if the skip below does not stop it. + .with(EnvFilter::new("debug")) + .with(Pipeline::diagnostics_layer(slot.clone())) + .init(); + + // Before the pipeline exists the layer is inert; this event is emitted + // into an empty slot and must simply be dropped rather than panic. + tracing::info!("otel-diagnostics-before-start"); + + let cfg = bugwarden::otel::resolve(&OtelEnv { + endpoint: Some(endpoint), + ..OtelEnv::default() + }) + .expect("the endpoint must resolve") + .expect("an endpoint means export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + slot.set(pipeline.clone()).expect("the slot fills once"); + + tracing::info!(answer = 42, "otel-diagnostics-probe"); + // What the exporter says about itself never goes on the wire: this is + // the shape of the drop warning, and exporting it would feed a failing + // exporter its own failures. + tracing::warn!(target: "bugwarden::otel", "otel-diagnostics-self-target"); + // And what its HTTP stack says about itself, which is the expensive + // case. These are the real shapes: `hyper_util`'s pool line carries + // the collector authority, and both are emitted on every flush, so + // exporting them makes each flush the cause of the next one. Emitted + // synthetically as well as by the live client, so the assertion does + // not depend on the timing of a real connection. + tracing::debug!( + target: "hyper_util::client::legacy::pool", + "pooling idle connection for otel-diagnostics-http-stack" + ); + tracing::debug!(target: "reqwest::connect", "starting new connection 'otel-diagnostics-http-stack'"); + tracing::debug!(target: "rustls::client::hs", "otel-diagnostics-http-stack"); + tracing::debug!(target: "h2::codec::framed_write", "otel-diagnostics-http-stack"); + tracing::debug!(target: "tower::buffer::worker", "otel-diagnostics-http-stack"); + + let bodies = { + let mut found = Vec::new(); + for _ in 0..80 { + found = posted.lock().expect("the posted-bodies lock").clone(); + if !found.is_empty() { + break; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + assert!( + !found.is_empty(), + "no diagnostics export reached the collector within 8s" + ); + found + }; + + let records: Vec<_> = bodies.iter().flat_map(|body| decode(body)).collect(); + let probe = records + .iter() + .find(|(body, _)| body.contains("otel-diagnostics-probe")) + .expect("the diagnostic must be exported"); + assert!( + probe.0.contains("answer=42"), + "an event's fields must survive into the body, not just its message: {:?}", + probe.0 + ); + assert!( + probe + .1 + .iter() + .any(|(k, v)| k == "bugwarden.stream" && v == "diagnostics"), + "the diagnostics stream must be tagged apart from the audit one: {:?}", + probe.1 + ); + assert!( + probe.1.iter().any(|(k, _)| k == "log.target"), + "the emitting target must ride along: {:?}", + probe.1 + ); + + assert!( + !records + .iter() + .any(|(body, _)| body.contains("otel-diagnostics-self-target")), + "the exporter's own diagnostics must never be exported: {records:?}" + ); + assert!( + !records + .iter() + .any(|(body, _)| body.contains("otel-diagnostics-before-start")), + "an event emitted before the pipeline existed cannot be exported: {records:?}" + ); + + // The export stack's own chatter never rides the export. Checked two + // ways: by the marker the synthetic events carry, and by the target + // attribute of every record that arrived — which also catches the + // LIVE client's events, whose text this test does not control. + assert!( + !records + .iter() + .any(|(body, _)| body.contains("otel-diagnostics-http-stack")), + "the export's own HTTP stack must never be exported: {records:?}" + ); + for (body, attrs) in &records { + let target = attrs + .iter() + .find(|(k, _)| k == "log.target") + .map(|(_, v)| v.as_str()) + .unwrap_or_default(); + assert!( + ![ + "bugwarden::otel", + "reqwest", + "hyper", + "rustls", + "h2", + "tower" + ] + .iter() + .any(|prefix| target.starts_with(prefix)), + "a record from the export's own stack reached the collector: \ + target {target:?}, body {body:?}" + ); + } + + // And the loop is dead, not merely quiet: with the export stack's + // events exported, every flush logs and those lines become the next + // batch, so an idle server posts forever at the batch interval. Four + // intervals of silence is the evidence that nothing feeds itself. + let settled = posted.lock().expect("the posted-bodies lock").len(); + tokio::time::sleep(Duration::from_millis(2500)).await; + let after_idle = posted.lock().expect("the posted-bodies lock").len(); + assert_eq!( + after_idle, + settled, + "an idle server must stop exporting; {} further request(s) means a flush \ + is producing the records for the next one", + after_idle - settled + ); + + pipeline.shutdown().await; +} diff --git a/crates/bugwarden/tests/otel_wiremock.rs b/crates/bugwarden/tests/otel_wiremock.rs new file mode 100644 index 0000000..336bbea --- /dev/null +++ b/crates/bugwarden/tests/otel_wiremock.rs @@ -0,0 +1,770 @@ +//! End-to-end tests of the OTLP audit export (issue #31). +//! +//! Each test drives a real [`BugWarden`] over an in-memory duplex MCP +//! transport against a mock Bugzilla, with the audit sink writing a real +//! JSONL file and an exporter pointed at a mock OTLP collector. The +//! assertions read the bytes that arrived at the collector and decode them +//! as protobuf, so the wire format is proven on the wire rather than +//! against the encoder's own idea of it. +//! +//! The diagnostics half of the export lives in `otel_diagnostics.rs`, +//! which needs a process-wide subscriber of its own. + +use std::collections::BTreeMap; +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + +use bugwarden::audit::{AuditConfig, AuditSink, AuditState, FailMode}; +use bugwarden::config::Cli; +use bugwarden::otel::{OtelEnv, Pipeline}; +use bugwarden::server::{BugWarden, USER_AGENT}; +use bugwarden_core::client::BugzillaClient; +use bugwarden_core::guard::Guard; +use bugwarden_core::policy::Policy; +use clap::Parser as _; +use rmcp::model::{CallToolRequestParams, CallToolResult}; +use rmcp::service::{RoleClient, RunningService}; +use rmcp::ServiceExt as _; +use serde_json::{json, Value}; +use wiremock::matchers::{method, path, query_param}; +use wiremock::{Mock, MockServer, ResponseTemplate}; + +const TRACEPARENT: &str = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; +const TRACE_ID: [u8; 16] = [ + 0x0a, 0xf7, 0x65, 0x19, 0x16, 0xcd, 0x43, 0xdd, 0x84, 0x48, 0xeb, 0x21, 0x1c, 0x80, 0x31, 0x9c, +]; +const SPAN_ID: [u8; 8] = [0xb7, 0xad, 0x6b, 0x71, 0x69, 0x20, 0x33, 0x31]; + +// --------------------------------------------------------------------------- +// A minimal protobuf reader, so the assertions read the wire and not the +// encoder. Field numbers come from the OTLP logs schema. +// --------------------------------------------------------------------------- + +fn read_varint(buf: &[u8]) -> Option<(u64, usize)> { + let mut value = 0u64; + for (i, byte) in buf.iter().enumerate().take(10) { + value |= u64::from(byte & 0x7f) << (7 * i); + if byte & 0x80 == 0 { + return Some((value, i + 1)); + } + } + None +} + +/// `(field number, wire type, payload)` for every field in `buf`. +fn fields(buf: &[u8]) -> Vec<(u32, u32, Vec)> { + let mut out = Vec::new(); + let mut i = 0usize; + while i < buf.len() { + let (tag, used) = read_varint(&buf[i..]).expect("a field tag"); + i += used; + let field = u32::try_from(tag >> 3).expect("a field number"); + let wire_type = u32::try_from(tag & 7).expect("a wire type"); + match wire_type { + 0 => { + let (value, used) = read_varint(&buf[i..]).expect("a varint"); + out.push((field, wire_type, value.to_le_bytes().to_vec())); + i += used; + } + 1 => { + out.push((field, wire_type, buf[i..i + 8].to_vec())); + i += 8; + } + 2 => { + let (len, used) = read_varint(&buf[i..]).expect("a length"); + i += used; + let len = usize::try_from(len).expect("a sane length"); + out.push((field, wire_type, buf[i..i + len].to_vec())); + i += len; + } + 5 => { + out.push((field, wire_type, buf[i..i + 4].to_vec())); + i += 4; + } + other => panic!("unexpected protobuf wire type {other}"), + } + } + out +} + +fn first(buf: &[u8], field: u32) -> Option> { + fields(buf) + .into_iter() + .find(|(f, _, _)| *f == field) + .map(|(_, _, v)| v) +} + +fn every(buf: &[u8], field: u32) -> Vec> { + fields(buf) + .into_iter() + .filter(|(f, _, _)| *f == field) + .map(|(_, _, v)| v) + .collect() +} + +/// One decoded OTel log record. +#[derive(Debug, Clone)] +struct Record { + body: String, + attrs: BTreeMap, + trace_id: Option>, + span_id: Option>, + severity: u64, + service_name: String, +} + +/// Decode an `ExportLogsServiceRequest` into its log records. +fn decode_records(payload: &[u8]) -> Vec { + let mut out = Vec::new(); + for resource_logs in every(payload, 1) { + let service_name = first(&resource_logs, 1) + .and_then(|resource| { + every(&resource, 1).into_iter().find_map(|kv| { + let key = String::from_utf8(first(&kv, 1)?).ok()?; + (key == "service.name") + .then(|| String::from_utf8(first(&first(&kv, 2)?, 1)?).ok()) + .flatten() + }) + }) + .unwrap_or_default(); + for scope_logs in every(&resource_logs, 2) { + for record in every(&scope_logs, 2) { + let mut attrs = BTreeMap::new(); + for kv in every(&record, 6) { + let key = String::from_utf8(first(&kv, 1).expect("an attribute key")) + .expect("a utf-8 key"); + let any = first(&kv, 2).expect("an attribute value"); + let value = match (first(&any, 1), first(&any, 3)) { + (Some(s), _) => String::from_utf8(s).expect("a utf-8 value"), + (None, Some(i)) => { + u64::from_le_bytes(i.try_into().expect("8 bytes")).to_string() + } + _ => String::new(), + }; + attrs.insert(key, value); + } + out.push(Record { + body: String::from_utf8( + first(&first(&record, 5).expect("a body"), 1).expect("a string body"), + ) + .expect("a utf-8 body"), + attrs, + trace_id: first(&record, 9), + span_id: first(&record, 10), + severity: first(&record, 2) + .map(|v| u64::from_le_bytes(v.try_into().expect("8 bytes"))) + .unwrap_or_default(), + service_name: service_name.clone(), + }); + } + } + } + out +} + +// --------------------------------------------------------------------------- +// Harness +// --------------------------------------------------------------------------- + +/// A served, audited, exporting server plus the handles assertions need. +struct Exported { + client: RunningService, + audit: Arc, + pipeline: Option>, + /// The audit file, `None` for an OTLP-only (fileless) sink. + audit_path: Option, + dir: tempfile::TempDir, +} + +impl Exported { + /// The audit file of a file-bearing server. + fn file(&self) -> &std::path::Path { + self.audit_path + .as_deref() + .expect("this server has an audit file") + } +} + +/// [`server_with_sinks`] in the shape most tests need: a file, an +/// exporter when `endpoint` names a collector, and the `open` fail mode. +async fn exporting_server(mock: &MockServer, endpoint: Option<&str>) -> Exported { + server_with_sinks(mock, endpoint, true, FailMode::Open).await +} + +/// Build a server with the given sink shape: an audit file or a fileless +/// (OTLP-only) sink, an exporter when `endpoint` names a collector, and +/// the fail mode the audit gate applies. +async fn server_with_sinks( + mock: &MockServer, + endpoint: Option<&str>, + with_file: bool, + fail_mode: FailMode, +) -> Exported { + let dir = tempfile::tempdir().expect("tempdir"); + let (audit_cfg, audit_path) = if with_file { + let path = dir.path().join("audit.jsonl"); + ( + AuditConfig { + path: Some(path.clone()), + fsync: false, + fail_mode: None, + rotate_max_bytes: 0, + rotate_keep: 8, + suppressed_ids: true, + }, + Some(path), + ) + } else { + (AuditConfig::fileless(), None) + }; + let sink = AuditSink::open(audit_cfg).expect("audit sink must open"); + + let pipeline = endpoint.map(|endpoint| { + let cfg = bugwarden::otel::resolve(&OtelEnv { + endpoint: Some(endpoint.to_string()), + ..OtelEnv::default() + }) + .expect("the endpoint must resolve") + .expect("an endpoint means export is on"); + Arc::new(Pipeline::start(cfg).expect("the pipeline must start")) + }); + let sink = match &pipeline { + Some(pipeline) => sink.with_export(pipeline.audit_exporter()), + None => sink, + }; + let audit = Arc::new(AuditState::new(sink, fail_mode, None)); + + let mut cli = Cli::parse_from([ + "bugwarden", + "--bugzilla-server", + &mock.uri(), + "--transport", + "stdio", + "--api-key", + "test-key", + ]); + cli.api_key_file = None; + let cfg = Arc::new(cli); + let guard = Arc::new(Guard { + policy: Policy::default(), + }); + let bz = + Arc::new(BugzillaClient::new(&mock.uri(), false, USER_AGENT).expect("client must build")); + let server = BugWarden::new(cfg, guard, bz) + .expect("server must build") + .with_audit(audit.clone()); + + let (client_io, server_io) = tokio::io::duplex(1 << 16); + tokio::spawn(async move { + if let Ok(running) = server.serve(server_io).await { + let _ = running.waiting().await; + } + }); + let client = ().serve(client_io).await.expect("MCP handshake must succeed"); + Exported { + client, + audit, + pipeline, + audit_path, + dir, + } +} + +/// All text blocks of a result, concatenated. +fn text_of(result: &CallToolResult) -> String { + result + .content + .iter() + .filter_map(|c| c.as_text()) + .map(|t| t.text.as_str()) + .collect() +} + +/// Wait (bounded) until the audit sink's `failing()` reads `want`, so a +/// broken health transition fails the test instead of hanging the suite. +async fn wait_failing(audit: &Arc, want: bool) { + for _ in 0..80 { + if audit.sink.failing() == want { + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + panic!("the audit sink did not reach failing()={want} within 8s"); +} + +fn world_bug(id: u64) -> Value { + json!({ + "id": id, + "summary": "a plain bug", + "product": "openSUSE", + "component": "Kernel", + "status": "NEW", + "severity": "normal", + "priority": "P3", + "keywords": [], + "groups": [], + "whiteboard": "", + "creation_time": "2020-01-01T00:00:00Z", + }) +} + +async fn mount_bugzilla(mock: &MockServer) { + Mock::given(method("GET")) + .and(path("/rest/bug")) + .and(query_param("id", "0")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ "bugs": [] }))) + .mount(mock) + .await; + Mock::given(method("GET")) + .and(path("/rest/bug")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ "bugs": [world_bug(7)] }))) + .mount(mock) + .await; +} + +async fn mount_collector(collector: &MockServer) { + Mock::given(method("POST")) + .and(path("/v1/logs")) + .respond_with(ResponseTemplate::new(200)) + .mount(collector) + .await; +} + +/// Call a tool, optionally stamping the request `_meta` with a +/// `traceparent` the way a traced client does. +async fn call( + client: &RunningService, + tool: &str, + args: Value, + traceparent: Option<&str>, +) -> CallToolResult { + let Value::Object(args) = args else { + panic!("tool arguments must be a JSON object"); + }; + let mut params = CallToolRequestParams::new(tool.to_string()).with_arguments(args); + if let Some(traceparent) = traceparent { + let mut meta = rmcp::model::RequestMetaObject::new(); + meta.set_traceparent(traceparent); + params.meta = Some(meta); + } + client + .call_tool(params) + .await + .expect("tool call must not be a protocol error") +} + +/// Wait until the collector has been posted to, and return everything it +/// received. Bounded, so a broken exporter fails the test instead of +/// hanging the suite. +async fn exported(collector: &MockServer) -> Vec { + for _ in 0..80 { + let requests = collector + .received_requests() + .await + .expect("request recording is on"); + if !requests.is_empty() { + return requests; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + panic!("no OTLP export reached the collector within 8s"); +} + +fn audit_lines(path: &std::path::Path) -> Vec { + std::fs::read_to_string(path) + .expect("audit file must be readable") + .lines() + .filter(|l| !l.is_empty()) + .map(str::to_owned) + .collect() +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn a_tool_call_is_exported_as_an_otlp_log_record() { + let mock = MockServer::start().await; + mount_bugzilla(&mock).await; + let collector = MockServer::start().await; + mount_collector(&collector).await; + + let served = exporting_server(&mock, Some(&collector.uri())).await; + call( + &served.client, + "bug_info", + json!({ "bug_ids": [7] }), + Some(TRACEPARENT), + ) + .await; + + let requests = exported(&collector).await; + for request in &requests { + assert_eq!( + request + .headers + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/x-protobuf"), + "OTLP/HTTP protobuf is the declared transport" + ); + assert!(!request.body.is_empty(), "an empty export carries nothing"); + } + + let records: Vec<_> = requests + .iter() + .flat_map(|r| decode_records(&r.body)) + .collect(); + let call_record = records + .iter() + .find(|r| r.attrs.get("bugwarden.event").map(String::as_str) == Some("tool_call")) + .expect("the tool call must be exported"); + + assert_eq!( + call_record + .attrs + .get("bugwarden.stream") + .map(String::as_str), + Some("audit") + ); + assert_eq!( + call_record.attrs.get("bugwarden.tool").map(String::as_str), + Some("bug_info") + ); + assert_eq!( + call_record + .attrs + .get("bugwarden.verdict") + .map(String::as_str), + Some("served") + ); + assert_eq!( + call_record + .attrs + .get("bugwarden.transport") + .map(String::as_str), + Some("stdio") + ); + assert!( + call_record.attrs.contains_key("bugwarden.seq"), + "records carry the sequence number that orders them: {:?}", + call_record.attrs + ); + assert_eq!(call_record.service_name, "bugwarden"); + assert_eq!(call_record.severity, 9, "a tool call is an INFO record"); + + // The client's own trace ids, as raw bytes, so a collector can join + // this record to the client trace that caused it. + assert_eq!( + call_record.trace_id.as_deref(), + Some(&TRACE_ID[..]), + "the traceparent's trace id must reach the wire as bytes" + ); + assert_eq!(call_record.span_id.as_deref(), Some(&SPAN_ID[..])); + + // The handshake is a record too, and it is exported like any other. + assert!( + records + .iter() + .any(|r| r.attrs.get("bugwarden.event").map(String::as_str) == Some("initialize")), + "the initialize record must be exported as well" + ); +} + +#[tokio::test] +async fn the_exported_body_is_the_file_record_byte_for_byte() { + let mock = MockServer::start().await; + mount_bugzilla(&mock).await; + let collector = MockServer::start().await; + mount_collector(&collector).await; + + let served = exporting_server(&mock, Some(&collector.uri())).await; + call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + + let requests = exported(&collector).await; + let records: Vec<_> = requests + .iter() + .flat_map(|r| decode_records(&r.body)) + .collect(); + let call_record = records + .iter() + .find(|r| r.attrs.get("bugwarden.event").map(String::as_str) == Some("tool_call")) + .expect("the tool call must be exported"); + + let lines = audit_lines(served.file()); + let file_line = lines + .iter() + .find(|l| l.contains("\"event\":\"tool_call\"")) + .expect("the file must hold the tool_call record"); + assert_eq!( + call_record.body.as_bytes(), + file_line.as_bytes(), + "the exported payload must carry exactly what the file carries (I12)" + ); + // And nothing else: no second copy, no re-serialization with extra + // fields. + assert_eq!( + records + .iter() + .filter(|r| r.attrs.get("bugwarden.event").map(String::as_str) == Some("tool_call")) + .count(), + 1, + "one record per call, in the export as in the file" + ); +} + +/// A port nothing listens on: bind it, learn the number, drop it. +fn dead_endpoint() -> String { + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("a free port"); + let addr = listener.local_addr().expect("an address"); + drop(listener); + format!("http://{addr}") +} + +#[tokio::test] +async fn a_dead_collector_fails_the_sink_and_open_mode_accounts_with_gaps() { + // Revised 2026-08-18: a configured collector is load-bearing. Under + // the `open` fail mode (the stdio default) serving continues — as it + // does for a failing FILE — but the sink reports failure and the + // undelivered window lands in `audit_gap` records, never silently. + let mock = MockServer::start().await; + mount_bugzilla(&mock).await; + let served = exporting_server(&mock, Some(&dead_endpoint())).await; + + let result = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert_ne!( + result.is_error, + Some(true), + "the open fail mode keeps serving through a collector outage" + ); + // The failed delivery puts the SINK into failure — the same state a + // failed file write produces, feeding the same gate. + wait_failing(&served.audit, true).await; + + // A later call is still served under `open`, and its record's write + // is preceded by an audit_gap accounting for the undelivered copies. + let again = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert_ne!(again.is_error, Some(true)); + let lines = audit_lines(served.file()); + assert!( + lines + .iter() + .filter(|l| l.contains("\"event\":\"tool_call\"")) + .count() + >= 2, + "the file keeps every record through an export outage: {lines:?}" + ); + assert!( + lines + .iter() + .any(|l| l.contains("\"event\":\"audit_gap\"") && l.contains("\"write_error\"")), + "the undelivered window must be accounted in an audit_gap: {lines:?}" + ); + // Audit losses are gap-accounted, not drop-counted: the drop counter + // is the diagnostics stream's alone. + let pipeline = served.pipeline.clone().expect("export is on"); + assert_eq!( + pipeline.dropped(), + 0, + "audit records must never ride the diagnostics drop counter" + ); + + // Shutdown returns rather than waiting on a collector that is gone. + let started = std::time::Instant::now(); + pipeline.shutdown().await; + assert!( + started.elapsed() < Duration::from_secs(30), + "the shutdown flush must be bounded" + ); +} + +#[tokio::test] +async fn a_mid_serve_collector_death_gates_uniformly_and_recovery_clears() { + // The task list's core scenario: collector dies while serving under + // `closed_all` (the http default) → uniform-text refusals; collector + // recovers → the gate reopens and the gap record accounts the window. + let mock = MockServer::start().await; + mount_bugzilla(&mock).await; + let collector = MockServer::start().await; + mount_collector(&collector).await; + + let served = server_with_sinks(&mock, Some(&collector.uri()), true, FailMode::ClosedAll).await; + let ok = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert_ne!(ok.is_error, Some(true), "a healthy collector serves"); + exported(&collector).await; + + // The collector dies mid-serve: every post is now refused (404). + collector.reset().await; + // Force a delivery attempt so the outage is noticed without waiting + // for organic traffic (served or refused depending on how the reset + // races the last flush — either way it queues delivery work), then + // wait for the sink to report failure. + let _racing = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + wait_failing(&served.audit, true).await; + + // Refused with the tool's uniform failure text — the same wording a + // failing FILE produces, chosen by tool name alone (no fingerprint). + let refused = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert_eq!(refused.is_error, Some(true)); + assert_eq!( + text_of(&refused), + "Failed to fetch bug information", + "an OTLP outage must reuse the audit gate's uniform refusal text" + ); + + // Recovery. The refusal above was recorded and queued; once the + // collector answers again the queue flushes, delivery health clears, + // and — exactly as with a recovered file — the first call after + // recovery writes the audit_gap and reopens the gate. + mount_collector(&collector).await; + let edge = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + // The edge call may still be refused (it is what carries the gap + // record out); after it the sink must clear within the batch bound. + let _ = edge; + wait_failing(&served.audit, false).await; + let after = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert_ne!( + after.is_error, + Some(true), + "recovered delivery must reopen the gate" + ); + let lines = audit_lines(served.file()); + assert!( + lines + .iter() + .any(|l| l.contains("\"event\":\"audit_gap\"") && l.contains("\"write_error\"")), + "the undelivered window must be accounted in an audit_gap: {lines:?}" + ); +} + +#[tokio::test] +async fn an_otlp_only_server_serves_and_creates_no_file() { + let mock = MockServer::start().await; + mount_bugzilla(&mock).await; + let collector = MockServer::start().await; + mount_collector(&collector).await; + + // Fileless sink under the strictest fail mode: with the collector + // healthy, serving works and every record goes to the collector. + let served = server_with_sinks(&mock, Some(&collector.uri()), false, FailMode::ClosedAll).await; + let result = call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert_ne!(result.is_error, Some(true), "an OTLP-only server serves"); + + let requests = exported(&collector).await; + let records: Vec<_> = requests + .iter() + .flat_map(|r| decode_records(&r.body)) + .collect(); + assert!( + records + .iter() + .any(|r| r.attrs.get("bugwarden.event").map(String::as_str) == Some("tool_call")), + "the tool call must reach the collector" + ); + assert!( + records + .iter() + .any(|r| r.attrs.get("bugwarden.event").map(String::as_str) == Some("initialize")), + "the handshake must reach the collector" + ); + + // No file created anywhere: the sink has no path and the working + // directory it could have written into holds nothing. + assert!(served.audit_path.is_none()); + assert_eq!( + std::fs::read_dir(served.dir.path()) + .expect("the tempdir is readable") + .count(), + 0, + "an OTLP-only sink must touch no filesystem" + ); +} + +#[tokio::test] +async fn the_startup_probe_retries_until_a_racing_collector_answers() { + // A collector that starts alongside the server may lose the race by + // a few seconds; the probe's bounded retry absorbs that, and only a + // collector that never answers refuses startup (unit-tested in + // `otel.rs` with the refusal's wording). + let collector = MockServer::start().await; + Mock::given(method("POST")) + .and(path("/v1/logs")) + .respond_with(ResponseTemplate::new(503)) + .up_to_n_times(2) + .mount(&collector) + .await; + mount_collector(&collector).await; + + let cfg = bugwarden::otel::resolve(&OtelEnv { + endpoint: Some(collector.uri()), + ..OtelEnv::default() + }) + .expect("the endpoint must resolve") + .expect("an endpoint means export is on"); + let pipeline = Arc::new(Pipeline::start(cfg).expect("the pipeline must start")); + pipeline + .probe() + .await + .expect("bounded retry must outlast a boot race"); + assert!( + collector + .received_requests() + .await + .expect("request recording is on") + .len() + >= 3, + "the probe must have retried through the failures" + ); + pipeline.shutdown().await; +} + +#[tokio::test] +async fn without_an_endpoint_nothing_is_exported() { + // The off switch is the absence of a configuration: `resolve` returns + // nothing, so main builds no pipeline, and a sink with no exporter + // attached reaches no collector even when one is listening. + assert!( + bugwarden::otel::resolve(&OtelEnv::default()) + .expect("an empty environment resolves") + .is_none(), + "no endpoint must mean no export configuration" + ); + assert!( + bugwarden::otel::resolve(&OtelEnv { + endpoint: Some(String::new()), + ..OtelEnv::default() + }) + .expect("an emptied endpoint resolves") + .is_none(), + "an emptied endpoint must mean no export configuration" + ); + + let mock = MockServer::start().await; + mount_bugzilla(&mock).await; + let collector = MockServer::start().await; + mount_collector(&collector).await; + + let served = exporting_server(&mock, None).await; + assert!(served.pipeline.is_none(), "no exporter is attached"); + call(&served.client, "bug_info", json!({ "bug_ids": [7] }), None).await; + assert!( + audit_lines(served.file()) + .iter() + .any(|l| l.contains("\"event\":\"tool_call\"")), + "the record is still written to the file" + ); + + // Long enough that an exporter running at the batch interval would + // have sent something. + tokio::time::sleep(Duration::from_millis(1500)).await; + assert!( + collector + .received_requests() + .await + .expect("request recording is on") + .is_empty(), + "with export off the collector must never be contacted" + ); +} diff --git a/docs/DESIGN.md b/docs/DESIGN.md index d76f0a6..a5634b9 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -96,14 +96,20 @@ Dependency direction: `bugwarden -> bugwarden-core`, never the reverse. least `summary` on `duplicate_of`. - **I12** The Bugzilla API key must never appear in logs, error messages, or tool results. Sanitize reqwest errors with `.without_url()` — the key may be - a URL query parameter. + a URL query parameter. The rule covers every secret the process holds, and + since 2026-08-18 that includes the OTLP export credential + (`OTEL_EXPORTER_OTLP_HEADERS`). The collector endpoint is held to a + weaker bar (never in an error, audit record, or a line this crate + writes; the HTTP stack may print the authority at `RUST_LOG=debug`): + see "OTLP export". - **I13** In read-only mode (policy or CLI) write tools are removed from the tool listing via `ToolRouter::remove_route`, not merely erroring. Same for `global.disabled_tools`. - **I15** The audit stream is never reachable through any MCP surface. When auditing is enabled, every tool call produces exactly one audit - record, persisted before the response is returned. The API key and - free-text bug content are unrepresentable in the audit event type. + record, accepted by every configured sink before the response is + returned (a file write, an export-queue accept, or both). The API key + and free-text bug content are unrepresentable in the audit event type. Client-visible responses are byte-identical with auditing on, off, or failing — except the scoped fail-closed refusals, which reuse the tools' existing uniform failure texts and never vary with the guard's verdict. @@ -1115,7 +1121,7 @@ clap derive `Cli`, with env fallbacks: | --use-auth-header | BUGZILLA_USE_AUTH_HEADER | false | Bearer to Bugzilla instead of api_key query param | | --read-only | MCP_READ_ONLY | false | tighten-only (I9) | | --policy | BUGWARDEN_POLICY | — | path to guard policy TOML; the container image presets it to /etc/bugwarden/policy.toml, so that one artifact fails closed on a missing mount instead of defaulting to allow-all (tightening only, I9) | -| --audit-config | BUGWARDEN_AUDIT_CONFIG | — | path to audit configuration TOML; without it no audit stream is written | +| --audit-config | BUGWARDEN_AUDIT_CONFIG | — | path to audit configuration TOML, or the exact value `none` to disable the file (OTLP-only when an endpoint is set). Unset with no OTLP endpoint writes no stream; an endpoint with no file decision, or `none` with no endpoint, is a startup error | | --insecure-no-auth | — (deliberately) | false | serve http with no bearer gate; refuses to start together with a token (see HTTP bearer authentication) | | — | BUGWARDEN_HTTP_TOKEN | — | bearer token, write scope; environment only, never a flag | | — | BUGWARDEN_HTTP_READ_TOKEN | — | bearer token, read scope; environment only, never a flag | @@ -1130,8 +1136,10 @@ is not a hostname or `host:port` and emits one info line stating whether Host validation is on or off (the list, when on — hosts only, I12). main.rs adds: the http bearer gate, resolved FIRST of all (see HTTP bearer authentication) so a token misconfiguration precedes every other -startup effect; and http without audit_config => tracing::warn (remote -tool calls leave no audit record). +startup effect; `select_sinks` (file path, `none`, OTLP endpoint) so the +two ambiguous spellings refuse before any file or task exists; and http +with no sink at all (`NoAudit`) => tracing::warn (remote tool calls leave +no audit record). ## Audit stream (crates/bugwarden/src/audit.rs + the server.rs wrapper) @@ -1144,13 +1152,18 @@ Decisions, all deliberate: extensions (verdict worst-wins merged, suppressed ids unioned). An unknown tool, a protocol error, or a missed enrichment still yields exactly one record — a poorer record is possible, an audit gap is not. - The record is persisted (sink is synchronous) before the response is - returned. `initialize` is always recorded, with no configuration knob - to turn it off; `list_tools` is not recorded in schema v1 — no event - kind exists for a listing, deliberately. -- **Boundary.** Records go only to the operator's JSONL file (0600, - parent 0700) — never stderr, never any MCP surface. The schema has no - field that could carry the API key or free-text bug content; client + The record is accepted by every configured sink before the response is + returned (the file write is synchronous; the OTLP hand-off only queues). + `initialize` is always recorded, with no configuration knob to turn it + off; `list_tools` is not recorded in schema v1 — no event kind exists + for a listing, deliberately. +- **Boundary.** Records go to the sinks the operator named + (`select_sinks`): the JSONL file (0600, parent 0700), an OTLP + collector, both, or neither. Never stderr, never any MCP surface. When + both run, the file write precedes the export, so the file never lacks a + record the collector has. A configured collector is load-bearing, not a + copy ("OTLP export", below). The schema has no field that could carry + the API key or free-text bug content; client parameters pass a key allowlist (identifiers and routing/vocabulary fields by value, strings capped at 1024 chars) and every other key is recorded as `{"_len": N}` — presence and size, never content. @@ -1346,6 +1359,216 @@ Decisions, all deliberate: field, or a prefix), was rejected here: it is a record-schema change and belongs to #34. +## OTLP export (crates/bugwarden/src/otel.rs) + +Added 2026-08-18 (issue #31). Revised the same day: a configured collector +is a load-bearing audit sink, not a best-effort copy. Until then the audit +boundary read "the guard does not speak OTLP, does not export anything +itself"; that sentence is SUPERSEDED. The operator chooses the sinks +(`select_sinks`): the JSONL file, an OTLP collector, both, or neither. +Every configured sink is load-bearing — a failed export puts the sink into +failure exactly as a failed file write does, and the operator's +`fail_mode` decides what the server then does about tool calls. Only the +diagnostics stream stays best-effort. Off entirely unless an operator +names an endpoint. + +- **Decision: export natively, do not only document it.** The original + issue proposed a collector example alone, on the reasoning that shipping + the file off-host is what makes it tamper-evident and that shipping is an + operator task. Tailing still ships (`examples/otel-collector.yaml` carries + both receivers) and is still the tamper-evidence path; native export + answers the other half, which tailing cannot: a live stream, correlated + with the client's trace ids (#28), from a process that already knows the + verdict, the rule and the session. The two are complementary and the + example runs both — the exported body IS the file line (or the line the + file would have carried), so a consumer deduplicates on `session` plus + `seq`. +- **Sink selection.** Turning the FILE off is an explicit act + (`BUGWARDEN_AUDIT_CONFIG=none`, compared by exact bytes so `./none` is + still a path), never an inference from absence. The four deployments + are: both knobs unset → no audit trail and no audit gate; a file path + alone → every pre-#31 deployment; a file path and an endpoint → both + sinks, file first; `none` and an endpoint → OTLP-only, no filesystem + touch. The two cells that could confuse absence with a choice are + startup errors: an endpoint with no file decision, and `none` with no + endpoint. An OTLP-only sink has no document, so `fail_mode` always + derives from the transport. +- **What is exported.** One OTel log record per `AuditEvent` the sink + persisted, plus the server's own `tracing` diagnostics; the two are told + apart by the `bugwarden.stream` attribute (`audit` / `diagnostics`). An + audit record's body is the audit line VERBATIM — when a file is + configured, the same bytes it holds, without the terminating newline and + without the leading one a torn-line repair may have prefixed; a fileless + sink exports the line it would have written — and its attributes are + `bugwarden.event`, `.seq`, `.transport`, `.session.id`, and, on a + `tool_call`, `.tool`, `.verdict` and `.rule`. Severity follows the KIND, + not the verdict: `tool_call` and `initialize` are INFO, `audit_gap` is + ERROR, because a gap is a loss of the record stream itself while a denial + is the guard working. The verdict rides an attribute, where a consumer + can filter on it. `trace_id`/`span_id` come from the record's `trace` + field when the client sent a valid `traceparent`, so a guard decision + joins the client trace that caused it; they are the same unauthenticated + client claims the audit schema documents, correlation hints and never + evidence. +- **Byte equality (I12).** `AuditSink::write_event` hands the exporter the + exact slice it wrote (or would have written), after the file write and + `sync_data` (when a file is configured) and while the sink lock is still + held. That ordering is the whole design: when a file exists it is + authoritative, so a record it never took exists nowhere else; export + order equals `seq` order; the exported payload cannot drift from the + file's because it is not re-serialized. `AuditExport` implementors must + not block — the call is under the sink lock — and the OTLP one only + queues. A record the queue will not take is REFUSED, never dropped. +- **Failure semantics: every configured sink is load-bearing.** A + collector that is down, slow, or answering 503 marks + `AuditExport::delivery_failing`, which `AuditSink::failing()` feeds into + the same `FailMode` gate a failed file write does. Records accepted and + then not delivered are counted in `take_lost` and surface as an + `audit_gap` (reason `write_error`); a full queue or a shut-down pipeline + refuses `accept` and the request path treats that as `AuditError::Export`. + Delivery health is a latch — false from the first failed request until + a successful AUDIT flush; a successful diagnostics flush does not + reopen it — so the gate does not flap once per batch for the whole + outage, and a log line getting through is not proof the audit sink + works. Two queues keep the streams apart (2048 each): a log storm must + not fill the audit queue and take the server down, and a dropped + diagnostic must not stop the guard. The batch is bounded (512 records, + 500 ms, a 10 s request timeout); a failed AUDIT batch is lost rather + than retried (a retry queue is a second unbounded buffer in front of a + collector that is already not answering) but never silently. Diagnostic + drops are counted and the counter is logged when the total crosses a + power of two — the diagnostic has to survive an outage that lasts, and + one line per lost record would be its own denial of service. On a + fileless sink the persist bar is acceptance onto the audit queue, not a + durable write: a crash before the next flush, or a shutdown that hits + the 5 s bound, loses the tail. That is the cost of turning the file + off. A served call's response is still byte-identical with export on, + off, or failing (I15); the refusals a closed fail mode produces are the + audit machinery's own, uniform per tool. I9 is untouched: nothing here + reaches the guard or loosens policy. +- **The drop line carries a count and a reason and nothing else.** The + reason is a closed vocabulary (`queue_full`, `network`, `http_status`, + `shutdown`) for the same purpose `GapReason` is one: a free-text reason + built from a transport error is exactly how an endpoint reaches a log + line. The `reqwest::Error` of a failed export is discarded rather than + logged, since it carries the request URL — the same rule + `.without_url()` exists for. Nothing that holds the endpoint or the + headers derives `Debug`; `Pipeline`'s is hand-written and content-free + because `AuditSink`'s derived one would otherwise print it. +- **What secrecy this actually buys, stated exactly.** The two claims are + not the same and were once written as if they were. + - The HEADERS never reach anything but the wire. No code path formats + them, no type prints them, no error names them (a malformed entry is + refused by position), and reqwest's byte-level tracing — the one thing + that would dump an `authorization:` header — is behind + `ClientBuilder::connection_verbose`, which defaults to false and which + this module never sets. + - The ENDPOINT is not secret to stderr, and claiming otherwise was + false. bugwarden itself never logs it, but at `RUST_LOG=debug` the + HTTP stack does: `hyper_util`'s pool logs "pooling idle connection for + " and `reqwest::connect` logs "starting new connection", + each carrying scheme, host and port. Suppressing those from stderr was + rejected — an operator debugging a collector that will not answer + needs exactly those lines, and the Bugzilla endpoint is already + printed at startup, so the authority is not the kind of thing this + project hides. What IS guaranteed is that the endpoint never reaches + the EXPORT, an audit record, or any line bugwarden writes itself. +- **Nothing the export emits is exported (the loop).** The diagnostics + layer drops events whose target begins with `bugwarden::otel`, + `reqwest`, `hyper` (which covers `hyper_util`), `rustls`, `h2` or + `tower`. Without this, at debug level, one flush's pool and connect + lines become records in the next batch, whose flush logs again: an idle + server exports forever at the batch interval, and the authority above + goes on the wire with it. Measured, not theorised — the test asserts + that an idle server's export count stops moving. + Two subtleties make this more than a target list. First, `reqwest` and + `hyper` log through the `log` crate, and `tracing-log`'s bridge (which + `SubscriberInitExt::init` installs) gives those events the literal + target `"log"` and demotes the real one to a `log.target` FIELD — so the + check reads that field where it exists, and a check on metadata alone + silently catches nothing. Second, the cost is accepted rather than + avoided: those same targets carry the BUGZILLA client's HTTP + diagnostics, which therefore stop being exported too. They still reach + stderr, and no filter available at the layer can tell one client's + events from the other's — they are the same crates on the same targets. +- **Configuration: the standard variables, and only those.** Endpoint, + headers, protocol and service name come from + `OTEL_EXPORTER_OTLP_ENDPOINT`, `_HEADERS`, `_PROTOCOL` and + `OTEL_SERVICE_NAME`. No command-line option exists, which matches the + container env-first convention (#104/#32) and, for `_HEADERS`, is the + same reasoning as the http bearer tokens: an option would publish the + credential through `ps` (I12). An unset OR EMPTY endpoint turns the + feature off completely — no task, no thread, no layer, no request — and + the protocol is therefore validated only once an endpoint exists, so a + fleet-wide `OTEL_*` environment cannot refuse to start a deployment that + exports nothing. `http/protobuf` is the one accepted protocol and every + other value is a startup error naming the variable; gRPC is deliberately + absent, which is what keeps the export on the rustls/reqwest stack the + Bugzilla client already resolves. The export client follows no HTTP + redirects: a 3xx would forward the audit body and any non-Authorization + collector credential to a host the operator did not name. `_HEADERS` + values are taken VERBATIM: + the OTLP specification describes them as percent-encoded, and decoding + would silently rewrite any credential containing a `%`, which is worse + than not implementing an encoding no collector requires. A malformed + entry is refused by POSITION, never by content, because a mispasted + credential is precisely what lands in the wrong position. +- **Startup and shutdown ordering.** The environment is resolved before the + subscriber is installed (parsing starts nothing, and an unusable protocol + must abort before records exist). `select_sinks` runs next, still before + any network or filesystem work, so the two ambiguous spellings refuse + without opening a file or spawning a task. The exporter task and its + delivery probe run AFTER the identity preflight and BEFORE the audit + file is created: a collector that will not take records refuses to start + without leaving a file behind (five attempts, `n × 500 ms` backoff, + posting one real diagnostics record — not an empty request some + collectors accept blindly, and not a fake audit event the schema does + not have). The diagnostics layer is installed at subscriber time but + reads an empty slot until the pipeline starts, so events emitted during + startup reach stderr and not the collector. At shutdown the queue is + flushed best-effort under a 5 s bound; over http that runs after + graceful shutdown, i.e. after the SIGINT/SIGTERM that triggered it, and + the flush is reached even when the transport returned an error. Over + stdio a signal cannot return from `main` (the stdin read is + uncancellable, issue #114), so those arms flush and then + `process::exit(0)` rather than skipping a load-bearing collector. +- **Dependency decision: hand-written encoder over the OpenTelemetry SDK.** + The SDK route (`opentelemetry` + `opentelemetry_sdk` + + `opentelemetry-otlp` + `opentelemetry-appender-tracing`, 0.32) was + resolved, built and run before being rejected. Three findings, in order + of weight. (1) Its `BatchLogProcessor` drives the export from a dedicated + OS thread with `futures_executor::block_on`, so the async reqwest client + panics on the first export with "there is no reactor running, must be + called from the context of a Tokio 1.x runtime" and the processor is dead + for the process lifetime; the supported combination is + `reqwest-blocking-client`, i.e. a SECOND internal Tokio runtime and + thread inside a process that already has one. (2) + `opentelemetry-appender-tracing` 0.32 does not honour the telemetry + suppression the SDK sets around its own export path, so the SDK's + internal `otel_error!` diagnostics — emitted through `tracing` on every + failed batch — are picked up by the appender and queued for export, + making a failing exporter its own source of records to export. (3) It + cannot produce the drop accounting this design asks for: `on_emit` + returns nothing, so a full queue is invisible to the caller, and the + processor logs the transport error itself, with the endpoint URL at debug + level. Against that, the whole OTLP logs schema this module needs is one + request message, four nested messages and a varint writer, and going + without adds ZERO crates to the lock file for a security-guard product — + the SDK route added 17 (`prost` and `prost-derive`, + `opentelemetry-proto`, and their own transitive tail), proc macros among + them. `cargo deny` therefore has nothing new to judge, and `Cargo.lock` + is unchanged by this feature. The cost is owned protobuf encoding, which the + unit tests pin field by field and the integration tests decode off the + wire. +- **Invariants.** I15 is untouched: the pipeline is reachable through no + MCP tool, resource or prompt, and no client can turn it on, off, or + inspect it. I9 is untouched: nothing here reaches the guard or loosens + policy. A served call's response is byte-identical with export on, off, + or failing (I15); the refusals a closed fail mode produces are the audit + gate's, uniform per tool. I12 gains the export headers as secret material + of the kind the invariant already names; the endpoint is the weaker bar + stated under "What secrecy this actually buys". + ## rmcp 3.1 usage notes Reference source is the rmcp this workspace pins, unpacked in the local @@ -1629,7 +1852,7 @@ wired, `server.rs` and `main.rs` are the reference. update_bug_fields, update_bug_dependencies, add_cc_to_bug, mark_as_duplicate, create_bug, add_attachment. - API key resolution: a match on `key_custody` (resolved once at startup, see Key custody — never re-read per request): `Server(key)` => the server's key, without touching the request at all; `PerRequest` => `ctx.extensions.get::()`, then `parts.headers.get(lowercased_header_name)`. -- HTTP serving: `let config = server.http_server_config()?.with_cancellation_token(ct.child_token());` — built while `server` can still be borrowed, since the body cap comes from its own guard policy; errors on an unparsable `--allowed-hosts` entry and logs the effective Host-validation state — then `StreamableHttpService::new(move || Ok(server.clone()), LocalSessionManager::default().into(), config)`, never a bare `StreamableHttpServerConfig::default()`, see the field table above — then `axum::Router::new().nest_service("/mcp", service)`, `tokio::net::TcpListener::bind`, graceful shutdown on SIGINT or SIGTERM cancelling `ct` (`shutdown_signal` in main.rs; issue #114). Stdio uses the same waiter across `serve` (the handshake wait an unused stdio container sits in) and `waiting`; a signal at either stage `process::exit(0)`s, because rmcp's stdio transport reads stdin via `spawn_blocking` and that read does not unblock while the client holds the pipe — returning from `main` drops the runtime onto that thread. +- HTTP serving: `let config = server.http_server_config()?.with_cancellation_token(ct.child_token());` — built while `server` can still be borrowed, since the body cap comes from its own guard policy; errors on an unparsable `--allowed-hosts` entry and logs the effective Host-validation state — then `StreamableHttpService::new(move || Ok(server.clone()), LocalSessionManager::default().into(), config)`, never a bare `StreamableHttpServerConfig::default()`, see the field table above — then `axum::Router::new().nest_service("/mcp", service)`, `tokio::net::TcpListener::bind`, graceful shutdown on SIGINT or SIGTERM cancelling `ct` (`shutdown_signal` in main.rs; issue #114). Stdio uses the same waiter across `serve` (the handshake wait an unused stdio container sits in) and `waiting`; a signal at either stage flushes the OTLP queue (when export is on) then `process::exit(0)`s, because rmcp's stdio transport reads stdin via `spawn_blocking` and that read does not unblock while the client holds the pipe — returning from `main` drops the runtime onto that thread. - Request `_meta` (SEP-414, e.g. `traceparent`): over every serialized transport the wire `params._meta` does NOT arrive in the params struct (`CallToolRequestParams.meta` stays `None`) — the SDK's custom @@ -2104,5 +2327,79 @@ wired, `server.rs` and `main.rs` are the reference. the child's stdin still held, so wrapping only `serve` or only `waiting` fails, and a handshake arm that only `return Ok(())` cannot go green via `wait_with_output` dropping the pipe. +- Unit tests (#[cfg(test)] in crates/bugwarden/src/otel.rs): configuration + resolution — an unset, emptied or blank endpoint resolves to NO + configuration and a bad protocol beside it is therefore not an error, + `http/protobuf` and an absent protocol are the only accepted transports + and every other value names the variable and the accepted spelling, the + logs path is appended once whether or not the endpoint ends in a slash, + `OTEL_SERVICE_NAME` is trimmed and defaults to `bugwarden`, headers parse + into pairs and are taken VERBATIM (percent-encoding is not decoded), and + a malformed entry — no separator, an empty or whitespace-carrying name, + an empty value, a newline in a value — is refused by POSITION with the + pasted credential absent from the error (I12) while a trailing or + doubled comma names no header and is not an error, and the logs-specific + variables override their general twins, turn export on when set alone, + fall back when emptied, and name themselves in a refusal; record + shaping — the body + is the given line unchanged, the documented attributes are all present, + severity follows the kind (`audit_gap` ERROR, the rest INFO), a valid + traceparent's ids reach the record as raw bytes and hex of the wrong + width or alphabet reaches it as nothing; protobuf encoding — varints + round-trip (including the canonical `300 = ac 02`), a request nests + resource, scope and records with the OTLP field numbers, a batch is one + request with one record each, and the trace ids land in the fixed-width + byte fields, with both timestamps pinned BY FIELD NUMBER (1 and 11, and + the reserved field 4 empty) and every severity pinned to its OTLP number + and text — a stamp or a severity written where no reader looks for it is + invisible to any assertion on the decoded value alone; drop accounting — + sixteen single drops log exactly five + lines (1, 2, 4, 8, 16), one 300-record batch crosses nine of them + (1 through 256) and still logs once with the threshold moved to 512, and + the line carries no endpoint and no header material; and `Pipeline`'s + hand-written `Debug` + carries neither, since `AuditSink`'s derived one would print it; + the startup probe refuses a dead collector without naming the endpoint + (I12) and leaves delivery marked failing. +- Sink-selection tests (#[cfg(test)] in crates/bugwarden/src/audit.rs): + the four deployments resolve, the two ambiguous spellings refuse naming + both ways out, and `none` is exact bytes (`None` and `./none` are + files). Exporter-as-sink: a fileless sink records through the exporter + alone; a refused hand-off fails the record while the file still keeps + it; delivery failure alone puts the sink in failure; undelivered losses + surface as an `audit_gap`; and the file write precedes the hand-off. +- Integration tests (crates/bugwarden/tests/otel_wiremock.rs, a wiremock + OTLP collector beside the wiremock Bugzilla): a real tool call over a + real MCP session arrives at `POST /v1/logs` as + `application/x-protobuf`, decoded off the wire rather than through the + encoder, carrying the documented attributes, `service.name`, INFO + severity and the client's traceparent ids as bytes, with the + `initialize` record exported beside it; the exported body is + byte-identical to the file's line and appears exactly once (I12); a + dead collector under `open` keeps serving, puts the sink in failure, + and accounts the window with `audit_gap` (audit losses never ride the + diagnostics drop counter), and the shutdown flush returns bounded; a + mid-serve collector death under `closed_all` refuses with the tool's + uniform failure text and recovery reopens the gate; an OTLP-only server + serves, exports initialize and the tool call, and touches no + filesystem; the startup probe retries through 503s until a racing + collector answers; and with no endpoint configured `resolve` yields + nothing, no exporter is attached, the file still gets the record and a + listening collector is never contacted. `otel_diagnostics.rs` is one + test in a binary of its own + (the layer only works through the one process-wide subscriber): a + diagnostic reaches the collector with its FIELDS as well as its message + and the `bugwarden.stream`/`log.target` attributes, an event emitted + before the pipeline existed is dropped rather than exported, and nothing + the export itself emits is exported: not this module's own target, and + not its HTTP stack's, checked both by marker and by the `log.target` of + every record that arrived, so the LIVE client's events are covered and + not only the synthetic ones. That test runs the subscriber at `debug`, + where the stack is loud enough to feed itself, and it finishes by + asserting an idle server's export count stops moving — the difference + between "the leak is plugged" and "the loop is dead". Its collector is a + hand-rolled socket rather than a `MockServer` precisely because wiremock + runs in-process and logs a line per request, which would feed the very + loop under test. - CI: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace --locked`, `cargo deny check`. diff --git a/examples/audit.toml b/examples/audit.toml index 58028f5..aba5daf 100644 --- a/examples/audit.toml +++ b/examples/audit.toml @@ -7,8 +7,9 @@ # configures the other half of that bargain — the operator's own record of # what was asked and what the guard decided. Audit records carry exactly # the facts the client must never see (guard verdicts, matched rule names, -# suppressed bug ids), which is why they go only to a local file you -# control: the stream is never exposed through any MCP surface, and it is +# suppressed bug ids). This document configures the FILE sink. Records +# may also go to a load-bearing OTLP collector (see README "OpenTelemetry +# export"); the stream is never exposed through any MCP surface, and it is # never mixed into the diagnostic stderr stream. # # Format: one JSON object per line (JSONL), schema version 1. Three record @@ -24,8 +25,11 @@ # call site; nothing fetched FROM Bugzilla is ever written here. # # bugwarden loads this file via --audit-config or the -# BUGWARDEN_AUDIT_CONFIG environment variable; without that flag no audit -# stream is written. +# BUGWARDEN_AUDIT_CONFIG environment variable. The exact value `none` +# disables the file so a collector can be the only sink. An OTLP +# endpoint with no file decision is a startup error — point here or set +# `none` — and so is `none` with no endpoint. Leaving both knobs unset +# writes no stream. # # Parsing is strict: unknown keys anywhere in this file are a startup # error, so a typo cannot silently disable a setting. diff --git a/examples/otel-collector.yaml b/examples/otel-collector.yaml new file mode 100644 index 0000000..df42852 --- /dev/null +++ b/examples/otel-collector.yaml @@ -0,0 +1,257 @@ +# OpenTelemetry Collector configuration for a bugwarden audit stream. +# +# Two receivers, because there are two ways the records can reach a +# collector and they answer different questions: +# +# otlp — bugwarden's own export (OTEL_EXPORTER_OTLP_ENDPOINT). Live, +# structured, and correlated with client traces. Load-bearing: +# a collector that will not take records refuses startup, and +# an outage after that engages the same fail_mode a failed +# file write does. +# filelog — tailing the JSONL file bugwarden writes. Lossless as far as +# the file goes — a collector that was down re-reads what it +# missed, at the price of re-reading what it did not (see +# `start_at` below) — and it is what turns the log into a +# tamper-evident record: once the lines are off the host, the +# host can no longer rewrite its own history. Slower, and +# blind to whatever the file has already rotated away. +# +# Run both. They carry the same records — the exported body IS the file +# line — so a consumer collapses the two copies on `session` plus `seq`, +# and the pair gives you live visibility plus an off-host copy. Treat that +# as near-exact rather than a key: with `fsync = true` a record whose write +# succeeded and whose sync did not is counted as dropped, so its `seq` can +# be reused by the next record and one (session, seq) can name two +# different records. Compare the bodies before collapsing, and the case +# resolves itself. +# +# Written against opentelemetry-collector-contrib 0.110+ (the `filelog` +# receiver lives in -contrib, not in the core distribution, and +# `on_error: send_quiet` needs a collector of at least that vintage). +# Validate it against your own build — `otelcol-contrib validate +# --config=...` — before deploying: this file ships checked for YAML +# well-formedness, not run against a live collector. +# +# --------------------------------------------------------------------------- +# BEFORE YOU DEPLOY THIS: what is in the stream +# --------------------------------------------------------------------------- +# +# The audit stream carries exactly the facts bugwarden refuses to tell an +# MCP client: guard verdicts, the NAMES of the policy rules that decided +# them, and the bug ids that were withheld (`guard.suppressed_ids`, on by +# default). That makes the stream at least as sensitive as the bugs it +# describes — anyone who can read it can enumerate the hidden ones — and +# its destination a policy decision, not a deployment detail. +# +# * Treat the collector, its queue, and every backend behind it as +# systems holding embargoed bug numbers. +# * `suppressed_ids = false` in audit.toml drops the ids and keeps the +# counts, if the destination cannot be trusted with them. +# * Nothing here is a substitute for the file's own permissions. +# +# --------------------------------------------------------------------------- +# BEFORE YOU DEPLOY THIS: permissions +# --------------------------------------------------------------------------- +# +# bugwarden creates the audit file mode 0600 and its parent directory 0700, +# owned by the user bugwarden runs as. A collector running as anybody else +# cannot read it, and the fix is never to loosen the file: +# +# * Run the collector as the SAME uid as bugwarden (in the container +# image that is 65532), or +# * put both in one supplementary group, `chgrp` the directory, and set +# it 0750 with the file 0640 — bugwarden re-creates the file mode 0600 +# on every rotation, so this needs a `umask`/`setgid` arrangement on +# the directory and is the more fragile of the two, or +# * give the collector a read-only bind mount of the directory and run it +# as the same uid. +# +# Do NOT make the directory world-readable. Do NOT run the collector as +# root to work around the mode: that swaps a permission problem for a +# collector process that can rewrite the record it is supposed to preserve. +# +# --------------------------------------------------------------------------- +# BEFORE YOU DEPLOY THIS: rotation +# --------------------------------------------------------------------------- +# +# bugwarden rotates by rename (`audit.jsonl` -> `audit.jsonl.1` -> …, +# `rotate_keep` files) and creates a fresh live file. The `include` glob +# below therefore matches the rotated files too: if the collector is behind +# when a rotation happens, the tail it has not read yet is in +# `audit.jsonl.1`, not in the new live file. The receiver identifies files +# by content fingerprint rather than by name, so a file it has already read +# is not re-read under its new name — that is what keeps this from +# duplicating records. +# +# Two things to keep true: +# * `rotate_keep` must be large enough that a file cannot be deleted +# before the collector has read it. The default (8 x 64 MiB) is +# generous; a collector down for a week is not. +# * Never let logrotate and bugwarden both rotate the file. If logrotate +# owns it, set `rotate_max_bytes = 0` and give logrotate `copytruncate` +# — bugwarden holds the file open and keeps appending to the same +# inode, which after a plain rename is the ROTATED file, so new records +# land in what everything else now treats as history (and vanish +# outright once logrotate finally unlinks it). + +receivers: + # bugwarden's native export. It speaks OTLP/HTTP with protobuf payloads + # and nothing else, so the grpc endpoint is deliberately absent. + otlp: + protocols: + http: + # 0.0.0.0 because the sidecar in compose.yaml is the documented + # deployment and bugwarden reaches it from ANOTHER container: a + # loopback bind there listens only inside the collector's own + # network namespace, so every record would be refused and counted + # dropped forever. This is the same reasoning the image applies to + # MCP_HOST — a container necessity, not a widening of trust — and + # it carries the same obligation: do NOT publish this port to the + # host, since the OTLP receiver has no authentication of its own. + # + # Running the collector on the host instead, beside a bugwarden + # that is not containerised? Then use 127.0.0.1:4318 here. + endpoint: 0.0.0.0:4318 + # Point OTEL_EXPORTER_OTLP_ENDPOINT at the collector's base URL — + # http://otel-collector:4318 in compose, http://127.0.0.1:4318 on + # the host. bugwarden appends /v1/logs itself. + + filelog/bugwarden-audit: + include: + - /var/log/bugwarden/audit.jsonl + - /var/log/bugwarden/audit.jsonl.[0-9]* + # From the beginning: the point of tailing is to lose nothing. The + # cost, and it is a real one: the receiver's read offsets live in + # memory unless a `file_storage` extension persists them, so a + # restarted collector replays every file it can still see and the + # backend gets duplicates — bounded by `rotate_keep`, but large. + # Deduplicate on `session` plus `seq` (see the caveat at the top), or + # add `file_storage` and point `service.extensions` at it. Note that + # the compose sidecar runs `read_only: true`, so file_storage is not a + # drop-in there: it needs a writable volume of its own. + start_at: beginning + poll_interval: 1s + # Big enough to tell two audit files apart by their first line, which + # carries a distinct `ts`/`seq` pair. + fingerprint_size: 1kb + # A failed write leaves at most ONE unparsable line per outage: either + # a torn line — the PREFIX of a record, so it starts with `{` and no + # cheap shape test can tell it from a whole one — or an empty line, + # where the failed write got no bytes out at all and the next record + # carried the healing newline alone. The `^\{` test therefore skips + # only the empty case, and the torn case is handled where it has to be: + # `on_error: send_quiet` passes an entry the parser could not read + # downstream with its body intact and without logging, instead of + # dropping it or filling the collector's own log with errors. Every + # operator after this one is guarded on the field it needs, so debris + # travels as a plain body and nothing errors on it. + operators: + - id: audit-json + type: json_parser + if: 'body matches "^\\{"' + on_error: send_quiet + parse_from: body + parse_to: attributes + timestamp: + parse_from: attributes.ts + layout_type: gotime + layout: "2006-01-02T15:04:05.999Z07:00" + + # Lift the fields worth querying on into the same attribute names the + # native OTLP export uses, so both paths land in one schema. + - type: move + if: 'attributes?.event != nil' + from: attributes.event + to: attributes["bugwarden.event"] + - type: move + if: 'attributes?.seq != nil' + from: attributes.seq + to: attributes["bugwarden.seq"] + - type: move + if: 'attributes?.session?.transport != nil' + from: attributes.session.transport + to: attributes["bugwarden.transport"] + - type: move + if: 'attributes?.session?.id != nil' + from: attributes.session.id + to: attributes["bugwarden.session.id"] + - type: move + if: 'attributes?.request?.tool != nil' + from: attributes.request.tool + to: attributes["bugwarden.tool"] + - type: move + if: 'attributes?.guard?.verdict != nil' + from: attributes.guard.verdict + to: attributes["bugwarden.verdict"] + - type: move + if: 'attributes?.guard?.rule != nil' + from: attributes.guard.rule + to: attributes["bugwarden.rule"] + + # The W3C ids the client stamped on the call (issue #28). Promoting + # them to the record's own trace context is what makes a guard + # decision line up with the client trace that caused it. They are + # UNAUTHENTICATED client claims — correlation hints, never evidence + # of who did what; attribution rests on the session and client + # fields, which stay in the body. + - type: trace_parser + if: 'attributes?.trace?.trace_id != nil' + trace_id: + parse_from: attributes.trace.trace_id + span_id: + parse_from: attributes.trace.span_id + + # A record stream that lost records is not an info line. + - type: severity_parser + if: 'attributes?["bugwarden.event"] != nil' + parse_from: attributes["bugwarden.event"] + mapping: + error: audit_gap + info: + - tool_call + - initialize + + # Everything else stays in the body exactly as the file holds it — + # suppressed ids, policy hash, scan counts — so a query can reach it + # without a schema migration here. + - type: add + field: attributes["bugwarden.stream"] + value: audit + - type: add + field: resource["service.name"] + value: bugwarden + +processors: + batch: + timeout: 5s + send_batch_size: 512 + # Bound the memory a collector that cannot reach its backend will take + # before it starts refusing data, so this process cannot become the + # reason the host runs out of memory. + memory_limiter: + check_interval: 1s + limit_mib: 256 + spike_limit_mib: 64 + +exporters: + # Replace with the backend you actually run. Keep the destination inside + # the same trust boundary as the bugs — see the sensitivity note above. + debug: + verbosity: normal + # Example of a real one; delete or replace. + # otlphttp/backend: + # endpoint: https://logs.internal.example.org + # headers: + # authorization: ${env:BACKEND_TOKEN} + # sending_queue: + # storage: file_storage/queue + +service: + pipelines: + logs: + receivers: [otlp, filelog/bugwarden-audit] + processors: [memory_limiter, batch] + exporters: [debug] + telemetry: + logs: + level: info