[rhuss]

Summary

Let the supervisor forward a sandboxed agent's OpenTelemetry traces to an OTLP collector, so operators see agent spans next to infrastructure spans without a sandbox egress rule. Operators opt in with [openshell.gateway.otlp.agent]. The agent exports to a reserved address inside the sandbox; the supervisor serves that address, attributes each batch with the sandbox id, and forwards it over OTLP/gRPC.

This is the minimal first iteration agreed on #3196 and supersedes that PR: the gateway hop, session protocol changes, and capability negotiation are gone because the compute driver hands the supervisor the collector endpoint directly (#3977). Part of #2641.

Related Issue

Closes #4275

Changes

• Gateway config: new [openshell.gateway.otlp.agent] table with enabled (default false) and an optional endpoint that falls back to [openshell.gateway.otlp].endpoint, like the OCSF table. The gateway validates it at startup. Without the table, or with enabled = false, the workload environment is unchanged even when trace export is on.
• Drivers (Docker, Podman, Kubernetes, VM): pass the agent endpoint to the supervisor as OPENSHELL_OTLP_AGENT_ENDPOINT (--otlp-agent-endpoint), separate from the supervisor's own trace endpoint. On Podman Machine the driver rewrites a loopback agent endpoint to the desktop host, as it does for the trace endpoint. The shared Kubernetes supervisor (feat(kubernetes): host sandboxes on the shared supervisor #4702) does not relay; its startup warning says so.
• Sandbox protocol: AgentSpec gains bool otlp_relay_active = 6, which tells the sandbox to inject the agent variables.
• openshell-sandbox: when the relay is active, every workload process (main, exec, SSH) gets OTEL_EXPORTER_OTLP_ENDPOINT=http://192.0.0.8:4318, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf as defaults applied after the provider variables, and an inherited OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is removed. When the creation request already names an OTLP endpoint, generic or traces-specific, nothing is injected and the caller's values stand, per the guidance on feat(observability): supervisor OTLP telemetry relay #3196. The collector address is never visible in the sandbox.
• openshell-supervisor-network: a generic ReservedDestination hook at the top of preauthorize_transparent_open serves staged opens to the reserved address inside the supervisor, ahead of host mapping, policy, and SSRF checks. Without a handler the address is denied immediately, logged once at warn.
• openshell-supervisor::otlp_relay: hyper HTTP/1.1 receiver (POST /v1/traces, protobuf only; 404, 408, 413, 415, 503 with Retry-After, all with google.rpc.Status bodies); wire-level enrichment that decodes only each ResourceSpans.resource and copies spans byte for byte, replacing openshell.sandbox.id and openshell.telemetry.source=agent; bounded buffer; one export task sending raw ExportTraceServiceRequest bytes through a lazy tonic channel with keepalive-based dead-connection detection. Exporter::with_channel is the seam for the shared collector transport requested on feat(observability): supervisor OTLP telemetry relay #3196. POST /v1/metrics and /v1/logs are drained and answered with an empty OTLP success so SDKs that export every signal do not retry; discards are counted, the first is logged, and the count appears in the summaries. FR-009 log cadence: first-occurrence warnings, 30 s summaries while counts change, final summary.
• Fixed limits, all documented: 2 MiB body, 10 s header and 30 s body timeouts, 16 connections, 2 concurrent enrichments, 512 ResourceSpans entries, 256 KiB of resources and 8192 decoded elements per request, 32 buffered batches, 5 s per export, 5 s flush. Worst-case memory attributable to the agent is about 111 MiB.
• Lifecycle: buffered batches are flushed before the exit report; when the access plane is retained for exec sessions the relay keeps serving them and shuts down at teardown. The relay stays inactive with a logged reason when the gateway did not enable it, the agent endpoint uses https, or the sandbox id is missing.
• openshell-otel-test-support: resource attributes and duplicate-key tracking on ReceivedTraces, start_on/try_start_on, and a relay module with the attribution keys and a request builder.
• e2e: tests/otlp_relay.rs (span with attribution, inert without a collector, flush before exit); OPENSHELL_E2E_OTLP_ENDPOINT opt-in in the podman and docker gateway wrappers, which also enable [openshell.gateway.otlp.agent] unless OPENSHELL_E2E_OTLP_AGENT=0; otlp_relay in PODMAN_CI_TESTS.
• Dev tooling: the local gateway tasks (tasks/scripts/gateway*.sh) and deploy/helm/openshell/ci/values-skaffold.yaml enable the agent table.
• Docs: "Agent trace relay" under OTLP Export in docs/how-it-works/gateways/configuration.mdx (including the new table), an "Agent traces" section in docs/about/architecture.mdx, the supervisor, network supervisor, sandbox, and Docker, Podman, Kubernetes, and VM driver READMEs, and the debug-openshell-cluster and helm-dev-environment skills.

Deferred, as agreed on #3196 and tracked on #2641: TLS and authentication on the relayed hop (one shared transport in openshell-otel), relay on the shared Kubernetes supervisor, retry on Unavailable, batching and tunable limits, forwarding agent metrics and logs, additional attribution attributes, JSON or gRPC receivers and gzip bodies, operator-configurable connection cap, OCSF event relay.

Review callout: when the relay is inactive, an export to the reserved address is denied without an OCSF network-denial event (a standard SDK retries every few seconds for the life of the agent). The inactive state is recorded once at startup as a ConfigStateChange; the proxy logs the first denied attempt at warn and later ones at debug.

Release Notes

Sandboxed agents can send OpenTelemetry traces to your collector without an egress rule. Add [openshell.gateway.otlp.agent] with enabled = true to gateway.toml; the supervisor tags each span with the sandbox id and forwards it to the agent endpoint, or to the shared OTLP endpoint when none is set.

Testing

• cargo test --lib and cargo clippy --tests for the touched crates: clippy clean, all tests pass except the known load-sensitive proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header, which passes 10 of 10 when run alone on both this branch and main
• Unit tests added/updated: [otlp.agent] parsing, endpoint fallback and validation, AgentSpec protobuf round trip of otlp_relay_active, receiver status codes and google.rpc.Status bodies, metrics and logs acknowledged and discarded, limits and back-pressure, wire-level enrichment (verbatim copy, byte and element budgets, nesting depth), exporter timeout and dead-connection detection, lifecycle flush and shutdown, proxy reserved-destination hook, sandbox environment composition for both injection branches in a re-executed test binary
• Full Podman e2e on 80a2f8a (macOS, Podman Machine): conformance plus every Rust target with --no-fail-fast. All three otlp_relay tests pass. Every other failure either also fails on main (9a85667) in this environment or passes on rerun; the failing targets were rerun on this branch with the relay on and off and on main with the same results
• Pre-commit checks (license headers, lockfiles, rustfmt, Helm lint and docs, markdown lint) pass
• Manual end-to-end check on podman with real agents, on the pre-rebase head: Claude Code 2.1.292 inside a sandbox (Vertex provider, CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, OTEL_TRACES_EXPORTER=otlp) and a Python OpenTelemetry SDK agent. Traces reached Tempo through an OpenTelemetry Collector with openshell.sandbox.id and openshell.telemetry.source=agent; Claude Code's interaction, llm_request and tool spans arrived intact.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (docs/about/architecture.mdx and crate READMEs; architecture/ was removed in docs: remove the architecture directory #3799)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rhuss]

@krishicks I rebased this onto current main. It now sits on top of #4545, so the supervisor exports OCSF as OTLP logs and relays agent traces, both to the same collector. The scope is unchanged from the first iteration you asked for: traces only, a single endpoint, plaintext on the relayed hop, and the Exporter::with_channel seam for the shared channel.

Since #4399 landed, the gateway exporter handles gzip and https while the relay hop still doesn't. That's the next step in #2642, where the gateway, supervisor and relay share one channel. I'd rather keep that out of this PR so it stays reviewable.

This PR hasn't run CI yet. Could you /ok to test b55b23c1a and take a look when you have time?

[krishicks]

/ok to test b55b23c

[github-actions]

Compatibility review (advisory)

Unaddressed breaking changes

The candidate silently replaces explicit sandbox-level OTLP protocol and compression environment values when no endpoint is supplied. This violates the v0.1.5 patch-train policy, and the PR discussion does not explicitly acknowledge affected existing consumers.

1. api-behavior — NOT ACKED

openshell sandbox create --env values for OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_COMPRESSION, and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION were preserved, but are now replaced with http/protobuf and none unless the request also sets an OTLP endpoint.

• Impact: Existing workloads that configure only OTLP protocol or compression are silently redirected to http://192.0.0.8:4318 and no longer export with their requested settings.
• Action: Preserve each explicitly supplied OTEL_EXPORTER_OTLP_* value, or move the change to a minor train with notice and guidance to set an endpoint explicitly.
• PR: feat(supervisor): relay agent OTLP traces to the collector #4276

Review details

• Execution: complete
• Candidate: PR #4276 (e6c136dd526b2265b738dc3aebc14f4a571de712)
• Baseline: 6ab87ce6926e42ed739312ee3d9253725d9b7924 (6ab87ce6926e42ed739312ee3d9253725d9b7924)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[krishicks]

The way #4500 was implemented, the gateway resolves the OTLP endpoint configuration for both traces and OCSF events as possibly different values based on the config, and hands this same config to the supervisor. This is because OCSF logs may end up going to a different place than operational traces. There's a similar concern with agent traces; these may want to go somewhere else. I think that means we should extend the configuration in the gateway to add another table specifically for where agent traces should go, similar to the table for OCSF events. This would be an optional override. That would give operators the ultimate flexibility in their deployment: ability to scale collector pools independently for different use cases, and also to re-export to different sinks (like an SIEM, or LGTM stack) easily.

You wouldn't need to do it as part of this PR, but a fast-follow PR would be good.

What do you think?

[krishicks]

I had Opus 5.5 review this and it came up with some legitimate (I think) concerns. Of these, I think probably 1 and 2 are worth addressing in this PR; the others could be follow-ups.

1. A sandbox user who wants to export to their own collector can't while the relay is active.
Network policy already allows this, since the proxy only intercepts 192.0.0.8:4318. The env injection gets in the way. The relay sets OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, which SDKs prefer over the generic variable. So a user who sets OTEL_EXPORTER_OTLP_ENDPOINT in an image ENV, an exec -e, or a shell export still sends traces to the relay. Only the creation-request check is honored, and sandbox users often don't control the creation request. The exec case also contradicts the docs, which say an exec env may override the injected values. Image ENV is overwritten on purpose, but env vars aren't an enforcement boundary (a process can export anywhere policy allows), so overriding it doesn't protect anything and only breaks users who set up their own collector.

Possible resolution: inject only the generic OTEL_EXPORTER_OTLP_* variables as a lowest-precedence default, don't override image-provided values, and keep provider values excluded. Then any destination the user sets, at any layer, wins as the OTel spec intends. The NeMo Relay trace-only case would need a separate answer.

2. Agents get OTLP variables even when there's nowhere to send them, and metrics and logs exports go to the relay too. Today a sandbox without a collector gets no OTLP variables. With this PR every workload gets them, so SDKs that turn on export when an endpoint is set retry against a denied address for the life of the process. And because the generic endpoint is set, agents with metrics or logs enabled send /v1/metrics and /v1/logs to the relay and get a steady stream of 404s.

Possible resolution: inject only when the relay is active, which needs the supervisor to signal the sandbox runtime. For metrics and logs, accept and discard those paths, or document the behavior.

3. The relay acknowledges spans before delivering them, so they can be lost while the agent believes they were delivered. The relay answers 200 once a batch is buffered, then makes one export attempt and discards the batch on failure. A collector restart or an UNAVAILABLE response (for example from a memory limiter) drops spans the agent's SDK counts as delivered. The only record is a counter.

Possible resolution: retry once on Unavailable or ResourceExhausted, or at least document that delivery is best-effort.

4. An unhealthy collector delays every sandbox's exit report by about 5 seconds. The flush before the exit report waits until the buffer drains or the flush timeout runs out. If the collector silently drops connections, each export spends the connect timeout and the buffer never drains, so every exit is reported late for as long as the collector is down.

Possible resolution: skip or shorten the flush when the most recent export failed, so a trace sidecar's health doesn't affect lifecycle reporting.

5. Operators get no log when buffered spans are lost on supervisor error paths. If the sandbox run exits early with an error after the relay starts, the relay is dropped and its export task aborted. The batches still pending aren't counted as failures, and the final summary only logs when there are failures, so nothing is logged.

Possible resolution: count pending() as discarded in Drop before writing the final summary.

6. A collector endpoint that works for the supervisor's own spans can turn the relay off. openshell-otel trims the endpoint and treats an empty value as unset; the relay does neither. An endpoint with surrounding whitespace, or an empty value passed through by a wrapper, works for infrastructure spans but disables the relay with a Medium-severity "endpoint rejected" event.

Possible resolution: parse the endpoint the same way openshell-otel does, ideally with a shared helper.

7. An agent can still label its spans with another sandbox's id. Resource attributes are rewritten correctly, including when the resource is split across repeated fields. Span and scope attributes pass through unchanged, though, and unscoped queries such as Tempo's { .openshell.sandbox.id = "X" } match span attributes too. The docs say to filter on resource attributes, but the module doc's claim that the workload "cannot impersonate another sandbox" is too strong.

Possible resolution: strip the two attribution keys at scope and span level as well, or narrow the claim.

Everything below this is a nit, I think:

Smaller items

• ReservedDestination is a generic hook, but the proxy also hard-codes the OTLP relay address and a process-wide warn-once flag. Could the "no handler installed" denial move behind the hook?
• The relay adds hyper, tonic-types and opentelemetry-proto to openshell-supervisor. With a shared transport in openshell-otel already planned, would the relay fit better there?
• Any Content-Encoding, including identity, gets a 415. Allowing identity is a one-line change.
• The ~111 MiB worst-case memory figure leaves out the batch being exported and tonic's encode copy, which add roughly 4 MiB.
• The 413 message says "2 MiB limit after attribution," but the enforced limit is 2 MiB plus the attribution allowance. The 503 sent during shutdown reuses the "buffer is full" text.

[rhuss]

Agreed, that's exactly where I was heading too. I'd keep a single endpoint in [openshell.gateway.otlp] as the default for every signal, and add more specific tables to fine-tune where each fan-out goes. #4500 already has that shape with traces and ocsf. So an optional [openshell.gateway.otlp.agent] table (name up for discussion) would fit naturally. Its endpoint falls back to the base endpoint, and the gateway resolves it and hands it to the supervisor next to the other two. That also gives us one place for per-lane settings like TLS, headers and compression later on.

I'll do this as a fast-follow PR once this one lands.

[rhuss]

Thanks for the thorough review. I went through each point against the code and they hold up, so here's my plan.

1 and 2, env injection. Very good point on 1, I'll switch to your suggestion. The relay will inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL as a default. It won't override image ENV or exec env, and provider values stay excluded.

For 2, the supervisor will signal the sandbox runtime so the variables are only injected while the relay is actually active. One caveat remains: with the generic endpoint set, agents that export metrics or logs will send /v1/metrics and /v1/logs to the relay. I propose keeping the 404 for those paths and documenting it until logs and metrics relaying lands. That way the loss is visible to the agent's SDK instead of being silently dropped. If you'd rather avoid the retry noise, I can accept and discard them with a 2xx instead. What do you prefer?

4, 5, 6, 7, and the nits on identity encoding, the memory figure and the 413/503 messages, I'll fix in this PR. For 4, the exit flush is skipped when the latest export failed, which also removes the double wait on the retained-access path. For 7, I'll narrow the module doc claim to resource attributes. Scrubbing span attributes would mean decoding spans, which the relay avoids for memory reasons.

3, retrying on Unavailable or ResourceExhausted, goes to a follow-up together with batching and backoff. Here I'll document that delivery is best-effort.

The ReservedDestination and dependency nits are follow-ups too. Moving the relay onto the shared transport in openshell-otel is already planned, and the hook cleanup fits in there.

[elezar]

/ok-to-test 3559561

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

[openshell.gateway.otlp].endpoint previously exported infrastructure telemetry only, but now also injects OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL into workloads lacking an endpoint and replaces an explicit protocol-only value.

• Impact: Existing plaintext-OTLP deployments silently redirect OpenTelemetry-enabled workload traces and generic metrics or logs to the relay, while workloads setting only OTEL_EXPORTER_OTLP_PROTOCOL=grpc run with http/protobuf instead.
• Action: Gate agent relaying behind a new default-off TOML and Helm setting and preserve every explicitly supplied OTEL_EXPORTER_OTLP_* value, or defer the change to a minor train with notice and migration guidance.
• PR: feat(supervisor): relay agent OTLP traces to the collector #4276

Review details

• Execution: complete
• Candidate: PR #4276 (bbd13a781f4f3cd1c5b578291159907503d96c41)
• Baseline: b2069a267c7c8c7cb9a8b652792167dab886cd83 (b2069a267c7c8c7cb9a8b652792167dab886cd83)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[rhuss]

Rebased onto main after #4701/#4702 and settled the two open points.

The relay is now opt-in through [openshell.gateway.otlp.agent] with enabled = true. Its optional endpoint falls back to [openshell.gateway.otlp].endpoint, like the OCSF table, so this also covers the per-lane endpoint you suggested. Drivers pass it to supervisors as OPENSHELL_OTLP_AGENT_ENDPOINT, separate from the trace endpoint, so enabling trace export alone no longer touches the workload environment. The shared Kubernetes supervisor does not relay yet.

For /v1/metrics and /v1/logs I went with accept and discard: the relay answers with an empty OTLP success, counts the discards and logs the first one, so SDKs that export every signal stop retrying.

Main moved the sandbox protocol to protobuf, so AgentSpec gains otlp_relay_active = 6. Without it the flag was silently dropped.

Could you /ok to test the new head when you have a moment?