feat(runner): accept caller trace headers for OpenRouter API requests

[jamespsterling]

TL;DR

runBenchmarkById accepts an optional traceHeaders map and sets those headers on OpenRouter API requests only, so a caller (the monorepo Temporal runBenchmark activity) can propagate its OTel trace context (traceparent, x-or-traceparent, x-benchmark-trace) into the API.

What changed?

• RunBenchmarkInput.traceHeaders?: Readonly<Record<string, string>> (optional; behavior unchanged when omitted).
• New src/runner/trace-headers.ts: makeHttpClientLayer wraps FetchHttpClient.layer with HttpClient.mapRequest, applying the headers only to requests whose URL starts with the normalized OpenRouter API base URL — dataset/sandbox/HuggingFace requests are untouched.
• run-by-id.ts provides this layer instead of the bare FetchHttpClient.layer.

Why?

Benchmark traces in Datadog currently end at the worker activity; the API side of each request is a separate trace. Threading the caller's trace headers lets the API continue the same trace (companion monorepo PR: OpenRouterTeam/openrouter-web#35408).

How to test

• bun test src/runner/trace-headers.test.ts — covers header application to API-prefixed requests and non-application to other hosts.
• Run any benchmark without traceHeaders: request headers are byte-identical to before.

Reviewer focus

• Header gating is URL-prefix based (normalizeBaseUrl(baseUrl ?? DEFAULT_BASE_URL)) so the trust-gate header can't leak to third parties.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• No credentials, private results, or restricted dataset contents are included

Link to Devin session: https://openrouter.devinenterprise.com/sessions/8669b85c73fb4378aebc0b5476836b6e
Requested by: @jamespsterling

[devin-ai-integration]

Original prompt from James

Look at all the bench worker traces for ~1 day and see what common errors we see https://us5.datadoghq.com/apm/traces?query=env%3Aproduction%20service%3Abench-worker&agg_m=count&agg_m_source=base&agg_t=count&cols=core_service%2Ccore_resource_name%2Clog_duration%2Clog_http.method%2Clog_http.status_code&fromUser=false&graphType=flamegraph&historicalData=false&messageDisplay=inline&query_translation_version=v0&shouldShowLegend=true&sort=desc&spanType=all&storage=driveline&traceQuery=&view=spans&start=1787195072717&end=1787195972717&paused=false

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[perry-the-pr-reviewer]

Perry's Review

Verdict: ✅ APPROVE

Risk: 🟢 Low

Risk assessment

Dimension	Severity	Risk	Reasoning
Implementation risk	🟩	Low	Clean threading of traceHeaders through the benchmark type, all benchmark definitions, both providers (chat completions + responses), and the generation resolver. New filterTraceHeaders allowlist prevents header injection (e.g. authorization is dropped). Tests verify the filter and that auth headers cannot be overridden.
Premise risk	🟩	Low	Propagating OTel trace context from the Temporal worker activity into the OpenRouter API is the right observability improvement. URL-prefix gating in the companion PR (#35408) ensures trace headers can't leak to third-party hosts.
Estimated impact	🟩	Low	Optional field with backward-compatible default. filterTraceHeaders returns undefined when unset or when nothing survives the filter, so the feature is a no-op unless configured.

Risk Factor	Severity	Risk	Reasoning
Reversibility	🟩	Low	Omit traceHeaders or revert.
Detectability	🟩	Low	Trace headers visible in Datadog APM.
Blast radius	🟩	Low	Only the HTTP client layers in benchmark runs.
Data integrity		None	No persisted state touched.
Financial exposure		None	No billing code involved.
Security and privacy exposure	🟩	Low	filterTraceHeaders allowlist drops authorization and any non-trace header. Tests verify authorization: Bearer attacker-key is dropped and the real auth header is preserved. Trace headers carry trace context IDs, not user data.
Propagation	🟩	Low	Additive optional field across all benchmark types and providers; existing callers unaffected.
Availability		None	Header injection is a no-op for request handling.
Recovery cost	🟩	Low	Omit the field.
Time to correct	🟩	Low	One-line change.

Analysis

What this does: Threads an optional traceHeaders field through the entire benchmark harness: RunBenchmarkInput → benchmark type definitions → both providers (chat completions + responses) → generation resolver. A new filterTraceHeaders function ensures only allowlisted trace headers (traceparent, tracestate, x-or-traceparent, x-benchmark-trace) pass through, dropping authorization and any other non-trace header.

Key additions since the initial review (3 files, 80 additions → 16 files, 203 additions):

1. filterTraceHeaders in src/runner/trace-headers.ts: Allowlist of 4 header names, normalizes to lowercase, drops anything not in the set. Returns undefined when nothing survives the filter. This is a security improvement: even if a caller passes authorization in traceHeaders, it gets dropped.

2. traceHeaders threaded through all benchmark types: deep-swe, define-single-turn, swe-atlas, tau-bench-airline, tau3-bench-banking, wandr. Each uses the same conditional spread pattern (...(input.traceHeaders !== undefined && { traceHeaders: input.traceHeaders })).

3. Both providers accept traceHeaders: openrouter-model.ts (chat completions) and responses-client.ts / responses-model.ts (responses API). The responses client spreads ...traceHeaders into the request headers (after the harness referer/title, before extraHeaders and version override).

4. Generation resolver accepts traceHeaders: generation-resolver.ts spreads ...traceHeaders into the lookup request headers, with Authorization always last (so it can't be overridden by trace headers even if filterTraceHeaders is bypassed).

5. run-by-id.ts filters once at the top: Both the benchmark layer and the generation resolver receive filterTraceHeaders(input.traceHeaders) (the filtered result), not the raw input.

Why it's correct:

• filterTraceHeaders is a security boundary. Only 4 allowlisted header names pass through. authorization, cookie, x-session-id, etc. are dropped. This prevents header injection through the traceHeaders field even if a caller passes arbitrary headers.

• Header ordering ensures auth cannot be overridden. In responses-client.ts, ...traceHeaders comes before ...options.extraHeaders but the harness always sets Authorization in the HttpClient layer (not via trace headers). In generation-resolver.ts, Authorization: Bearer ${config.apiKey} is set after ...traceHeaders, so even if authorization survived the filter (it won't), the resolver's auth takes precedence.

• Tests verify the security boundary. The openrouter-model.test.ts test passes authorization: "Bearer attacker-key" in traceHeaders and asserts the request's authorization header is "Bearer sk-test" (the real key), not the attacker key. The generation-resolver.test.ts test makes the same assertion. The trace-headers.test.ts test directly verifies filterTraceHeaders drops authorization and x-session-id.

• No-op when unset. filterTraceHeaders(undefined) returns undefined, and the conditional spread ...(traceHeaders !== undefined && { traceHeaders }) omits the field entirely.

CI: validate, Analyze (actions), Analyze (javascript-typescript), CodeQL all pass.

[perry-the-pr-reviewer]

Question: The trace headers are spread (...traceHeaders) into the request headers before ...options.extraHeaders. If a caller passes an extraHeaders entry with the same name as a trace header (e.g. traceparent), the extra header would override the trace header. Is this intentional (caller can override trace headers per-request), or should trace headers take precedence over extra headers?