feat(auth): configurable runtime-token header (server + SDK) to avoid gateway Authorization collision [HYBIM-866]

[josjeon]

Problem

When Agent Control runs behind the O11y gateway, the gateway overwrites Authorization with its own downstream identity JWT — clobbering AC's runtime-eval token on the hot path (HYBIM-866, under epic HYBIM-741). The runtime token and the gateway's identity JWT both want the Authorization header.

Fix

Make the runtime token ride a configurable header on both sides, selected by AGENT_CONTROL_RUNTIME_TOKEN_HEADER (default Authorization). Behind the gateway, point both sides at a dedicated header (e.g. X-Agent-Control-Runtime-Token); the runtime token rides that header while the gateway keeps Authorization for its identity JWT — no collision.

• Authorization: keeps the mandatory Bearer scheme (existing contract).
• Dedicated header: carries the raw token (no Bearer prefix).
• Default unchanged: with the env unset, behavior is byte-identical to today.

Server (verify side)

• auth_framework/providers/local_jwt.py: LocalJwtVerifyProvider takes a header_name (default Authorization) and reads the token from it. Bearer required only on Authorization; raw token accepted on a dedicated header.
• auth_framework/config.py: _resolve_runtime_token_header() reads AGENT_CONTROL_RUNTIME_TOKEN_HEADER (blank → default), passed into the provider when runtime mode is jwt.

SDK (send side)

• sdks/python/.../client.py: AgentControlClient gains a runtime_token_header param (+ same env var). Sends the runtime token on the configured header — raw on a dedicated header, Bearer on Authorization (single _format_runtime_token helper is the sole authority for that rule).
• The API key is preserved as the outer gateway credential: when the runtime token rides a dedicated header, the API key still rides its own header (X-API-Key), so a request can authenticate at the gateway while the runtime JWT is verified by Agent Control. The existing same-header guard prevents any collision.
• High-level SDK: agent_control.init() accepts runtime_token_header, stores it in session state, and threads it into the evaluation clients (evaluation.py, control_decorators.py); it is cleared on reset.

Configuration (behind the gateway)

# server
AGENT_CONTROL_RUNTIME_AUTH_MODE=jwt
AGENT_CONTROL_RUNTIME_TOKEN_SECRET=<secret>
AGENT_CONTROL_RUNTIME_TOKEN_HEADER=X-Agent-Control-Runtime-Token

# SDK (must match the server header)
AgentControlClient(..., runtime_token_header="X-Agent-Control-Runtime-Token")
# or agent_control.init(..., runtime_token_header="X-Agent-Control-Runtime-Token")
# or AGENT_CONTROL_RUNTIME_TOKEN_HEADER=X-Agent-Control-Runtime-Token

Tests

• Server (test_auth_framework.py): default Bearer path; default rejects raw on Authorization; dedicated header reads raw token and coexists with a gateway Authorization JWT; Bearer also accepted on dedicated header; missing-header error names the configured header; blank header_name rejected; env resolver (unset → default, set → trimmed, whitespace → default).
• Server, app-level (test_runtime_token_exchange_endpoint.py): end-to-end through /api/v1/evaluation exercising config wiring + Operation.RUNTIME_USE routing — runtime token on a dedicated header with an outer Authorization gateway JWT is accepted; a token presented on Authorization is rejected (401) when a custom header is configured.
• SDK (test_client.py): header resolution (param/env/default, blank rejected, whitespace-env fallback); raw token on dedicated header with Authorization free while the API key is preserved on its own header; default sends Bearer on Authorization; auto-mode fallback keeps the API key when the exchange is unavailable.
• SDK, high-level (test_init_validation.py): init() stores runtime_token_header in session state; defaults to None when unset; validates the header up front (blank and bad field-name rejected before state is mutated); a positional call through target_id proves the new param is appended last and does not shift any existing slot (controls_file onward).

Security notes

• Header isolation: with a dedicated header configured, the verifier reads only that header — a token presented on Authorization is ignored, so it can't be smuggled past the gateway boundary.
• No token leakage: auth error messages reference the header name only, never the token value.
• Signature / scope / target-binding checks (verify_runtime_token) are unchanged.

Notes / scope

• Backwards compatible: unset env → identical behavior. No change to the API-key or none runtime modes.
• Design follows the O11y api-service precedent: support both auth methods, gateway stays neutral, opt-in, default preserved.
• Server-side tests require the repo's Postgres test fixture (run in CI); the SDK suite runs standalone.
• Validated end-to-end on a devstack: with the SDK sending the token on X-Agent-Control-Runtime-Token and the server reading the same header, runtime JWT exchange, control steering, and control-span ingestion all worked. This confirms the SDK and server agree on the custom header. It does not yet exercise the gateway-collision path (the devstack has no O11y gateway overwriting Authorization).
• Gateway forwarding of the custom header is confirmed with the gateway owners; the real gateway-collision validation on lab0 is pending (blocked on getting the branch image into Artifactory as ao-agent-control).