Skip to content

perf: ship the shared agent-build session default-on with a kill switch - #1395

Merged
philmerrell merged 2 commits into
developfrom
feature/agent-build-shared-session-default-on
Sep 30, 2026
Merged

philmerrell merged 2 commits into
developfrom
feature/agent-build-shared-session-default-on

Conversation

@philmerrell

@philmerrell philmerrell commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

#1377 shipped one process-wide boto3 session for the first-turn agent build, off by default behind a two-arm dev A/B with the decision rule written before the data. The A/B ran on dev on 2026-09-30 (15 cold first turns per arm, interleaved, every turn its own Runtime process, all 30 ok, arm labels verified server-side, sessions soft-deleted afterwards) and met the rule decisively. This PR ships it default-on.

Readout (median / p75, ms):

Stage control shared session
agent_build.session_mgr_clients 446 / 497 8 / 9
agent_build.session_mgr (network) 197 / 217 207 / 237
agent_build.strands_agent 55 / 72 5 / 5
agent_build.finalize (restore) 100 / 109 99 / 103
agent_build (group) 869 / 955 372 / 408
prelude total 1497 / 1655 1003 / 1049
client: first token 4211 / 4632 3564 / 3902

Decision metric (session_mgr_clients + session_mgr): control ranged 601 to 813, the shared arm 171 to 289. The distributions do not overlap, and no stage regressed (session_mgr's +10 is inside its own spread). Warm-up did its part: Runtime logs show every fresh container building the three shared clients in 15 to 53 ms and reading the strategy ids at warm-up (median 193 ms, one outlier at 3.2 s, all ok), on the daemon thread and never on a request.

What changed

  • One kill switch replaces the experiment. AGENT_BUILD_SHARED_SESSION_ENABLED (agent_build_shared_session_enabled): unset or empty means on, only the literal false disables, house style. Off, every SDK builds its own session exactly as before and warm-up skips the shared step, including the one strategy-id connection. AGENT_BUILD_EXPERIMENT, AGENT_BUILD_ARMS, agent_build_experiment_arm and memory_shared_clients_enabled are gone; a Runtime still carrying the old variable reads as nothing (test pinned).
  • No CDK entry. The Runtime is at 48 of its 50 environment variables and unset means on, so this follows the runtime-only flags (AGENT_STATUS_ENABLED, MID_TURN_STEERING_ENABLED, ...) rather than taking a slot. Added to the docs-site runtime-only table.
  • The session_id plumbing that only picked an arm is removed from ModelConfig.to_bedrock_config, AgentFactory._create_bedrock_model / create_agent and ChatAgent. _discover_strategy_ids keeps shared_session= as part of its cache key so a switched-off process still makes its own call.
  • turn_prelude stamps sharedSession (a property, never a dimension) in place of buildArm; processBuilds stays.
  • The experiment script is retired with the ab mode it needed. The readout lives in docs/specs/turn-latency-preamble.md PR-6, now marked shipped, which also keeps the record of the withdrawn off-loop arm.

Nothing about the mechanism changes: the session, its client() sharing, the warm-up placement, the SDK MemoryClient rebinding and the BedrockModel boto_session hand-off are exactly what the A/B measured.

Cost / TTFT

  • Nothing reaches the prompt, toolConfig or restored history.
  • Nothing runs on the request path that did not run on the winning arm; the flag read is one environment lookup where the arm hash used to be. Startup, on the warm-up daemon thread: three client constructions and one GetMemory read, unchanged.
  • Expected effect on every deployment's cold first turn is the table above: about half a second off the build and the first token.

After merge

  • AGENT_BUILD_EXPERIMENT has already been removed from the dev Runtime out of band (back to 48 variables, image unchanged), so dev runs the plain build until this deploys and the default-on session after.
  • Watch the first Runtime V2 restore's logs for the warm-up connection, per turn-path-ttft.md §5 P2.

Tests

  • Backend full suite from the branch: 10997 passed, 3 skipped (run from the worktree with PYTHONPATH pointed at the branch).
  • Flag: everything but false is on; false in any case is off; the retired experiment variable is ignored.
  • Shared session (as before, re-keyed to the new flag): warm-up builds each client once and discovers the strategy ids once, and now skips the whole step when switched off; the factory hands the SDK constructor and MemoryClient the shared session by default and nothing when off; BedrockModel receives boto_session and no region_name by default, neither when off, and the shared session resolves the same region as a fresh one; reset_cached_clients drops the session.

🤖 Generated with Claude Code

philmerrell and others added 2 commits September 30, 2026 10:36
…kill switch

The dev A/B (docs/specs/turn-latency-preamble.md PR-6, 2026-09-30,
15 cold first turns per arm) met the rule written before the data:
`agent_build` median 869ms -> 372ms, first token at the client
4211ms -> 3564ms, decision metric ranges 601-813 vs 171-289 with no
overlap and no stage regressed.

`AGENT_BUILD_EXPERIMENT` and its arms are replaced by
`AGENT_BUILD_SHARED_SESSION_ENABLED` (unset means on; only `false`
disables, and then warm-up skips the shared step too). The `session_id`
plumbing that existed only to pick an arm goes from
`ModelConfig.to_bedrock_config`, `AgentFactory` and `ChatAgent`;
`turn_prelude` stamps `sharedSession` instead of `buildArm`. The
experiment script is retired with the flag it needed. No CDK entry: the
Runtime is at 48 of 50 environment variables and unset means on.

Nothing reaches the prompt, `toolConfig` or restored history; nothing
runs on the request path that did not before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@philmerrell
philmerrell merged commit 7501158 into develop Sep 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant