Relaunch live agents when their inference credential rotates - #427
Conversation
1632d46 to
1d7af4a
Compare
09c121d to
bb62374
Compare
Inference sources are resolved and rendered into a run's deployed bytes once; a rotated provider key only ever reached the next deploy, so an open workbench kept sending the dead key after Settings said the new one was saved. Every deploy now records a digest of the resolved chain on the run's launch row — room invites, routine fires and webhook launches alike; the send-time drift check and a new provider-connect hook compare it against the current catalog and relaunch on a mismatch. Fixes CL-6687.
The credential-rotation rebase carried a one-line bun.lock drift that would fail bun install --frozen-lockfile. This PR does not add a workspace package.
6ff4637 to
969c8eb
Compare
TheGreatAxios
left a comment
There was a problem hiding this comment.
critique · request-changes
Credential rotation relaunches drifted runs, but a connect-path relaunch is still swallowed for 30s.
- packages/chat/src/platform-adapter.ts:533-556 — hasDriftedSources returns false when now - checkedAt < 30_000. A provider connect that lands inside that window never reaches reconcileDriftedRun.
- packages/chat/test/platform-adapter.test.ts — no test drives connect-then-send inside 30s and expects relaunch. Rotate-path tests pass because they send after the cache expires or on a fresh adapter.
CL-6687's connect-path outcome is not met until that cache is gone or invalidated on connect.
GitHub will not accept request-changes on the author's own PR; this comment is the review.
…eys-never-reach-live-agents
A recent send stamped the 30s interval, so a connect inside that window never reached the drift check. Drop the stamp when the tenant's inference sources are reconciled so the sweep and the next send can see the new key.
TheGreatAxios
left a comment
There was a problem hiding this comment.
critique · approve
Connect-then-send inside 30s is closed. reconcileInferenceSources now deletes sourcesCheckedAt per binding before hasDriftedSources, and the new test stamps the interval then connect+send.
- packages/chat/src/platform-adapter.ts:744 — stamp delete before reconcileDriftedRun
- packages/chat/test/platform-adapter.test.ts:3306 — connecting a provider then sending within the check interval relaunches onto the new key
Should-fix (not blocking): stamp delete is per-iteration after listLaunchesForTenant, so a concurrent send to a later agent in the same tenant can still race. Typical one-agent paste-then-send is closed.
GitHub will not accept approve on the author's own PR; this comment is the review.
Fixes CL-6687 — https://linear.app/abklabs/issue/CL-6687
Problem
Inference sources (decrypted provider key included) are resolved at deploy time and rendered into a run's bytes; nothing re-reads them while the run is resident. After an Anthropic 401, pasting a new key in Settings returned 200 from
/connections/anthropic/complete, yet the sidecar kept loggingAPI key is invalid [HTTP 401] credential_failurefor the already-open workbench. The send-time drift check (reconcileDriftedRun, CL-6588) compares onlyfoldedBodycontent, which never carries the secret, so a rotation was invisible to it. Folded runs are self-anchored, so vendor'scredential-pushskips them too.Change
@corbits/folded-runs: extractresolveLaunchSourcesfromdeployAtHead; addinferenceSourcesDigest(SHA-256 over the resolved chain, secret included; only the hash leaves).deployAtHead/launchFoldedRun/wakeFoldedRunreturn the digest.@corbits/chat: nullableworkbench_launch.sources_digest(migration0024), recorded on every wake and relaunch.reconcileDriftedRunalso relaunches when the stored digest differs from today's resolution; the per-send check is throttled to one resolution per participant per 30s. A row with no digest yet gets today's chain recorded as its baseline. NewHubChatPlatform.reconcileInferenceSources(tenantId).@corbits/webhook-triggersgains arecordLaunchSourcesport; the hub's routine launcher callsrecordSourcesDigestafterlaunchFoldedRun(their mapping row is written before the deploy resolves the chain).apps/hub: the existingonConnectedhook kicksreconcileInferenceSources(fire-and-forget,reportErroron failure) when an inference provider's credential lands, so the operator's fix reaches the open room without waiting for a message.Re-saving the same key yields the same digest → no relaunch.
Tests
packages/folded-runs/src/launch.test.ts: digest is key-order-stable, secret-sensitive, default-source-sensitive, never contains the secret.packages/chat/test/platform-adapter.test.ts: rotated key → relaunch on the new key; same key → left alone; pre-column row → baseline recorded, no relaunch; second send inside the throttle window resolves nothing;reconcileInferenceSourcessweep relaunches.packages/webhook-triggers/test/launch.test.ts,apps/hub/src/routine-launcher.test.ts: the digest is recorded after a standalone launch.Review
Critique pass (sonnet) found: routine/webhook launches never recorded a digest (fixed above), per-send resolution cost (throttled), duplicated
isInferenceProvider(now imported from@workbench/connections).