Skip to content

Relaunch live agents when their inference credential rotates - #427

Merged
TheGreatAxios merged 13 commits into
mainfrom
cl-6687-rotated-api-keys-never-reach-live-agents
Aug 29, 2026
Merged

TheGreatAxios merged 13 commits into
mainfrom
cl-6687-rotated-api-keys-never-reach-live-agents

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes CL-6687 — https://linear.app/abklabs/issue/CL-6687

Problem

Inference sources (decrypted provider key included) are resolved at deploy time and rendered into a run's bytes; nothing re-reads them while the run is resident. After an Anthropic 401, pasting a new key in Settings returned 200 from /connections/anthropic/complete, yet the sidecar kept logging API key is invalid [HTTP 401] credential_failure for the already-open workbench. The send-time drift check (reconcileDriftedRun, CL-6588) compares only foldedBody content, which never carries the secret, so a rotation was invisible to it. Folded runs are self-anchored, so vendor's credential-push skips them too.

Change

  • @corbits/folded-runs: extract resolveLaunchSources from deployAtHead; add inferenceSourcesDigest (SHA-256 over the resolved chain, secret included; only the hash leaves). deployAtHead / launchFoldedRun / wakeFoldedRun return the digest.
  • @corbits/chat: nullable workbench_launch.sources_digest (migration 0024), recorded on every wake and relaunch. reconcileDriftedRun also relaunches when the stored digest differs from today's resolution; the per-send check is throttled to one resolution per participant per 30s. A row with no digest yet gets today's chain recorded as its baseline. New HubChatPlatform.reconcileInferenceSources(tenantId).
  • Standalone launches record the digest too: @corbits/webhook-triggers gains a recordLaunchSources port; the hub's routine launcher calls recordSourcesDigest after launchFoldedRun (their mapping row is written before the deploy resolves the chain).
  • apps/hub: the existing onConnected hook kicks reconcileInferenceSources (fire-and-forget, reportError on failure) when an inference provider's credential lands, so the operator's fix reaches the open room without waiting for a message.

Re-saving the same key yields the same digest → no relaunch.

Tests

  • packages/folded-runs/src/launch.test.ts: digest is key-order-stable, secret-sensitive, default-source-sensitive, never contains the secret.
  • packages/chat/test/platform-adapter.test.ts: rotated key → relaunch on the new key; same key → left alone; pre-column row → baseline recorded, no relaunch; second send inside the throttle window resolves nothing; reconcileInferenceSources sweep relaunches.
  • packages/webhook-triggers/test/launch.test.ts, apps/hub/src/routine-launcher.test.ts: the digest is recorded after a standalone launch.

Review

Critique pass (sonnet) found: routine/webhook launches never recorded a digest (fixed above), per-send resolution cost (throttled), duplicated isInferenceProvider (now imported from @workbench/connections).

@TheGreatAxios
TheGreatAxios force-pushed the cl-6687-rotated-api-keys-never-reach-live-agents branch 2 times, most recently from 1632d46 to 1d7af4a Compare August 28, 2026 12:18
@TheGreatAxios
TheGreatAxios force-pushed the cl-6687-rotated-api-keys-never-reach-live-agents branch 2 times, most recently from 09c121d to bb62374 Compare August 29, 2026 15:34
Inference sources are resolved and rendered into a run's deployed bytes
once; a rotated provider key only ever reached the next deploy, so an
open workbench kept sending the dead key after Settings said the new one
was saved. Every deploy now records a digest of the resolved chain on the
run's launch row — room invites, routine fires and webhook launches
alike; the send-time drift check and a new provider-connect hook compare
it against the current catalog and relaunch on a mismatch.

Fixes CL-6687.
The credential-rotation rebase carried a one-line bun.lock drift that
would fail bun install --frozen-lockfile. This PR does not add a
workspace package.
@TheGreatAxios
TheGreatAxios force-pushed the cl-6687-rotated-api-keys-never-reach-live-agents branch from 6ff4637 to 969c8eb Compare August 29, 2026 17:41

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critique · request-changes

Credential rotation relaunches drifted runs, but a connect-path relaunch is still swallowed for 30s.

  • packages/chat/src/platform-adapter.ts:533-556 — hasDriftedSources returns false when now - checkedAt < 30_000. A provider connect that lands inside that window never reaches reconcileDriftedRun.
  • packages/chat/test/platform-adapter.test.ts — no test drives connect-then-send inside 30s and expects relaunch. Rotate-path tests pass because they send after the cache expires or on a fresh adapter.

CL-6687's connect-path outcome is not met until that cache is gone or invalidated on connect.

GitHub will not accept request-changes on the author's own PR; this comment is the review.

A recent send stamped the 30s interval, so a connect inside that window never reached the drift check. Drop the stamp when the tenant's inference sources are reconciled so the sweep and the next send can see the new key.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critique · approve

Connect-then-send inside 30s is closed. reconcileInferenceSources now deletes sourcesCheckedAt per binding before hasDriftedSources, and the new test stamps the interval then connect+send.

  • packages/chat/src/platform-adapter.ts:744 — stamp delete before reconcileDriftedRun
  • packages/chat/test/platform-adapter.test.ts:3306 — connecting a provider then sending within the check interval relaunches onto the new key

Should-fix (not blocking): stamp delete is per-iteration after listLaunchesForTenant, so a concurrent send to a later agent in the same tenant can still race. Typical one-agent paste-then-send is closed.

GitHub will not accept approve on the author's own PR; this comment is the review.

@TheGreatAxios
TheGreatAxios merged commit 7591d4d into main Aug 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant