diff --git a/devlog/_plan/260830_lane_q_2892_gaps_3_4/000_units.md b/devlog/_plan/260830_lane_q_2892_gaps_3_4/000_units.md new file mode 100644 index 0000000000..d1515809ed --- /dev/null +++ b/devlog/_plan/260830_lane_q_2892_gaps_3_4/000_units.md @@ -0,0 +1,139 @@ +# Lane Q — issue #2892 gaps 3 and 4 + +The last two of the five gaps #2892 raised against the merged stored-Pool 401 +recovery path. Gaps 1–2 shipped as `8f199fcb6` (#2920), gap 5 as `84049830e` +(#2922). An independent recon audit re-derived both remaining gaps from current +`dev` and confirmed each is still reachable — and corrected the reporter on one +point, recorded below. + +## Gap 3 — a rotated grant never reaches an inactive same-grant alias + +A successful refresh persists the rotated credential to the flight owner only, via +the generation CAS at `src/codex/account-store.ts:748`. A live joiner can CAS the +result onto its own record. Nothing writes to a third category: a non-deleted +record carrying the same `refreshGrantFingerprint` that is not participating in +the flight. + +`findFreshCredentialForGrant` (`src/codex/account-store.ts:393`) is a pre-fetch +lookup and propagates nothing. So the alias keeps a refresh token that upstream +has just rotated away. The next refresh on that alias sends a dead grant, and +`invalid_grant` is classified `revoked` — which retires a healthy account. That +classification is correct behavior for a genuinely dead grant; the defect is that +the grant died because we rotated it and never told the alias. + +### The design an adversarial audit rejected + +My first plan had two branches: an untouched alias adopts the rotated credential +whole, and an alias whose access token had changed concurrently keeps its own +access token but takes **only** the rotated refresh token. An independent audit +refused that second branch, with two findings I could not rebut: + +- The generation fence in `src/codex/plan-from-token.ts:32` treats a higher + generation as proof of a **newer access-token JWT**, which is what lets JWT plan + claims supersede an older WHAM observation. Bumping a generation while + deliberately keeping the old access token lets a stale JWT overwrite an + authoritative plan. `tests/codex-plan.test.ts:129` already pins that meaning. +- A flight is keyed by grant and does not record participant account ids + (`src/codex/account-store.ts:308`), so a scan cannot distinguish a dormant alias + from a live joiner. Rotating a joiner's grant while preserving its 401-rejected + access token makes the provenance CAS inapplicable, and the recursion's + freshness shortcut then returns the rejected bearer — defeating 401 recovery in + exactly the case the branch existed to serve. + +### What ships instead + +One batch compare-and-swap, one `persist`, and a deliberately narrow eligibility +test. An alias is repaired only when it is provably an untouched duplicate of the +pre-refresh credential: same old grant fingerprint, same access token, same +expiry, and the same `chatgptAccountId` as the owner. Such an alias receives the +rotated access token, refresh token, and expiry **together**, so the generation +bump keeps meaning what every fence already assumes. `replacedAt` and the +validation metadata are preserved, because the probe-lease lineage check accepts +only an intact `G → G+1`. + +Anything else is left alone: a differing access token, a differing account id, or +a tombstone. The `chatgptAccountId` equality requirement is not decoration — a +fingerprint is `sha256` of the refresh token and carries no identity claim +(`src/codex/account-store.ts:62`), and no repository invariant guarantees one +grant cannot span two account ids. + +This is a **partial** close of gap 3, and the issue comment says so. Dormant +duplicates stop being retired for a grant we rotated away; a mixed alias still is. +Healing that case needs durable grant lineage and verified identity binding, which +the current fingerprint-and-generation model cannot express safely. + +The flight's returned `resolvedGrantFingerprint` stays the **old** fingerprint: +joiners wait on that key, and retagging it would make every legitimate joiner look +foreign. + +## Gap 4 — stale credential evidence writes unscoped state + +The reporter described an async interleaving between validation and mutation. That +part is wrong and worth stating: `recordCodexUpstreamOutcome` is synchronous +(`src/codex/routing.ts:2095`) and there is **no `await`** between the generation +check at `src/codex/routing.ts:2210` and the mutations at 2216–2223. The +same-process race the issue describes is not reachable. + +The cross-process race is real regardless. The check is an unlocked synchronous +store read (`src/codex/account-store.ts:186`) while writers coordinate under the +mutation lock, and OS preemption needs no `await`. The side effects then carry no +credential identity: health entries have no generation field, reauth state is a +bare `Set` fenced only by config generation, and affinity clearing removes +every entry for the account. + +Affinity is already self-invalidating on the next generation check. Health and +reauth were not. + +My first attempt snapshotted the state, mutated, then re-read the generation and +rolled back. Two reviewers independently rejected it, correctly: a replacement can +land at any point *after* `recordCodexUpstreamOutcome` returns, so a post-write +read narrows the window without closing it. @Ingwannu reproduced the surviving +ordering on the exact head — record a 401 at G, return, then persist G+1, and the +quarantine still applied to G+1. + +The evidence is now tagged with the credential it came from and checked when it is +*read*, which is what actually settles it. `credentialFailureGeneration` holds the +generation a 401/403 was derived from, and the health readers (`shouldFailover`, +`getCodexUpstreamHealth`) drop a failure whose credential is gone. The reauth set +became a map from account id to the generation that justified the flag, with +`undefined` preserved as an account-wide mark so a login flow with no specific +credential still quarantines unconditionally. + +Affinity clearing stays un-reverted: entries already carry a generation and +self-invalidate, so re-adding swept entries would be the worse bug. +`recordCodexUpstreamOutcome` stays synchronous — many callers consume it as +`void` (`src/server/responses/core.ts:391`), so making it async would silently +leave mutations unawaited. The config lock is still never taken on the request +path; it runs with `busy_timeout=0`, and per-outcome acquisition would turn +contention into request errors. + +## The alias plan note + +Review also caught that propagation installs the rotated JWT on an alias but left +its configured plan alone: a `plus → pro` rotation gave the alias a Pro credential +while its plan stayed `plus`, and the cached-token fast path never repairs that, so +quota scoring and the 30-day projection stayed wrong until a restart or a WHAM +refresh. Each propagated alias is now reconciled at its **own** committed +generation, which is why the commit returns `{ id, generation }` rather than ids — +aliases need not share a generation, and the plan note is generation-fenced. + +That last point produced the one genuinely vacuous assertion of this unit: with a +single `saveCodexAccountCredential` per record, owner and alias generations +coincided, so an assertion about the per-alias fence passed even when the code used +the owner's generation. The fixture now advances the alias twice so the generations +diverge, and the mutation turns red. + +## Constraints the audit flagged + +The refresh-flight map is keyed by the old grant (`src/codex/account-store.ts:315`) +and joiner provenance deliberately carries that old fingerprint, so alias +propagation must not disturb that ordering. The config lock runs with +`busy_timeout=0` and must stay synchronous, so the routing path must not acquire +it per outcome. Affinity requires exact credential-generation equality, so any +alias generation bump has to be reasoned about rather than assumed harmless. + +## Evidence standard + +Each regression is driven red by a named mutation, using the existing blocked-fetch +seam rather than a timing sleep. Any assertion that survives its mutation is +deleted rather than kept. diff --git a/src/codex/account-runtime-state.ts b/src/codex/account-runtime-state.ts index a2a6495e6a..ff036429d7 100644 --- a/src/codex/account-runtime-state.ts +++ b/src/codex/account-runtime-state.ts @@ -1,18 +1,43 @@ import { captureConfigGeneration, type GenerationContext } from "../lib/state-store-sweeper"; +import { isCodexAccountGenerationLive } from "./account-store"; -const reauthAccounts = new Set(); +/** + * Accounts quarantined for reauthentication, each remembering WHICH credential produced the + * evidence (#2892 gap 4). + * + * A 401 describes one credential, not an account. Recording only the id let a 401 raced by a + * cross-process credential replacement quarantine the replacement: the flag outlived the credential + * it was evidence about, and routing then refused a perfectly good credential until a restart. A + * post-write re-read cannot fix that — the replacement may land at any point after the write — so + * the generation travels WITH the flag and is checked when the flag is read. + * + * `undefined` means "no credential generation was supplied", which stays account-wide: callers such + * as a login flow have no specific credential in hand, and their quarantine must not silently expire. + */ +const reauthAccounts = new Map(); let lastReconciledGeneration = 0; let liveAccountIds = new Set(); -export function markAccountNeedsReauth(id: string, writerGeneration = captureConfigGeneration()): void { +export function markAccountNeedsReauth( + id: string, + writerGeneration = captureConfigGeneration(), + credentialGeneration?: number, +): void { if (writerGeneration < lastReconciledGeneration && !liveAccountIds.has(id)) return; - reauthAccounts.add(id); + // An account-wide mark supersedes a generation-scoped one: it is the stronger claim. + if (credentialGeneration === undefined || !reauthAccounts.has(id)) { + reauthAccounts.set(id, credentialGeneration); + return; + } + const existing = reauthAccounts.get(id); + if (existing === undefined) return; + reauthAccounts.set(id, Math.max(existing, credentialGeneration)); } export function reconcileCodexReauthState(context: GenerationContext): number { if (context.generation <= lastReconciledGeneration) return 0; let removed = 0; - for (const id of reauthAccounts) { + for (const id of [...reauthAccounts.keys()]) { if (context.codexAccountIds.has(id)) continue; reauthAccounts.delete(id); removed += 1; @@ -23,7 +48,16 @@ export function reconcileCodexReauthState(context: GenerationContext): number { } export function isAccountNeedsReauth(id: string): boolean { - return reauthAccounts.has(id); + if (!reauthAccounts.has(id)) return false; + const credentialGeneration = reauthAccounts.get(id); + if (credentialGeneration === undefined) return true; + // The credential this evidence describes is gone, so the evidence is spent. Drop it rather than + // re-deriving the same answer on every read. + if (!isCodexAccountGenerationLive(id, credentialGeneration)) { + reauthAccounts.delete(id); + return false; + } + return true; } export function clearAccountNeedsReauth(id: string): void { diff --git a/src/codex/account-store.ts b/src/codex/account-store.ts index d919171364..011ac0692e 100644 --- a/src/codex/account-store.ts +++ b/src/codex/account-store.ts @@ -218,6 +218,97 @@ export function saveCodexAccountCredentialIfGeneration( }); } +/** + * Commit a refreshed credential to its owner AND to any record that is provably an untouched + * duplicate of the pre-refresh credential (#2892 gap 3). + * + * A refresh normally rotates the refresh token, and the owner CAS above changes only the owner's + * record. A second non-deleted record holding the same grant that is not participating in the + * flight therefore keeps a refresh token upstream has just rotated away. Its next refresh sends a + * dead grant, and `invalid_grant` classifies as `revoked` — retiring a healthy account because we + * rotated its grant and never told it. + * + * Eligibility is deliberately narrow, and each condition earns its place: + * + * - Same pre-refresh grant fingerprint, access token, AND expiry. Anything else means the alias was + * updated concurrently, and repairing only its grant while keeping its own access token would + * advance a generation without advancing the access-token JWT. `plan-from-token` reads a higher + * generation as proof of a newer JWT (that is how JWT plan claims supersede a WHAM observation), + * so that combination lets a stale JWT overwrite an authoritative plan. It would also hand a live + * forced-refresh joiner back its own 401-rejected bearer: flights are keyed by grant and do not + * record participants, so a scan cannot tell a dormant alias from a joiner, and the recursion's + * freshness shortcut does not re-compare against the rejected token. + * - Same `chatgptAccountId` as the owner. A fingerprint is `sha256` of the refresh token and + * carries no identity claim; no invariant here guarantees one grant cannot span two account ids, + * so identity is compared rather than assumed. + * + * The rotated access token, refresh token, and expiry move together, keeping a generation bump + * meaning what every fence already assumes. `replacedAt` and the validation metadata survive + * because the probe-lease settlement check accepts only an intact `G → G+1` lineage. + * + * One lock acquisition and one `persist` for the owner and every alias: `persist` writes the whole + * store, so a second pass would open a window in which some records hold the dead grant. + */ +export function commitRefreshedCodexCredentialWithAliases( + id: string, + generation: number, + cred: CodexAccountCredentials, +): { committed: boolean; propagatedAliases: { id: string; generation: number }[] } { + return withCredentialMutationLockSync(() => { + const store = loadCodexAccountRecordStore(); + const current = store[id]; + if (!current || current.generation !== generation || current.deletedAt != null || !current.credential) { + return { committed: false, propagatedAliases: [] }; + } + const priorCredential = current.credential; + const priorFingerprint = recordGrantFingerprint(current); + const refreshGrantFingerprint = priorCredential.refreshToken === cred.refreshToken + ? current.refreshGrantFingerprint ?? refreshGrantFingerprintForToken(cred.refreshToken) + : refreshGrantFingerprintForToken(cred.refreshToken); + store[id] = { + credential: cred, + generation: generation + 1, + refreshGrantFingerprint, + replacedAt: current.replacedAt, + ...preservedValidationMetadata(current), + }; + + // Each alias carries its OWN committed generation: aliases need not share one, and the plan + // reconciliation below is generation-fenced, so an id alone would be reconciled at the wrong fence. + const propagatedAliases: { id: string; generation: number }[] = []; + // Nothing to propagate when the grant did not actually rotate: the aliases already hold it. + // An absent owner identity fails closed: two empty strings compare equal but prove nothing about + // which upstream account either record was meant to use, and a matching bearer snapshot only + // shows they copied the same token once. Leave those dormant records alone. + if ( + priorFingerprint !== undefined + && priorCredential.refreshToken !== cred.refreshToken + && !!priorCredential.chatgptAccountId + ) { + for (const [aliasId, alias] of Object.entries(store)) { + if (aliasId === id || alias.deletedAt != null || !alias.credential) continue; + if (recordGrantFingerprint(alias) !== priorFingerprint) continue; + if (alias.credential.accessToken !== priorCredential.accessToken) continue; + if (alias.credential.expiresAt !== priorCredential.expiresAt) continue; + if (!alias.credential.chatgptAccountId) continue; + if (alias.credential.chatgptAccountId !== priorCredential.chatgptAccountId) continue; + const aliasGeneration = alias.generation + 1; + store[aliasId] = { + // The alias keeps its OWN chatgptAccountId value, which the guard above proved equal. + credential: { ...cred, chatgptAccountId: alias.credential.chatgptAccountId }, + generation: aliasGeneration, + refreshGrantFingerprint, + replacedAt: alias.replacedAt, + ...preservedValidationMetadata(alias), + }; + propagatedAliases.push({ id: aliasId, generation: aliasGeneration }); + } + } + persist(store); + return { committed: true, propagatedAliases }; + }); +} + export function tombstoneCodexAccount(id: string): number { return withCredentialMutationLockSync(() => { const store = loadCodexAccountRecordStore(); @@ -288,6 +379,12 @@ function withCredentialMutationLockSync(fn: () => T): T { type CodexTokenResult = { accessToken: string; chatgptAccountId: string; generation: number }; type CodexRefreshResult = CodexTokenResult & { credential?: CodexAccountCredentials; + /** + * Records that adopted this refresh's rotated credential through same-grant propagation, each + * with its own committed generation (#2892 gap 3). Carried on the result so the flight settles + * every plan in one place rather than the commit doing its own (#2933). + */ + propagatedAliases?: { id: string; generation: number }[]; /** * Grant the returned credential actually belongs to. * @@ -394,12 +491,20 @@ function findFreshCredentialForGrant( refreshGrantFingerprint: string, excludeId: string, rejectedAccessToken?: string, + expectedChatgptAccountId?: string, ): CodexAccountCredentials | null { const now = Date.now(); const records = loadCodexAccountRecordStore(); + // Adoption copies another record's access AND refresh tokens onto the caller, so the two records + // must be the same upstream identity. A grant fingerprint is `sha256` of the refresh token and + // carries no identity claim, and nothing here guarantees one grant cannot span two accounts, so + // require both ids to be present and exactly equal rather than inferring identity from the grant. + if (!expectedChatgptAccountId) return null; for (const [candidateId, candidate] of Object.entries(records)) { if (candidateId === excludeId || candidate.deletedAt != null || !candidate.credential) continue; if (recordGrantFingerprint(candidate) !== refreshGrantFingerprint) continue; + if (!candidate.credential.chatgptAccountId) continue; + if (candidate.credential.chatgptAccountId !== expectedChatgptAccountId) continue; // A sibling alias can hold a still-unexpired copy of the exact token upstream // just rejected. Reusing it would bump the generation and replay the identical // bearer — a second 401 dressed up as recovery. @@ -679,6 +784,7 @@ async function resolveCodexToken( refreshGrantFingerprint, id, forced?.rejectedAccessToken, + lockedCred.chatgptAccountId, ); if (sameGrantFreshCredential) { if (!saveCodexAccountCredentialIfGeneration(id, startGeneration, sameGrantFreshCredential)) { @@ -745,14 +851,26 @@ async function resolveCodexToken( expiresAt: safeExpiresAt, chatgptAccountId: lockedCred.chatgptAccountId, }; - if (!saveCodexAccountCredentialIfGeneration(id, startGeneration, updated)) { + // Commit to the owner and, in the same write, to any record that is still an untouched + // duplicate of the credential this flight started from (#2892 gap 3). Without this the rotated + // grant reaches only the owner and live joiners, and a dormant same-grant record is left + // holding a refresh token upstream has invalidated. + const commit = commitRefreshedCodexCredentialWithAliases(id, startGeneration, updated); + if (!commit.committed) { throw new CodexCredentialGenerationConflictError(); } + if (commit.propagatedAliases.length > 0) { + console.warn(`[codex-auth] rotated refresh grant propagated to ${commit.propagatedAliases.length} dormant same-grant account record(s)`); + } return { accessToken: updated.accessToken, chatgptAccountId: updated.chatgptAccountId, generation: startGeneration + 1, credential: updated, + // Aliases that adopted this rotated credential travel on the result so the FLIGHT settles + // their plans in the same single place as the owner's (#2933). Each carries its own committed + // generation because the plan note is generation-fenced. + ...(commit.propagatedAliases.length > 0 ? { propagatedAliases: commit.propagatedAliases } : {}), // The grant this flight was OPENED for, not the rotated one it produced. Joiners // are waiting on that key, and a successful refresh normally rotates the refresh // token — tagging the new grant would make every legitimate joiner look foreign. @@ -774,6 +892,12 @@ async function resolveCodexToken( */ const refreshPromise = fetchPromise.then(async (result): Promise => { await notePlanFromRefreshedAccessToken(id, result.accessToken, result.generation); + // One settlement path for the whole flight: the refreshing account, then any dormant alias that + // adopted the same rotated JWT. An alias holds the identical access token, so a changed + // `chatgpt_plan_type` applies to it too, and its cached-token fast path would never reconcile it. + for (const alias of result.propagatedAliases ?? []) { + await notePlanFromRefreshedAccessToken(alias.id, result.accessToken, alias.generation); + } return result; }).finally(() => { if (refreshLocks.get(refreshGrantFingerprint) === flight) refreshLocks.delete(refreshGrantFingerprint); diff --git a/src/codex/routing.ts b/src/codex/routing.ts index 24fce08cfa..250aac9636 100644 --- a/src/codex/routing.ts +++ b/src/codex/routing.ts @@ -5,7 +5,7 @@ import { codexAccountLogLabel } from "./account-label"; import { isCodexAccountPaused } from "./account-pause"; import { clearCodexAccountPin, codexAccountPriorityLookup, pinnedCodexAccountId } from "./account-priority"; import { isCodexAccountUsable, type CodexAccountUsabilityOptions } from "./account-usability"; -import { isAccountNeedsReauth, markAccountNeedsReauth } from "./account-runtime-state"; +import { clearAccountNeedsReauth, isAccountNeedsReauth, markAccountNeedsReauth } from "./account-runtime-state"; import { POOL_KEY_CODEX, normalizeAccountPoolStickyLimit, @@ -91,6 +91,15 @@ type CodexUpstreamHealth = { * flaky account without throwing CodexAccountCooldownError (hard-only). */ softAvoidUntil?: number; + /** + * Credential generation a 401/403 quarantine was derived from (#2892 gap 4). + * + * Provenance lives ON the entry rather than in a side map keyed by account id. A side map spends + * "whatever health is current when the old credential is found dead", which deletes a later + * unrelated entry: a G1 401, then a G2 save, then a genuine G2 503 would lose the 503. Only the + * entry that carries this field can be spent, and any later write simply replaces it. + */ + credentialFailureGeneration?: number; }; const CODEX_DEFAULT_QUOTA_COOLDOWN_MS = 60_000; @@ -127,6 +136,22 @@ const upstreamHealth = new Map(); * from account-wide Retry-After/default throttles and transient health. */ const quotaScopedHealth = new Map>(); +/** + * Spend a credential-failure health entry whose credential no longer exists (#2892 gap 4). + * + * A 401/403 describes one CREDENTIAL, not an account, and a replacement can land at any point after + * the outcome is recorded — so re-reading the store inside `recordCodexUpstreamOutcome` narrows the + * window without closing it. The reader decides instead, and it may only spend an entry that + * actually carries credential provenance: a later transient or quota write replaces the entry and + * with it the tag, so this can never delete evidence that belongs to a different failure. + */ +function dropSpentCredentialFailure(accountId: string): void { + const health = upstreamHealth.get(accountId); + const generation = health?.credentialFailureGeneration; + if (health === undefined || generation === undefined) return; + if (isCodexAccountGenerationLive(accountId, generation)) return; + upstreamHealth.delete(accountId); +} let lastReconciledGeneration = 0; let liveHealthAccountIds = new Set(); @@ -311,6 +336,7 @@ export function reconcileCodexRoutingHealth(context: GenerationContext): number export function getCodexUpstreamHealth( accountId: string, ): CodexUpstreamHealth | null { + dropSpentCredentialFailure(accountId); return upstreamHealth.get(accountId) ?? null; } @@ -690,7 +716,13 @@ function withProbeLeaseReleased(health: CodexUpstreamHealth, now: number): Codex */ function preservedCooldownFields(health: CodexUpstreamHealth | undefined): Partial { if (!health) return {}; - const { consecutiveFailures: _f, consecutiveSuccesses: _s, lastFailureStatus: _st, lastFailureAt: _at, softAvoidUntil: _sa, ...cooldownFields } = health; + // `credentialFailureGeneration` is provenance for ONE credential failure, so it must not survive + // into a later transient or quota entry — otherwise that entry inherits the tag and gets spent + // when the old credential dies, deleting evidence that was never about it (#2892 gap 4 review). + const { + consecutiveFailures: _f, consecutiveSuccesses: _s, lastFailureStatus: _st, lastFailureAt: _at, + softAvoidUntil: _sa, credentialFailureGeneration: _cg, ...cooldownFields + } = health; return cooldownFields; } @@ -1587,6 +1619,7 @@ function applyQuotaAutoSwitch( function shouldFailover(config: OcxConfig, accountId: string, now: number): boolean { const threshold = config.upstreamFailoverThreshold ?? 3; if (threshold <= 0) return false; + dropSpentCredentialFailure(accountId); const health = upstreamHealth.get(accountId); if (health?.lastFailureAt && now - health.lastFailureAt > CODEX_FAILURE_WINDOW_MS) return false; return !!health && health.consecutiveFailures >= threshold; @@ -2111,6 +2144,16 @@ export function recordCodexUpstreamOutcome( const now = meta.now ?? Date.now(); const outcomeClass = classifyCodexUpstreamOutcome(outcome, meta.denial); const quotaScope = codexQuotaScopeForModel(meta.modelId); + /* + * Spend a stale credential failure BEFORE any branch reads health (#2892 gap 4 review). + * + * Reader-side spending alone is not enough: the transient and workspace branches derive their new + * entry from the current one, so a spent G1 401 would donate its `consecutiveFailures` to G2's + * first genuine 503 and drop the tag while doing it. The account then reaches the failover + * threshold one failure early, and no later read can tell. Clearing it here means every branch + * starts from evidence that still describes a live credential. + */ + dropSpentCredentialFailure(accountId); if (outcomeClass === "success") { const scopedProbe = meta.probeQuotaScope ? scopedHealthFor(accountId, meta.probeQuotaScope) @@ -2213,13 +2256,39 @@ export function recordCodexUpstreamOutcome( ) { return; } + /* + * The pre-check above closes the same-process race, but not a cross-process one (#2892 gap 4). + * `isCodexAccountGenerationLive` is an unlocked read while credential writers coordinate under + * the mutation lock, and OS preemption needs no `await` — so another process can replace the + * credential after this check, or at any point after this whole function returns. No re-read here + * can close that: a replacement is always free to land one instruction later. + * + * Taking the credential lock is not an option either: it runs with `busy_timeout=0`, so acquiring + * it per outcome would turn ordinary contention into thrown request-path errors. + * + * So the evidence is TAGGED with the credential it describes and judged when it is READ. The + * health entry carries `credentialFailureGeneration` and the reauth map carries the same + * generation; `dropSpentCredentialFailure` and `isAccountNeedsReauth` discard an entry whose + * credential is gone. A later transient or quota write replaces the entry along with its tag, and + * `preservedCooldownFields` drops the tag explicitly, so this provenance can never be spent + * against a failure it did not describe. + * + * Affinity sweeping needs no tag: an affinity entry already carries a credential generation and + * self-invalidates on the next check, and re-adding swept entries would be a worse bug. + */ upstreamHealth.set(accountId, { consecutiveFailures: 1, lastFailureStatus, lastFailureAt: now, + // Provenance rides on the entry: only this failure can be spent when its credential dies. + ...(meta.credentialGeneration !== undefined + ? { credentialFailureGeneration: meta.credentialGeneration } + : {}), }); quotaScopedHealth.delete(accountId); - markAccountNeedsReauth(accountId, writerGeneration); + // The reauth flag carries the same provenance, so a replacement landing after this call cannot + // inherit a quarantine that was never about it. + markAccountNeedsReauth(accountId, writerGeneration, meta.credentialGeneration); clearThreadAccountMapForAccount(accountId); return; } diff --git a/src/providers/openai-sidecar.ts b/src/providers/openai-sidecar.ts index 892b08462b..70e788882b 100644 --- a/src/providers/openai-sidecar.ts +++ b/src/providers/openai-sidecar.ts @@ -140,6 +140,9 @@ export async function resolveFirstUsableOpenAiSidecar( probeLeaseId: authContext.probeLeaseId, probeQuotaScope: authContext.probeQuotaScope, writerGeneration: authContext.writerGeneration, + // 401/403 here is evidence about this exact stored credential; without the generation a + // replacement inherits the quarantine (#2892 gap 4). + ...(authContext.kind === "pool" ? { credentialGeneration: authContext.generation } : {}), }, ), }; @@ -172,6 +175,8 @@ export async function resolveFirstUsableOpenAiSidecar( threadId: authContext.affinityKey, probeLeaseId: authContext.probeLeaseId, writerGeneration: authContext.writerGeneration, + // Same fence as the exact-account recorder above (#2892 gap 4). + ...(authContext.kind === "pool" ? { credentialGeneration: authContext.generation } : {}), }, ), } diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 4557af6c03..b06237fac9 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -399,6 +399,11 @@ export function sidecarOutcomeRecorder( probeLeaseId: authCtx.probeLeaseId, probeQuotaScope: authCtx.probeQuotaScope, writerGeneration: authCtx.writerGeneration, + // A vision or web-search sidecar can return 401/403, and that is evidence about the exact + // stored credential it used. Without the generation it becomes an account-wide quarantine + // that a replacement inherits (#2892 gap 4). `main-pool` has no stored-record generation, so + // it keeps the unfenced account-wide semantics. + ...(authCtx.kind === "pool" ? { credentialGeneration: authCtx.generation } : {}), }) : undefined; } diff --git a/tests/codex-account-store.test.ts b/tests/codex-account-store.test.ts index 6dcebd4d1e..fe7274496d 100644 --- a/tests/codex-account-store.test.ts +++ b/tests/codex-account-store.test.ts @@ -1005,6 +1005,144 @@ describe("codex-account-store CRUD", () => { globalThis.fetch = originalFetch; } }); + + test("a successful refresh advances an untouched dormant same-grant alias in the same write (#2892 gap 3)", async () => { + const { getValidCodexToken, readCodexAccountRecord, saveCodexAccountCredential } = + await import("../src/codex/account-store"); + const expiresAt = 0; + const shared = { refreshToken: "dormant-grant", expiresAt, chatgptAccountId: "acc" }; + // Owner drives the refresh. `dormant` is an untouched duplicate that never calls in — the + // record the rotated grant used to skip, leaving it to send a dead grant on its next refresh. + saveCodexAccountCredential("dormant-owner", { accessToken: "shared-old", ...shared }); + saveCodexAccountCredential("dormant-alias", { accessToken: "shared-old", ...shared }); + // Negative cases: each must be left strictly alone. + saveCodexAccountCredential("alias-other-account", { + accessToken: "shared-old", + refreshToken: "dormant-grant", + expiresAt, + chatgptAccountId: "different-acc", + }); + saveCodexAccountCredential("alias-moved-on", { accessToken: "already-newer", ...shared }); + saveCodexAccountCredential("alias-other-grant", { + accessToken: "shared-old", + refreshToken: "unrelated-grant", + expiresAt, + chatgptAccountId: "acc", + }); + const aliasGeneration = readCodexAccountRecord("dormant-alias")!.generation; + const otherAccountGeneration = readCodexAccountRecord("alias-other-account")!.generation; + const movedOnGeneration = readCodexAccountRecord("alias-moved-on")!.generation; + + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => Response.json({ + access_token: "rotated-access", + refresh_token: "rotated-grant", + expires_in: 3600, + })) as typeof fetch; + + try { + await getValidCodexToken("dormant-owner"); + + const owner = readCodexAccountRecord("dormant-owner")!; + expect(owner.credential).toMatchObject({ accessToken: "rotated-access", refreshToken: "rotated-grant" }); + + // The dormant alias adopts the rotated credential WHOLE — access token, refresh token, and + // expiry together — so its bumped generation still means "newer JWT", which is what the + // plan-from-token fence reads it as. + const alias = readCodexAccountRecord("dormant-alias")!; + expect(alias.credential?.refreshToken).toBe("rotated-grant"); + expect(alias.credential?.accessToken).toBe("rotated-access"); + expect(alias.credential?.expiresAt).toBe(owner.credential!.expiresAt); + expect(alias.credential?.chatgptAccountId).toBe("acc"); + expect(alias.generation).toBe(aliasGeneration + 1); + expect(alias.refreshGrantFingerprint).toBe(owner.refreshGrantFingerprint); + + // A same-grant record on a DIFFERENT chatgpt account is not provably the same identity: a + // fingerprint is sha256 of the refresh token and carries no identity claim. + const otherAccount = readCodexAccountRecord("alias-other-account")!; + expect(otherAccount.credential?.refreshToken).toBe("dormant-grant"); + expect(otherAccount.generation).toBe(otherAccountGeneration); + + // An alias whose access token already moved on must NOT be given a generation bump with a + // stale JWT, and must not be handed back a possibly-rejected bearer. + const movedOn = readCodexAccountRecord("alias-moved-on")!; + expect(movedOn.credential?.accessToken).toBe("already-newer"); + expect(movedOn.credential?.refreshToken).toBe("dormant-grant"); + expect(movedOn.generation).toBe(movedOnGeneration); + + expect(readCodexAccountRecord("alias-other-grant")!.credential?.refreshToken).toBe("unrelated-grant"); + } finally { + globalThis.fetch = originalFetch; + } + }); + + test("a tombstoned same-grant record is not resurrected by grant propagation (#2892 gap 3)", async () => { + const { getValidCodexToken, readCodexAccountRecord, saveCodexAccountCredential, tombstoneCodexAccount } = + await import("../src/codex/account-store"); + const shared = { accessToken: "tomb-old", refreshToken: "tomb-grant", expiresAt: 0, chatgptAccountId: "acc" }; + saveCodexAccountCredential("tomb-owner", { ...shared }); + saveCodexAccountCredential("tomb-deleted", { ...shared }); + tombstoneCodexAccount("tomb-deleted"); + + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => Response.json({ + access_token: "tomb-new", + refresh_token: "tomb-rotated", + expires_in: 3600, + })) as typeof fetch; + try { + await getValidCodexToken("tomb-owner"); + const deleted = readCodexAccountRecord("tomb-deleted")!; + expect(deleted.deletedAt).toBeGreaterThan(0); + expect(deleted.credential).toBeUndefined(); + } finally { + globalThis.fetch = originalFetch; + } + }); + + + test("a same-grant sibling on a DIFFERENT upstream identity is never adopted (#2892 review)", async () => { + const { forceRefreshCodexPoolToken, getCodexAccountCredential, readCodexAccountRecord, saveCodexAccountCredential } = + await import("../src/codex/account-store"); + // Both records share one stored grant, but they claim different upstream accounts. Adoption + // copies BOTH tokens, so treating a shared fingerprint as proof of identity would hand this + // caller another account's credential. + saveCodexAccountCredential("foreign-caller", { + accessToken: "rejected-token", + refreshToken: "foreign-grant", + expiresAt: Date.now() + 3600_000, + chatgptAccountId: "acct-one", + }); + saveCodexAccountCredential("foreign-sibling", { + accessToken: "sibling-fresh", + refreshToken: "foreign-grant", + expiresAt: Date.now() + 3600_000, + chatgptAccountId: "acct-two", + }); + const generation = readCodexAccountRecord("foreign-caller")!.generation; + + const originalFetch = globalThis.fetch; + let tokenCalls = 0; + globalThis.fetch = (async () => { + tokenCalls += 1; + return Response.json({ access_token: "own-new", refresh_token: "own-rotated", expires_in: 3600 }); + }) as typeof fetch; + try { + const result = await forceRefreshCodexPoolToken("foreign-caller", { + rejectedGeneration: generation, + rejectedAccessToken: "rejected-token", + }); + // A real refresh must have run instead of adopting the foreign sibling. + expect(tokenCalls).toBe(1); + expect(result.accessToken).toBe("own-new"); + expect(getCodexAccountCredential("foreign-caller")?.accessToken).not.toBe("sibling-fresh"); + // The sibling is untouched: this path must not write to another identity's record. + expect(getCodexAccountCredential("foreign-sibling")?.accessToken).toBe("sibling-fresh"); + } finally { + globalThis.fetch = originalFetch; + } + }); + }); describe("shared refresh flight plan reconciliation (#2892 gap 2 follow-up)", () => { @@ -1091,4 +1229,37 @@ describe("shared refresh flight plan reconciliation (#2892 gap 2 follow-up)", () resetJwtPlanNotesForTests(); } }); + + test("records with EMPTY account ids are never treated as the same identity (#2892 review)", async () => { + const { getValidCodexToken, readCodexAccountRecord, saveCodexAccountCredential } = + await import("../src/codex/account-store"); + // Grant fingerprint, access token and expiry all match, and both ids are "". Two empty strings + // compare equal but prove nothing about which upstream account either record was meant to use, + // so propagation must fail closed rather than write a credential into an unidentified record. + const shared = { accessToken: "anon-old", refreshToken: "anon-grant", expiresAt: 0, chatgptAccountId: "" }; + saveCodexAccountCredential("anon-owner", { ...shared }); + saveCodexAccountCredential("anon-alias", { ...shared }); + const aliasGeneration = readCodexAccountRecord("anon-alias")!.generation; + + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => Response.json({ + access_token: "anon-new", + refresh_token: "anon-rotated", + expires_in: 3600, + })) as typeof fetch; + try { + await getValidCodexToken("anon-owner"); + + // The owner still rotates normally. + expect(readCodexAccountRecord("anon-owner")!.credential?.refreshToken).toBe("anon-rotated"); + // The unidentified record is untouched, generation included. + const alias = readCodexAccountRecord("anon-alias")!; + expect(alias.credential?.accessToken).toBe("anon-old"); + expect(alias.credential?.refreshToken).toBe("anon-grant"); + expect(alias.generation).toBe(aliasGeneration); + } finally { + globalThis.fetch = originalFetch; + } + }); + }); diff --git a/tests/codex-plan.test.ts b/tests/codex-plan.test.ts index 0c4b9c0936..788264e1da 100644 --- a/tests/codex-plan.test.ts +++ b/tests/codex-plan.test.ts @@ -198,3 +198,52 @@ describe("WHAM-wins plan provenance gate (release-audit fix)", () => { }); }); + +describe("rotated-JWT plan reconciliation across propagated aliases (#2892 gap 3)", () => { + test("a plan-changing rotated JWT reconciles the alias, not just the refresh owner", async () => { + const { getValidCodexToken, readCodexAccountRecord } = await import("../src/codex/account-store"); + const oldJwt = chatgptPlanJwt("plus"); + const shared = { refreshToken: "plan-grant", expiresAt: 0, chatgptAccountId: "acct" }; + saveCodexAccountCredential("plan-owner", { accessToken: oldJwt, ...shared }); + saveCodexAccountCredential("plan-alias", { accessToken: oldJwt, ...shared }); + // Advance the alias so its generation DIVERGES from the owner's. Without this both land on the + // same number and an assertion about the per-alias fence would pass even if the code used the + // owner's generation — a vacuous test. Re-saving the identical credential keeps the record an + // eligible untouched duplicate while bumping only its generation. + saveCodexAccountCredential("plan-alias", { accessToken: oldJwt, ...shared }); + saveCodexAccountCredential("plan-alias", { accessToken: oldJwt, ...shared }); + saveConfig({ + ...loadConfig(), + codexAccounts: [ + { id: "plan-owner", email: "owner@test", plan: "plus" }, + { id: "plan-alias", email: "alias@test", plan: "plus" }, + ], + } as OcxConfig); + + const rotatedJwt = chatgptPlanJwt("pro"); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => Response.json({ + access_token: rotatedJwt, + refresh_token: "plan-grant-rotated", + expires_in: 3600, + })) as typeof fetch; + try { + await getValidCodexToken("plan-owner"); + + // The alias holds the rotated Pro JWT after propagation... + const alias = readCodexAccountRecord("plan-alias")!; + expect(alias.credential?.accessToken).toBe(rotatedJwt); + + // ...so its configured plan must be reconciled too. Reconciling only the owner left the alias + // on "plus" while carrying a Pro credential, and its cached-token fast path never repairs + // that, so quota scoring stayed wrong until a restart or a WHAM refresh. + const accounts = loadConfig().codexAccounts ?? []; + expect(accounts.find(a => a.id === "plan-owner")?.plan).toBe("pro"); + expect(accounts.find(a => a.id === "plan-alias")?.plan).toBe("pro"); + // The alias is fenced at its OWN committed generation, not the owner's. + expect(accounts.find(a => a.id === "plan-alias")?.planCredentialGeneration).toBe(alias.generation); + } finally { + globalThis.fetch = originalFetch; + } + }); +}); diff --git a/tests/codex-routing.test.ts b/tests/codex-routing.test.ts index eb93bc14c2..1ec9f8cfda 100644 --- a/tests/codex-routing.test.ts +++ b/tests/codex-routing.test.ts @@ -34,7 +34,8 @@ import { tryAcquireCodexQuotaProbeLease, } from "../src/codex/routing"; import { clearPoolRotationState } from "../src/codex/pool-rotation"; -import { removeCodexAccountCredential, saveCodexAccountCredential } from "../src/codex/account-store"; +import { captureConfigGeneration } from "../src/lib/state-store-sweeper"; +import { readCodexAccountRecord, removeCodexAccountCredential, saveCodexAccountCredential } from "../src/codex/account-store"; import { clearAccountNeedsReauth, clearAccountQuota, @@ -484,6 +485,138 @@ describe("codex routing", () => { expect(resolveCodexAccountForThread("credential-403-next", config)).toBe("b"); }); + test("a 401 does not quarantine a credential that replaced the rejected one AFTER the outcome (#2892 gap 4)", () => { + const config = makeConfig(); + updateAccountQuota("a", 10); + updateAccountQuota("b", 20); + saveTestCredential("a"); + const generation = readCodexAccountRecord("a")!.generation; + + // Record the 401 while the rejected credential is still the live one, so every side effect is + // legitimately applied. This is the ordering @Ingwannu reproduced: the replacement lands AFTER + // recordCodexUpstreamOutcome returns, which no post-write re-read inside it can ever observe. + recordCodexUpstreamOutcome(config, "a", 401, { credentialGeneration: generation }); + expect(isAccountNeedsReauth("a")).toBe(true); + expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 1, lastFailureStatus: 401 }); + + // Another process replaces the credential. The 401 was evidence about a credential that no + // longer exists, so it must not hold the replacement out of rotation. + saveTestCredential("a"); + expect(readCodexAccountRecord("a")!.generation).toBe(generation + 1); + + expect(isAccountNeedsReauth("a")).toBe(false); + expect(getCodexUpstreamHealth("a")).toBeNull(); + expect(resolveCodexAccountForThread("gap4-replacement-selectable", config)).toBe("a"); + }); + + test("a 401 on the live credential still quarantines the account (#2892 gap 4 does not over-roll-back)", () => { + const config = makeConfig(); + updateAccountQuota("a", 10); + updateAccountQuota("b", 20); + saveTestCredential("a"); + const generation = readCodexAccountRecord("a")!.generation; + + // No concurrent replacement: the evidence is about the credential still in the store, so every + // side effect must survive. This is the assertion that stops the rollback from being a blanket + // "never quarantine" regression. + recordCodexUpstreamOutcome(config, "a", 401, { credentialGeneration: generation }); + + expect(isAccountNeedsReauth("a")).toBe(true); + expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 1, lastFailureStatus: 401 }); + }); + + + test("a later transient failure is not deleted by a spent credential-failure tag (#2892 gap 4 review)", () => { + const config = makeConfig(); + updateAccountQuota("a", 10); + updateAccountQuota("b", 20); + saveTestCredential("a"); + const generation = readCodexAccountRecord("a")!.generation; + + // G1 401, then the credential is replaced, then a GENUINE 503 against G2 — all before any + // health read. Provenance keyed only by account id would spend "whatever health is current" + // and delete this 503; provenance on the entry cannot, because the 503 write replaced the tag. + recordCodexUpstreamOutcome(config, "a", 401, { credentialGeneration: generation }); + saveTestCredential("a"); + recordCodexUpstreamOutcome(config, "a", 503); + + expect(getCodexUpstreamHealth("a")).toMatchObject({ lastFailureStatus: 503 }); + expect(isAccountNeedsReauth("a")).toBe(false); + }); + + test("a workspace denial overwriting a spent credential failure survives the read (#2892 gap 4 review)", () => { + const config = makeConfig(); + updateAccountQuota("a", 10); + updateAccountQuota("b", 20); + saveTestCredential("a"); + const generation = readCodexAccountRecord("a")!.generation; + + recordCodexUpstreamOutcome(config, "a", 401, { credentialGeneration: generation }); + saveTestCredential("a"); + // A workspace denial is a different ownership class and must not be collateral damage. + recordCodexUpstreamOutcome(config, "a", 403, { denial: "workspace" }); + + expect(getCodexUpstreamHealth("a")).toMatchObject({ lastFailureStatus: 403 }); + }); + + + test("a sidecar 401 does not quarantine the credential that replaced it (#2892 gap 4 review)", async () => { + const { sidecarOutcomeRecorder } = await import("../src/server/responses/core"); + const config = makeConfig(); + updateAccountQuota("a", 10); + updateAccountQuota("b", 20); + saveTestCredential("a"); + const generation = readCodexAccountRecord("a")!.generation; + + // A vision or web-search sidecar returns 401 for a stored Pool credential. Recording that + // without the credential generation produced an account-wide quarantine, so the replacement + // inherited it and the account stayed unroutable. + const record = sidecarOutcomeRecorder(config, { + kind: "pool", + accountId: "a", + // Use the CURRENT captured generation, as a production pool auth context does. A hardcoded 0 + // is below whatever reconciliation state earlier tests advanced to, so + // recordCodexUpstreamOutcome could reject the outcome at its writer-generation guard and the + // assertion would pass without ever reaching the credential-generation logic under test. + writerGeneration: captureConfigGeneration(), + generation, + accessToken: "access-a", + chatgptAccountId: "acct-a", + }); + expect(record).toBeDefined(); + record!(401); + // Guard the guard: if this is false the outcome never applied, so the assertions below would be + // vacuous rather than proving the replacement is not quarantined. + expect(isAccountNeedsReauth("a")).toBe(true); + + saveTestCredential("a"); + expect(isAccountNeedsReauth("a")).toBe(false); + expect(getCodexUpstreamHealth("a")).toBeNull(); + }); + + + test("a spent credential failure does not donate its failure count to a later transient (#2892 gap 4 review)", () => { + const config = makeConfig(); + updateAccountQuota("a", 10); + updateAccountQuota("b", 20); + saveTestCredential("a"); + const generation = readCodexAccountRecord("a")!.generation; + recordCodexUpstreamOutcome(config, "a", 401, { credentialGeneration: generation }); + saveTestCredential("a"); + // G2's first genuine transient must start the count at 1. Inheriting the spent 401's count + // pushes the account over the failover threshold a failure early, and because the transient + // write drops the provenance tag, no later read can detect that it happened. + recordCodexUpstreamOutcome(config, "a", 503); + expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 1, lastFailureStatus: 503 }); + // The same inheritance path exists for a workspace denial. + clearCodexUpstreamHealthForAccount("a"); + recordCodexUpstreamOutcome(config, "a", 401, { credentialGeneration: readCodexAccountRecord("a")!.generation }); + saveTestCredential("a"); + recordCodexUpstreamOutcome(config, "a", 403, { denial: "workspace" }); + expect(getCodexUpstreamHealth("a")).toMatchObject({ consecutiveFailures: 1, lastFailureStatus: 403 }); + }); + + test("connect failures contribute to transient failover", () => { const config = makeConfig(); updateAccountQuota("a", 10);