Account Pooler: a rejected proactive OAuth refresh errors an account whose access token is still valid; background refresh keeps replaying the rejected token #6167

Summary

When Account Pooler's proactive OAuth refresh (the last 300 s before expiresAt) gets HTTP 400 invalid_grant on the request path, the pool marks the account as errored straight away, even though its access token is still valid for up to 5 more minutes. The account then stays out of routing until someone runs bb pool account enable or bb pool account refresh, and here that left three threads waiting 2 h 10 min for another account's reset. The 5-minute background usage refresh has the opposite problem: it sets no error and re-sends the already rejected refresh token every 5 minutes (158 times in 13.4 h for one account). I'm asking to change both, as described under Expected.

Versions and environment

• bb 0.46.1 (latest release, desktop-v0.46.1), installed with npm (bb-app), running as a systemd user service. Debian 13, Linux 6.12.113 x86_64, Node v24.22.1.
• Built-in Account Pooler (account-pool 0.1.0) with Claude and Codex routing on, and the built-in Provider Retry.
• Claude Code 2.1.306–2.1.308 during the incidents (2.1.309 since 2026-10-23 03:05Z). codex-cli 0.163.0 during the incidents (0.164.0 now). All incidents below are on Claude accounts. Codex accounts go through the same hub code, and the pinned test covers both.
• Unusual setup:
  • The pool accounts were added with bb pool account add --provider claude --import, and the host keeps using the same logins: terminal claude, the account switcher ccswap v0.39.2, and pooled Claude Code threads (which still refresh ~/.claude/.credentials.json themselves). So two holders share one refresh-token chain.
  • A local script automation (not a bb feature) copies the newer token between the host login and the pool's secret file, then runs bb pool account enable. It shortened every incident below. Without it, the accounts would have stayed errored.
  • switchThreshold has been 1 since 2026-10-21T23:20Z, so the threshold played no part in the 10-22 and 10-23 incidents.
• Unchanged on main:
  • plugins/account-pool/src/{hub,provider-adapter,operations,store}.ts are identical between desktop-v0.46.1 and 1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5.
  • Since the tag, the plugin has only had UI, env-listing and typecheck-tooling commits (Make Account Pooler rows clickable across their highlighted area #5900, List provider models with plugin-contributed provider environment #5767, Typecheck with tsrs instead of TypeScript 7 tsc #5861).
  • The installed bundle (server/dist/builtin-plugins/account-pool/dist/server.js) contains the same logic.

Steps to reproduce

I saw this in production and did not reproduce it on demand. It needs a second holder of the same grant to win a sub-second race.

Live path (what happened here):

1. bb plugin enable account-pool, then bb pool account add --provider claude --import. This copies the host login from ~/.claude/.credentials.json, including the refresh token. The docs recommend it: "Prefer --import for an existing Claude Code login".
2. Keep using the same login on the host.
3. In the last 5 minutes before the shared expiresAt, the host side refreshes first and a pooled request arrives just after. Both sides use the same 300 s window, so nothing special is needed for this to happen.
   • At 2026-10-23T00:49:50Z the host-side refresh came from the Claude Code process of pooled thread thr_cv4aev22bx. Why a pool-routed thread refreshes the host login at all is Account Pooler: pool-routed Claude Code and Codex threads, and CLIs they start, should authenticate only with the hub token and leave the host's own logins untouched #6171.
   • The pool's own refresh for that thread's next request failed 0.44 s later.
4. bb pool account list --json now shows the account with error: "OAuth refresh failed with HTTP 400. invalid_grant.". If no other account is eligible, requests get a 429 with a far-off Next account … time.

Deterministic path (unverified: I did not run it). An existing upstream test asserts exactly this for Claude and Codex. My checkout is read-only and has no dependencies installed.

cd plugins/account-pool
pnpm exec vitest run --config vitest.config.ts src/server.test.ts -t "keeps invalid_grant accounts excluded despite a valid access token"

What the test does (server.test.ts#L5694-L5716, #L5776-L5812):

• It sets expiresAt to now + 10 min, advances 6 min (4 min of validity left) and makes the token endpoint return 400 invalid_grant.
• It asserts a 429, an account error and zero upstream calls.
• After the token endpoint is fixed, it still asserts a 429 and no new refresh call.

Background replay: when an account's refresh token has been rejected and no request selects that account, each 5-minute refreshUsage() tick re-sends the same refresh token, and the account never enters error. I saw this in the log but did not write a test for it.

Did not reproduce with transient token-endpoint failures (network, timeout, 5xx). Those keep serving the unexpired token with backoff, as intended. For example, account C had 155 s left at 2026-10-20T14:11:46Z and kept serving after a network error.

Expected vs actual

Actual. Times are 2026-10-23 UTC. The pool log is ~/.bb/plugins/account-pool/logs/plugin.log; account ids are redacted.

00:49:50.876  {"ts":1792716590876,"level":"warn","message":"Account Pooler claude account <account B> OAuth refresh failed: OAuth refresh failed with HTTP 400. invalid_grant. expiresAt=1792716887319, forced=false."}
# access token still valid for 296 s; forward() marks account B as errored

# bb thread log <thr> --json --all, provider/error events
00:49:51.367 thr_cv4aev22bx  API Error: Request rejected (429) · No Account Pooler account is currently eligible. Next account is expected to be available at 2026-10-23T02:59:59.798Z (in 7809 seconds).
00:49:53.635 thr_cehnf3bqqh  API Error: Request rejected (429) · No Account Pooler account is currently eligible. Next account is expected to be available at 2026-10-23T02:59:59.798Z (in 7807 seconds).
00:49:55.110 thr_t9xt22j7id  API Error: Request rejected (429) · No Account Pooler account is currently eligible. Next account is expected to be available at 2026-10-23T02:59:59.798Z (in 7805 seconds).

# 00:50:07.143-00:50:09.118  the local sync script writes the host's new token into the pool secret file and runs `bb pool account enable`; account B is usable again
# 03:00:25-03:00:35  Provider Retry re-submits the three turns (client/turn/requested, retryAttempt 2), at the quoted reset
#                    (those retries then failed for an unrelated local reason, a Node binary path removed by an upgrade; not part of this issue)

# Background loop, 2026-10-19, account A: the same rejected refresh token every ~5 min; the account is not marked errored
{"ts":1792405100229,"level":"warn","message":"Account Pooler claude account <account A> OAuth refresh failed: OAuth refresh failed with HTTP 400. invalid_grant. expiresAt=1792405160014, forced=false."}
{"ts":1792405406371,"level":"warn","message":"Account Pooler claude account <account A> OAuth refresh failed: OAuth refresh failed with HTTP 400. invalid_grant. expiresAt=1792405160014, forced=false."}
# ... 158 x HTTP 400 invalid_grant, all with the same expiresAt, through
{"ts":1792453492373,"level":"warn","message":"Account Pooler claude account <account A> OAuth refresh failed: OAuth refresh failed with HTTP 400. invalid_grant. expiresAt=1792405160014, forced=false."}
# the last line (23:44:52Z, off the 5-minute cadence) came from a request, and only that one set the error

Expected:

1. Keep serving while the access token is valid. Treat a non-forced refresh that gets 400/401 while expiresAt > now the way the transient branch already treats it: return the stored access token and do not set the error (hub.ts#L1149-L1155). Record the rejection, keyed by the rejected refresh token.
2. Set the error only when the token can no longer be used, that is, when the access token expires or when upstream rejects it. The existing 401/403 path already handles the second case: it forces a refresh, then calls rejectCredential() and markError (hub.ts#L652-L683).
3. Re-read the stored secret before setting the error, as rejectCredential() already does for upstream rejections (hub.ts#L806-L823). If its refresh token differs from the rejected one, use it.
4. Never replay a refresh token the token endpoint already rejected, on the request path or in the 5-minute background loop.
   • This follows the reasoning in Keep Account Pooler accounts usable when upstream rejects fresh tokens #4385: a rejected token cannot succeed, and replaying a rotated Codex refresh token can trip reuse detection.
   • Once the access token has expired, the background loop should set the error just as the request path does. Then select() and accountAvailability() stop counting an unusable account as eligible.
5. Recover without a manual command when the credential changes. While an account is errored because its refresh token was rejected, each background tick re-reads the stored secret. This is a local file read with no network call. If the refresh token has changed, refresh with it and clear the error on success. enable and refresh keep working as they do today.
6. Make error changes observable. Publish ACCOUNT_POOL_ACCOUNTS_CHANGED when the error is set or cleared. Today markError() and the clear after a successful refresh only write the quota row (hub.ts#L1381-L1384, #L1105-L1114). A cross-plugin "availability changed" signal would also let Provider Retry re-check pending turns. That is requested separately in Provider Retry: re-evaluate pending and declined pool-routed retries when Account Pooler availability changes #6172 and is not part of this issue.

I deliberately do not ask for blind retries of the same refresh token with backoff. The 158 replays above never succeeded, and #4385 explains why they cannot.

Open question I could not verify: does the provider keep accepting the old access token after another holder has refreshed the same grant? If it does not, point 2 still sets the error on the first upstream 401, so the change costs at most one extra round trip.

Evidence

Code at 1e80ca7:

• The refresh window is 300 s (provider-adapter.ts#L121-L134).
• A token-endpoint 400/401 is permanent (OAuthRefreshError); everything else is transient (#L175-L186).
• forward() calls markError on a non-transient refresh failure (hub.ts#L530-L549).
• freshSecret():
  • It re-reads the secret file at the start of each flight (hub.ts#L1049-L1053).
  • An errored account throws its stored error right after that read, so a replaced secret is ignored until enable/refresh (#L1075-L1076).
  • Permanent failures are rethrown (#L1116-L1129).
• Selection skips errored accounts (hub.ts#L853-L861).
• accountAvailability() skips them too. It quotes the earliest reset among the remaining accounts, or returns unavailable/authentication when every enabled account is errored (hub.ts#L1294-L1332).
• The background loop runs every 5 min (hub.ts#L164-L178).
  • It sets the error only when recover is true. That happens only on a forced refresh of an already errored account.
  • Otherwise a rejected refresh is logged and dropped, with no backoff (hub.ts#L334-L369).
• Only manual actions clear the error:
  • enable (operations.ts#L171-L179);
  • refresh (operations.ts#L198-L206);
  • a forced usage refresh from Settings → Usage, which calls the same hub.refreshUsage(id, true).
• The upstream-rejection path re-checks the stored secret before marking an error (hub.ts#L783-L834).
• The docs describe the current behaviour (accounts-and-routing.md#L60-L74).

Pool log, 2026-10-19T02:35Z to 2026-10-23T11:41Z. The table lists the first invalid_grant for each stored token. "s left" is expiresAt minus the log ts.

Other sources:

• The race timing comes from the investigation, which compared credential-file mtimes and Claude Code transcripts against plugin.log. In all three request-path cases, a host-side Claude Code process refreshed the same grant 0.44–0.72 s before the pool's own attempt.
• Heal times come from the sync automation's run records.
• Thread events come from bb thread log <thr> --json --all.
• Investigation: bb thread thr_agudxusj98, reports under pool-research/ in its thread storage. An independent Codex review in the same investigation reached the same conclusion on this defect.

What you ruled out

• Not Disabling Account Pooler breaks Claude: running turns 503-retry for ~3 min, new threads fail with an expired local OAuth login #4352. That issue is the other side of the shared-grant race: disabling the pool leaves the host login with a dead refresh token. Here the shared grant is only the trigger; this issue is about how the pool reacts when it loses. Moving accounts to --login grants removes the trigger here, but the pool still errors an account whose access token is valid whenever its refresh token is rejected.
• Not Account Pooler: pool-routed Claude Code and Codex threads, and CLIs they start, should authenticate only with the hub token and leave the host's own logins untouched #6171. That one asks that pool-routed threads stop refreshing the host login, which is one source of the winning refreshes here. Fixing it removes one trigger but not the pool's reaction.
• Not Keep Account Pooler accounts usable when upstream rejects fresh tokens #4385 (merged). It stopped fresh tokens rejected upstream from becoming permanent errors, and it deliberately kept a token-endpoint 400/401 permanent. This request keeps that. It only defers the error while the access token is valid, and stops the background replay that Keep Account Pooler accounts usable when upstream rejects fresh tokens #4385's own reasoning argues against.
• Not fix(account-pool): recover from temporary OAuth refresh outages #3115 (landed through the stacked, merged fix(account-pool): preserve provider session affinity #3117). It keeps the valid token on transient refresh failures. This asks for the same on a 400/401 while the token is still valid.
• Not Return Account Pool accounts rejected during an outage once usage succeeds #4383 (closed). It cleared errors on usage success and replayed refresh tokens once per cycle. I am not asking for replays.
• Not provider-retry never retries Account Pooler 429/502 failures #4839, Fix provider retries for exhausted account pools #5283 or Retry turns that fail on provider server and connection errors #5931. Provider Retry does retry these 429s. The long wait comes from the reset time the pool quotes, because errored accounts are skipped.
• Not Account Pooler: all-exhausted error doesn't say when an account frees up #5224 (the 429 message text), Account Pooler quota exhaustion leaves Codex reconnecting inside active turns and blocks the shared concurrency queue #6116 (Codex reconnecting inside a turn), Account Pooler: last-resort accounts, per-account switch thresholds, and an opt-in order by earliest weekly reset #4468 / Account Pooler blocks Claude extra usage when subscription quota reaches the switch threshold #4475 (selection order and threshold), or open PR fix(account-pool): reject duplicate OAuth account imports #5709 (duplicate-import rejection, which does not touch refresh).
• Not the threshold: switchThreshold was 1 during the 10-22 and 10-23 incidents.
• Not transport: the failures are HTTP 400 invalid_grant from the token endpoint. The network failures in the same log took the transient path correctly.
• Still on main at 1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5 (see Versions).

Suggested priority and effort (optional)

Medium priority. It affects pools whose accounts were added with --import (the docs recommend it) while the host keeps using the same login. Workarounds are a manual bb pool account enable/refresh, or switching to --login grants. No data is lost, but threads can stall until another account's reset (2 h 10 min here). Effort is low to medium: mostly freshSecret() and refreshAccountUsage(), plus updating the pinned test.

Checks

• I reproduced this on the latest release or on main, or I say above that I could not.
• I searched open and closed issues for the same problem.
• If an agent wrote this, the body ends with > AGENT GENERATED and links the thread or report.

Investigation thread: thr_agudxusj98. Affected threads: thr_cv4aev22bx, thr_cehnf3bqqh, thr_t9xt22j7id, thr_biw5z6pxki.

AGENT GENERATED