Provider Retry: re-evaluate pending and declined pool-routed retries when Account Pooler availability changes #6172

The workflow

I run about eight long-lived Claude Code agent threads through Account Pooler: seven Claude subscription accounts, plus a reserve account I keep disabled until I need it. Provider Retry runs with default settings.

Sometimes the pool briefly cannot serve a thread. The turn then fails with the pool's own 429, and Provider Retry queues a retry for the "next account" time that the pool quotes. I then fix the pool in one of these ways:

• enable or add an account;
• clear an errored account;
• change switchThreshold.

I expect the waiting threads to resume within about a minute. They don't. Each retry row keeps the sendAt it got when the turn failed, and fires hours later at the reset time quoted back then. The only way to resume earlier is to find every waiting thread and send its row by hand.

What happens today

Versions.

• bb 0.46.1 (npm bb-app, Linux x86_64, Node v24.22.1; desktop-v0.46.1 is the latest release), with the built-in account-pool 0.1.0 and provider-retry plugins.
• The threads are Claude Code (2.1.308 / 2.1.309).
• codex-cli 0.163.0 at the time (0.164.0 now). Codex threads use the same Provider Retry path, but none were involved in these incidents.

Unchanged on main. I checked main at 1e80ca7b44bccd06c6da75ed9ca6ba497bd380f5, which is current upstream HEAD. Since 0.46.1:

• provider-retry only gained Retry turns that fail on provider server and connection errors #5931, which adds 5xx/connection backoff in a different branch of decideRetry;
• account-pool only had UI, env-listing and typecheck-tooling changes (Make Account Pooler rows clickable across their highlighted area #5900, List provider models with plugin-contributed provider environment #5767, Typecheck with tsrs instead of TypeScript 7 tsc #5861).

Code (main @ 1e80ca7):

• Provider Retry decides once per failure.
  • It acts only on turn.failed, and for each failure it calls provider-retry.v1.availability once.
  • It then either declines, or calls threads.retry with a fixed sendAt (server.ts#L55-L110).
  • Its test pins this: "listens for one event and answers no hook" (server.test.ts#L435-L457).
• The send time is fixed at failure time: sendAt = max(retryAt, now) + 15 s + random(0–30 s) (retry-policy.ts#L53-L60).
• Declines are final. Nothing looks at these again:
  • pool unavailable is declined as pool-unavailable;
  • a retryAt beyond maximumWait is declined as beyond-maximum-wait (retry-policy.ts#L126-L143);
  • an availability RPC that throws or takes longer than 5 s becomes unavailable/source-unavailable, which is also declined.
• A short error can cause a permanent decline. accountAvailability skips accounts in error (hub.ts#L1294-L1332). If every enabled account is errored, even for a few seconds, the answer is unavailable/authentication, and the turn is declined for good.
• The pool knows when availability changes, but only tells the UI.
  • add and enable call onAccountsChanged and onAccountEnabled; enable also clears error and heldUntil (operations.ts#L77-L97, operations.ts#L171-L179).
  • Config changes publish ACCOUNT_POOL_CONFIG_CHANGED (server.ts#L109-L121).
  • onAccountsChanged is a bb.realtime.publish, and onAccountEnabled is a usage refresh (server.ts#L208-L221).
  • bb.realtime.publish is a WebSocket broadcast to connected clients (backend-contract.ts#L906-L914), so no other plugin's server sees it.
  • The 5-minute background usage refresh (hub.ts#L164-L178) updates quota observations and sends no signal at all.
• The docs promise doesn't reach waiting threads. The pool docs say an early reset or plan upgrade "takes effect on the next turn" (accounts-and-routing.md#L62-L66). A thread parked on a retry row has no next turn until its sendAt.
• Restart doesn't help. Pending rows survive a restart with their original sendAt (SKILL.md#L33-L34). Nothing re-queries availability on start.
• A plugin can't move a pending retry earlier without side effects.
  • Core allows one live retry per failed request (turn-retry.ts#L26-L41, turn-retry.ts#L197-L206).
  • queuedMessages.update changes only input (threads.ts#L436-L439).
  • Send-now, which bb provider-retry retry uses (cli.ts#L49-L76), bypasses every plugin wait, including Concurrency Limit (dispatch-attempt.ts#L224-L230).

Observed. All times are UTC. Sources: bb thread log <thr> --json, bb pool account list --json (createdAt) and the pool's plugin.log. Account names are anonymized.

Controls showing that the pool could serve and only the schedule was stale:

• thr_biw5z6pxki, sent early. It hit the same 429 in the first two episodes. Its retry row was sent ahead of its schedule 3 min after the threshold change (23:23:48) and 16 s after the account was added (10:13:03). Both turns completed normally.
• thr_cv4aev22bx kept working through the pool. On 10-23 it received a message from another thread and ran three turns through the pool between 01:04:56 and 01:33:24. Its own retry row stayed queued until 03:00:25 the whole time.

Verbatim 429 (thr_cehnf3bqqh, 2026-10-23T00:49:53.635Z):

API Error: Request rejected (429) · No Account Pooler account is currently eligible. Next account is expected to be available at 2026-10-23T02:59:59.798Z (in 7807 seconds).

The pool's plugin.log, 2.8 s earlier. Why this error happens at all is #6167:

{"ts":1792716590876,"level":"warn","message":"Account Pooler claude account <account B> OAuth refresh failed: OAuth refresh failed with HTTP 400. invalid_grant. expiresAt=1792716887319, forced=false."}

A stale row also fires after the thread has moved on.

• What happened: thr_cv4aev22bx's row fired at 03:00:25 and sent the agent-only "Please continue." as a retry of request creq_9krud6u47q, which had failed at 00:49:51. By then the thread had already finished three newer turns.
• Where the check is: the "failed turn must be the thread's latest" check runs when the retry is created (turn-retry.ts#L43-L84). I found no such check on the drain path.
• Scope: this may deserve its own bug. A re-check right before sending would cover it.

Diagnostics gap. bb provider-retry explain keeps only the last decision. For these threads it now shows the 03:00 not-retryable declines, so I rebuilt the 00:49 schedules from the thread logs.

Not reproduced on demand. The evidence is production logs plus a source read of 0.46.1 and main. I did not write a test.

What you would expect

1. The pool announces availability changes. Account Pooler sends a coalesced "availability may have changed" signal per provider when:

   • an account is added or enabled;
   • an error or hold is cleared (enable, refresh, a successful recovery);
   • a setting that affects eligibility changes (switchThreshold, routing);
   • a usage observation moves the earliest availability earlier (early reset, plan upgrade);
   • a parent pool's availability changes.

   bb.realtime only reaches UI clients, so this needs a plugin-to-plugin path. Two options: a discoverable provider-retry.v1.availabilityChanged RPC that the pool calls (the mirror image of today's provider-retry.v1.availability discovery), or a core plugin event.

2. Provider Retry re-evaluates. It re-queries availability:

   • when it gets the signal;
   • on plugin start;
   • on a bounded fallback interval while pool-routed rows are pending (for example every 5 min), which also covers parent pools and missed signals.

   It re-queries for every pending pool-routed retry row. It also re-queries for every turn it declined as pool-unavailable or beyond-maximum-wait, as long as that thread is still in error on the same request. Then, by result:

   • ready: send soon, staggered with jitter so N threads don't all hit one newly enabled account at once.
   • blocked with an earlier retryAt: move sendAt earlier.
   • blocked with a later retryAt: move sendAt later rather than spend an attempt on a send that will fail.
   • still unavailable: keep waiting, bounded by maximumWait counted from the original failure.

3. Re-check right before sending. When a pool-routed row comes due, query availability again:

   • if it is still blocked, reschedule without spending an attempt;
   • if the thread is archived or deleted, or the failed request is no longer the thread's latest, drop the row.

4. Existing invariants stay.

   • Re-evaluation never consumes an attempt; the cap stays at 5 attempts (retry-policy.ts#L95-L99).
   • There is still one live row per failed request.
   • Wakes go through message.dispatch (Concurrency Limit and other plugin waits), not Send-now.

   That probably needs a small core API to move a queued row's sendAt. Two options: accept sendAt in queuedMessages.update, or let threads.retry replace the pending row for the same request.

5. Diagnostics. bb provider-retry explain shows re-evaluations as well as the last decision, for example "rescheduled 03:00:23 → 00:50:31 after pool availability changed".

Acceptance. All of these can be tested with a fake pool and fake timers:

• Enable. Account A is blocked until T+2h and account B is disabled. A pooled turn fails with 429, so the row is set for about T+2h. After bb pool account enable B, the row dispatches within about a minute and passes message.dispatch.
• Other fixes. The same holds for bb pool account add, for clearing an errored account, and for a switchThreshold change.
• All accounts errored. The turn is declined as pool-unavailable. Enabling one account schedules the retry.
• Beyond maximumWait. The turn is declined. Adding an account schedules the retry.
• Restart. With a pending row and a pool that is now ready, the row is re-evaluated on start.
• Many threads. Ten pending threads are sent staggered, and re-evaluation leaves their attempt numbers unchanged.
• Thread moved on. A thread finishes newer turns while its row waits. The row is dropped, and no "Please continue." is sent.

Context and alternatives

• Workaround today: after any pool change, run bb provider-retry status, then bb provider-retry retry <thread-id> for each waiting thread. This has three problems:
  • it is manual and easy to forget;
  • it uses Send-now, so it bypasses Concurrency Limit;
  • automating it means an external polling script, which is what I want to avoid.
• maximumWait doesn't fix it. Setting it to "24 hours" or "No limit" only trades permanent declines for longer blind waits.
• Not a duplicate of:
  • provider-retry never retries Account Pooler 429/502 failures #4839 (closed by Fix provider retries for exhausted account pools #5283 and Retry turns that fail on provider server and connection errors #5931) asked Provider Retry to schedule pool 429s against the earliest recovery time. That works now. This request is about what happens after that one decision.
  • Fix provider retries for exhausted account pools #5283 (merged) added the one-shot availability RPC and kept "the existing durable queue, attempt cap, maximum wait, buffer, and jitter". It does no re-evaluation.
  • Retry turns that fail on provider server and connection errors #5931 (merged) adds 5xx/connection backoff, which is a different branch.
  • Account Pooler quota exhaustion leaves Codex reconnecting inside active turns and blocks the shared concurrency queue #6116 (open): Codex sleeps inside the turn, so turn.failed never fires. That fix hands control to the pool-aware scheduler, which then also needs this re-evaluation to act on early account recovery. The two are complementary.
  • Plugin SDK: three small gaps in RPC and realtime — no non-500 status for handler-side refusals, useRealtime is own-plugin only, experimental_discoverRpc has no signal #4944 (open, item 2) covers the same missing cross-plugin signal from the frontend side: useRealtime is own-plugin only. This request needs the plugin-server side of that gap.
  • Per-thread opt-out for provider-retry #4131, Account Pooler: last-resort accounts, per-account switch thresholds, and an opt-in order by earliest weekly reset #4468, Seamless bb updates: wait until idle or update now and resume agents #6127, Public request-aware handoff ownership for safe provider fallback #5552 and Account Pooler: show and spend Claude and Codex usage-limit resets per account #4157/Provider usage: show and spend Codex and Claude usage-limit resets #6153 don't cover it either. The last pair, spending usage-limit resets, would be one more source of availability changes.
• Out of scope:

  • Why the pool became unavailable is reported separately:

    • a proactive refresh failure marks an account with a still-valid token as error (Account Pooler: a rejected proactive OAuth refresh errors an account whose access token is still valid; background refresh keeps replaying the rejected token #6167);
    • switchThreshold acts as a hard exclusion (Account Pooler: when every account is past switchThreshold, the pool returns 429 with a far reset although upstream still allows requests #6169).

    This request covers recovery from any cause.

  • The 10-23 03:00 retries then failed with missing_executable: claude-code tried to spawn a removed Node path after a Node upgrade on this host. They were declined as not-retryable, and thr_cv4aev22bx's run was cut off by a host-daemon restart. Neither belongs here.

• Investigation: bb thread thr_agudxusj98. Its reports are in that thread's storage and are not public.
• Suggested priority and effort:
  • Medium-high priority. It hits any pooled setup whose pool recovers before the quoted reset.
  • No work is lost, but agents sit idle for hours (2 h 10 m to 2 h 40 m here). A manual per-thread workaround exists.
  • Effort is medium: a pool signal, a Provider Retry re-query, and a small core reschedule API.

Checks

• I searched open and closed issues for the same request.
• If an agent wrote this, the body ends with > AGENT GENERATED.

AGENT GENERATED