Skip to content

fix(bb-prover): replace a dead pooled bb verifier, by letting the instance heal itself - #354

Open
charlielye wants to merge 19 commits into
mainfrom
cl/bb-verifier-retryable
Open

charlielye wants to merge 19 commits into
mainfrom
cl/bb-verifier-retryable

Conversation

@charlielye

@charlielye charlielye commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #351 and keeps its verifier half; supersedes it. Addresses https://github.com/AztecProtocol/barretenberg-claude/issues/4427 from the node side. Fixes A-2222. Relies on aztec-packages#25548, which main's bb.js pin carries.

The RPC proof verifier borrows long-lived bb processes from the BBJsFactory pool. When one died, the pool put the dead handle back and handed it out for the life of the node, and verifyProof turned the resulting error into { valid: false }, so valid transactions were rejected as having invalid proofs until restart. Because a dead instance failed instantly it also took far more than its share of the queue.

The instance heals itself, so the pool does not have to

A pooled instance now spawns with bb.js's respawn. When its bb process dies it replaces it, and the next borrower gets a working one. That means the pool needs no liveness check, no eviction and no maintenance loop: returning an instance unconditionally is correct again, exactly as it already is for the AVM simulator pool, which has had this shape since aztec-packages#25073.

respawn is an explicit BBJsFactoryOptions field, which BBCircuitVerifier sets for its pool. A fresh-per-call instance does not get it, having nothing to heal, and neither does anything holding a Chonk accumulation or a batch-verifier session, where a replacement process would silently lose state.

This is why the pool half of #351 is gone here. BBJsApi.isAlive(), the RunningPromise maintenance run and the eviction path are all replaced by one option.

The pool is a queue of slots

The pool used to start every bb at once in a separate startup phase, and each way that phase could fail needed its own handling: a cached startup failure that wedged the factory, a shutdown that hung while it ran, and a race against destruction that leaked a handler on every borrow.

It is now a queue of poolSize slots, created up front. A borrower takes a slot and starts its bb if it has none. A slot whose bb fails to start goes back empty for the next borrower to try again, so the instances that did start are kept.

A borrower waits in two places only. Waiting for a slot ends at once when the factory is destroyed, because destroying it cancels the queue. Starting a slot's bb ends when the start does, which bb.js bounds at 60 s. So shutdown can wait up to that long for a verification that is starting a bb when the node stops; this is deliberate, and the shutdown tests pin it.

The retry keys off the failure, not a liveness query

bb.js marks an environmental failure with retry: true, feature-detected rather than imported, so the same check holds across the bb.js and ipc-runtime package boundaries. It is one type guard, isRetryableError in @aztec-labs/foundation/error, shared with the AVM simulator pool:

export function isRetryableError(err: unknown): boolean {
  return err instanceof Error && 'retry' in err && err.retry === true;
}

Asking an instance whether it is alive can only ever be a guess, because the process can die between the answer and the next call. More importantly it cannot tell a dead helper from a bad proof, which is the misattribution this issue is really about: the node reported an infrastructure fault as an invalid user transaction and booked it into IVC_VERIFIER_FAILURE_COUNT.

Kept from #351

Unchanged, and the better part of it:

  • verifyProof retries a verification whose bb died or could not start, once, and then throws ProofVerifierUnavailableError rather than returning { valid: false }. RPC admission fails with that instead of rejecting the transaction, and QueuedIVCVerifier does not count it as a verification failure.
  • Any other failure still returns valid: false.
  • QueuedIVCVerifier.stop() stops the verifier before draining its queue, so a verification waiting for a slot fails instead of blocking shutdown. One already starting a bb holds shutdown until that start ends, as above.

Testing

Run in CI. The double models a healing instance: a death fails the call in flight, retryably, and leaves the instance usable, because a replacement process serves the next call.

  • bb_verifier.test.ts covers valid, invalid, a bb error while healthy, a retry after a death that succeeds on the replacement without the pool growing, unavailable after two deaths, retrying a failed spawn rather than rejecting the proof, unavailable when a per-call instance cannot start, starting no more bb processes than the pool holds, and stop() while verifications wait on a bb that is still starting.
  • bb_js_backend.test.ts covers the pool itself: reuse without exceeding the pool size, a slot whose bb failed to start being started again by the next borrower, destroy() releasing a borrower waiting for a slot at once and one starting a bb when its start ends, and an instance borrowed across destroy() being destroyed on release rather than pooled.

The retry contract these tests are written against is verified end to end in aztec-packages#25548, against a real bb.

Size

Against main: +472/−94 across 10 files, most of it tests and the test double. The pool file is +66/−75.

@greptile-apps

greptile-apps Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

[High risk] Adds automatic respawn to pooled bb verifier instances.

The PR appears behaviorally sound on the reviewed paths, but the test gates must satisfy the repository's explicit TypeScript requirement before merging.

Fix All in Claude CodeFindings

  1. P2 Escaped resolver in test ▶

Summary

The PR makes pooled bb verifier instances respawn after process death, retries environmental verification failures, and changes pool startup and shutdown handling.

  • Adds slot-based lazy pool initialization and lifecycle tests.
  • Shares retryable-error detection across the verifier, prover, and AVM pool.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Verification request] --> B[Borrow pool slot]
  B --> C[Start bb if slot is empty]
  C --> D[Verify proof]
  D -->|bb process failure| E[Retry once]
  E --> B
  D -->|proof verdict| F[Return validity]
  E -->|retry exhausted| G[Verifier unavailable]
  F --> H[Release slot]
  G --> H
Loading

Reviews (3) · Last reviewed commit: "refactor(bb-prover): make the bb.js pool..."

Comment thread yarn-project/bb-prover/src/verifier/queued_chonk_verifier.ts
Comment thread yarn-project/bb-prover/src/verifier/bb_verifier.test.ts
Comment thread yarn-project/bb-prover/src/verifier/bb_verifier.ts Outdated

@fcarreiro fcarreiro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Written by Claude on behalf of Facundo)

I read and reasoned through this; I didn't run it (CI fails at compile until bb.js is bumped, as the warning says). The verifier half and the stop() ordering look right to me, and letting the instance heal itself is simpler than #351's pool. The main issue is that going back to main's pool brings back a permanent wedge on a failed startup spawn (inline on bb_js_backend.ts:389). That wedge is also why Greptile's two test comments are right: "waits for a bb instance to start…" gets ProofVerifierUnavailableError, and "stops a queued verifier…" hangs on initPromise.

Other notes:

  • On Greptile's P2 about the as cast in isRetryableFailure: it is the same check as isRetryable in simulator/src/public/avm_simulator_pool.ts:56. A shared type guard in foundation would satisfy the style rule and avoid a second copy.
  • 11 of the 12 commits are #351's, and the last one reverts most of them, so squash on merge.

Comment thread yarn-project/bb-prover/src/bb/bb_js_backend.ts Outdated
Comment thread yarn-project/bb-prover/src/verifier/bb_verifier.ts Outdated
Comment thread yarn-project/bb-prover/src/verifier/bb_verifier.ts Outdated
Comment thread yarn-project/bb-prover/src/bb/bb_js_backend.ts Outdated
charlielye added a commit to AztecProtocol/aztec-packages that referenced this pull request Oct 1, 2026
…request (#25548)

Addresses
AztecProtocol/barretenberg-claude#4427 from
the bb.js side. The aztec-node side is
[aztec-node#354](aztec-labs-eng/aztec-node#354).

A pooled bb verifier whose process dies is handed back to the pool and
handed out again on every later borrow. The pool cannot tell, because
the socket backend marks itself permanently unusable and every later
call fails with a bare `Socket not connected`. The node therefore
reports a dead helper process as an invalid transaction proof,
persistently, and books it into the metric that means users are
submitting bad proofs.

This gives an owner the two things it needs, in the shape the AVM
simulator pool already relies on.

### A failed call says whether retrying can help

An environmental failure now carries `retry: true`: the bb process died,
its connection broke, or it could not be started. Only a binary that
cannot be executed stays non-retryable, because retrying cannot fix it.

The bare property is the whole contract, feature-detected rather than
imported:

```ts
if (err instanceof Error && (err as Error & { retry?: unknown }).retry === true) { ... }
```

That is deliberately the same convention `ipc-runtime`'s transports use,
so a caller written against one works unchanged against the other. It is
also what lets a verifier distinguish a bad proof from a dead helper,
which is the misattribution half of the issue.

### The socket backend can replace a dead process, on request

`respawn` is a new `BackendOptions` flag, off by default. With it on,
the bb process and its connection are one unit swapped as a whole: the
next call starts a replacement, while calls already in flight still fail
retryably. Concurrent callers that find the connection down share one
replacement, so a single death costs a single process, and a replacement
that arrives after `destroy()` is not left running.

It is opt-in because a replacement has none of the state a command
sequence establishes: no SRS loaded over the connection, no Chonk
accumulation, no batch-verifier session with its registered keys. An
owner that runs any such sequence must leave it off, so a death fails
loudly instead of the next call quietly running against a process that
has forgotten everything. The verifier pool qualifies because every
verification carries its proof and key in the call.

With this, a pool needs no liveness check and no maintenance loop:
returning an instance unconditionally becomes correct, exactly as it
already is for the AVM pool.

### Also

`BarretenbergSync.initSingleton()` cached a failed initialization for
the life of the process, so one bad spawn was permanent. It now clears
the failure, as the asynchronous singleton already did.

### Testing

`native_socket.test.ts`, against the fake bb the existing tests use: a
killed bb fails the call retryably and keeps failing without the option;
with it on the next call is served by a replacement process, a different
pid; four concurrent calls that find the connection down start exactly
one replacement; `destroy()` during a replacement leaves no process
running; a replacement that cannot start fails retryably too; a
connection that breaks while the process keeps running leaves no bb
behind.

`singleton.test.ts` covers the initialization fix and fails without it.

Checked against a real bb as well: with the option off a killed bb gives
a retryable error and keeps doing so, and with it on the next call
transparently returns the same hash from a fresh process.

### What this does not cover

`BarretenbergSync` runs on the shared-memory backend, which has no
respawn and no way to report a death mid-call: the NAPI receive loop
retries without a deadline, and because the call blocks the event loop
the process exit is never even observed, so the caller wedges rather
than fails. That path is untouched here and is deliberately left alone:
the synchronous bb runs one thread doing hashes and signatures, so it is
the least likely process on the machine to be killed, and the fix would
mean threading a liveness check into the C++ client for a case that may
never happen. #25546 covers the idle-death half of it by replacing a
dead singleton, so the two PRs cover different backends rather than one
superseding the other.

### Note on direction

bb.js's hand-written backends are replaced by `ipc-runtime`'s in the
codegen migration (#25362). Nothing above is lost in that move:
`ipc-runtime`'s spawned backend already carries the same `retry`
contract and the same opt-in respawn, so the callers written against
this keep working and the implementation here is deleted. That is why
this is expressed as the retry contract rather than as a liveness query,
which would have to become part of the generated client's interface.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@charlielye
charlielye force-pushed the cl/bb-verifier-retryable branch from 8bfa68a to d58a9da Compare October 1, 2026 12:58
@charlielye
charlielye requested a review from fcarreiro October 1, 2026 17:57
@spalladino
spalladino added this pull request to stack #384 October 2, 2026 20:19
}
// Racing destruction as well, so a borrow does not outlive the factory when the pool is still
// starting; a bb that never comes up would otherwise block shutdown indefinitely.
await Promise.race([this.initPromise, this.destroyedSignal.promise]);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Written by Claude on behalf of Facundo)

This leaks memory on every borrow. destroyedSignal stays pending until destroy(), and each Promise.race attaches a handler to it that is only released when it settles. The race runs on every getInstance(), long after the pool has started, so every verifyProof leaves one behind for the life of the node.

Measured on Node with forced GC: about 312 bytes retained per race on a pending promise, and 0 without it, over 1M iterations. A node that verifies 1M txs between restarts holds about 300 MB of dead handlers.

Suggest racing only while the pool is still starting, e.g. if (!this.pool) { await Promise.race([...]) }. Once the pool exists, destroy() already releases waiting borrowers through pool.cancel().

fcarreiro and others added 16 commits October 5, 2026 08:24
A pooled bb instance whose process died was returned to the pool and handed
out again, so every later borrow of it failed. The pool now checks
BBJsApi.isAlive(): a dead instance is destroyed when it is returned or found
idle, and a replacement is spawned when a borrower finds no idle instance
while the pool is below its size. A failed spawn fails that borrow and is
retried by the next one, and a failed pool initialization is retried too.

Requires a bb.js release with Barretenberg.isAlive().
…nvalid proof

BBCircuitVerifier.verifyProof turned every error into { valid: false }, so a
bb process that died was reported as an invalid transaction proof and counted
as a verification failure. A call that fails because its bb died is now
retried once on another pooled instance; if that bb dies too, or no instance
can be started, verifyProof throws ProofVerifierUnavailableError. RPC
admission then fails with that error instead of rejecting the tx as invalid,
and QueuedIVCVerifier does not record it as a failed verification. A call
that fails while bb is alive still means bb rejected the proof.
…p a partly started pool

A borrower waiting on a full pool was stranded when a borrowed instance died,
because the eviction freed room without spawning anything. Returning a dead
instance now spawns its replacement, which goes straight to a waiting
borrower. A borrow whose replacement spawn fails waits for a borrowed
instance instead of failing, and only throws when the pool holds none. Pool
initialization keeps the instances that started and spawns the rest on
demand, rather than destroying them all when one fails.
…e checked

ClientProtocolCircuitVerifier.verifyProof documents that it rejects, rather
than reporting the proof invalid, when the proof could not be checked.
BBCircuitVerifier's docs no longer claim that every failure while bb is alive
is a rejection of the proof, and its failure log is structured.
…not strand them

A borrower waiting on an empty pool counted on an in-flight spawn or a
borrowed instance to come back, and was never woken when the spawn failed or
the instance died without a replacement. Waiting borrowers now re-check every
second: they spawn an instance themselves when the pool is below its size,
and throw when that fails while no instance exists or is being spawned. A
dead instance's replacement is spawned in the background, so returning it
does not hold up the verification that found it dead.
…royed

A borrower re-checking after destroy(), or an eviction just before it, spawned a bb only to destroy it. The pool test that waits for a borrowed instance also plans enough failed spawns for re-checks that land before that instance returns.
…tenance run

The pool no longer spawns from the borrow path. Every second a maintenance run destroys the pooled instances whose bb
died and spawns instances until the pool is back at poolSize, retrying failed spawns on the next run. A borrower skips
dead instances and waits for a live one however long that takes, as it does while every instance is busy.

destroy() does not wait for a spawn in flight; the spawn destroys its instance when it completes. QueuedIVCVerifier stops
its verifier before draining its queue, so a verification waiting for a bb instance fails instead of blocking shutdown.
…n the factory is destroyed

destroy() retires every dead member as well as the idle ones, so a dead instance a borrow dropped between maintenance runs
is destroyed. Dispose returns an instance to the pool as before; the borrow path skips it if it is dead.
QueuedIVCVerifier.stop() drains its queue even if stopping the verifier fails.
…r instead of a member list

The idle queue is the only record of idle instances. Pool maintenance sweeps it, destroying dead instances and putting
the live ones back, and spawns until idle, borrowed and spawning instances reach poolSize. A borrow destroys a dead
instance it takes. Each instance is idle, borrowed or destroyed, so destroy() tears down the idle queue as on main.
The count covers idle, borrowed and spawning instances: a spawn adds one, and a failed spawn or a destroyed dead instance
removes one. Pool maintenance spawns up to poolSize from it, and the dispose code is main's again.
… the pool

Keeps the verifier half of this branch and replaces the pool half with one option.

A pooled instance now spawns with bb.js's `respawn`, so an instance whose bb process
dies replaces it and the next borrower gets a working one. The pool needs no liveness
check, no eviction and no maintenance loop: returning an instance unconditionally is
correct again, exactly as it already is for the AVM simulator pool.

The verification retry keys off the failure rather than a liveness query. bb.js marks
an environmental failure with `retry: true`, feature-detected rather than imported, so
the same check works across the bb.js and ipc-runtime package boundaries. Asking an
instance whether it is still alive can only ever be a guess, since the process can die
between the answer and the next call, and it would not distinguish a dead helper from
a bad proof, which is what makes the node report one as the other.

The double models the same thing: a death fails the call in flight, retryably, and
leaves the instance usable, because a replacement process serves the next call.

Needs a bb.js release carrying aztec-packages#25548, which adds `respawn` and the
`retry` property. Until the pin moves this does not build, as the branch it adjusts
did not. The bb.js side is tested there, including against a real bb.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
`ProvingError` was the only retryable failure the prover recognised, so a bb process
that died mid-call surfaced as permanent and the job was not retried. Both the prover
and the verifier now ask the same question of the error, and the classifier lives next
to the backend that raises it rather than once per caller.

`ProvingError.retry` still answers true, since the check is on the property rather
than the class.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Reverting the pool to main's brought back main's permanent wedge, which #351 had also
been fixing: initPromise caches its rejection, so one failed spawn made every later
borrow fail for the life of the process. It is cleared on failure now, as the bb.js
singleton's is, so the next borrow starts the pool again.

A borrow also raced destruction, so a pool whose bb never comes up no longer holds up
shutdown: the wait ends when the factory is destroyed and the borrow says so.

The verifier spends its attempt budget on both halves. A bb that could not be started
is worth another go, on the same terms a bb that died under the call gets; anything
else, such as the factory being destroyed, will not improve by asking again. Either
way no proof was checked, so neither is ever reported as the proof's fault.

The double now fails a spawn retryably, as BBJsInstance.create does, rather than
showing a test a permanent failure where production has a transient one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
… test

A failed spawn is now retried, so failing it once let the second attempt
succeed and the proof verify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…into respawn

The bb-prover and AVM pool copies of the retry check become one type guard in
foundation, without a cast. Respawn is now an explicit BBJsFactory option the
verifier sets, rather than implied by pooling, so a future pooled owner of
stateful sessions does not get it by accident. The retry wording says attempts,
since with respawn a retry usually lands on the same instance's new process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
charlielye and others added 2 commits October 5, 2026 08:24
… is starting

Every borrow raced the init promise against destroyedSignal, which stays
pending until destroy(). Each race leaves a handler on it, so every
verification leaked one for the life of the node: about 312 bytes, or ~300 MB
per million verifications. Once the pool exists, destroy() already releases
waiting borrowers through pool.cancel().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…their bb on demand

The pool had a separate startup phase that spawned every instance at once, and
each failure mode it introduced needed its own patch: a cached startup failure
that wedged the factory, a shutdown that hung while it ran, and a race against
destroy that leaked a handler per borrow.

Without that phase there is nothing to patch. The pool is a queue of slots,
created up front; a borrower takes one and starts its bb if it has none. A
borrower now waits only for a slot, which destroy() releases by cancelling the
queue, or for its own bb to start, which bb.js bounds with its startup deadline.
A bb that fails to start costs only its slot, which goes back empty, so the
instances that did start are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
@charlielye
charlielye force-pushed the cl/bb-verifier-retryable branch from ce51b30 to faad42f Compare October 5, 2026 08:25
Comment on lines +41 to +42
let finishStart!: () => void;
factory.planNextInstance([], new Promise<void>(resolve => (finishStart = resolve)));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Escaped resolver in test This test stores a promise resolver in a mutable let outside the executor. The repository’s TypeScript guide requires promiseWithResolvers for deferred gates instead. The shutdown test in bb_verifier.test.ts uses the same pattern. Please satisfy this repository requirement before merging.

Context Used: yarn-project/CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code

…seWithResolvers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4

@fcarreiro fcarreiro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Written by Claude on behalf of Facundo)

Approving. The slot pool is simpler than either earlier pool, the Promise.race leak is gone, and the retry and unavailable paths look right to me. I reasoned through the code; I didn't run it. The one inline comment is a suggestion, not a blocker.

The PR description is out of date against the current code:

  • It doesn't describe the slot queue that faad42f1b4 introduced, where each slot starts its bb on first use and a slot whose bb failed to start goes back empty for the next borrower.
  • "Not run, for the reason in the warning above": the warning is gone, the bb.js pin carries aztec-packages#25548, and CI runs these tests.
  • "Kept from #351" says a verification waiting for an instance "fails instead of blocking shutdown". That holds for one waiting for a slot, but one starting a bb holds stop() until its start ends (inline comment).
  • "Testing" names "waiting for an instance to start", which is now "retries a failed spawn rather than rejecting the proof". It also leaves out the new pool tests in bb_js_backend.test.ts and the "starts no more bb processes than the pool holds" case.
  • "Size" says the pool file is +19 and its test file is back to its original state. Against main they are now +66/−75 and +55.

throw new Error('BBJsFactory has been destroyed');
let instance: BBJsApi;
try {
instance = slot.instance ??= await this.createInstance();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Written by Claude on behalf of Facundo)

Ideally shutdown wouldn't wait here. Suppose a verification is starting this slot's bb when the node stops. QueuedIVCVerifier.stop() destroys the factory and then waits in queue.end() for that verification. The verification stays on this await until bb.js gives up the start, which takes up to STARTUP_TIMEOUT_MS, 60 s. The class comment documents this and the shutdown test pins it, so it's deliberate, but a node stuck on a slow bb start then stops slowly too.

To avoid the wait without bringing back the leak, race only the start against destruction, and detach the listener when the start settles. For example, once('destroyed') on an EventEmitter before the start, and off in a finally. Then nothing stays attached after a borrow, and a borrow that finds its slot started already doesn't race at all. On destroy the borrow releases the slot and throws. The abandoned start still has to destroy whatever it produces, e.g. void start.then(instance => instance.destroy(), () => {}).

Not blocking.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants