fix(bb-prover): replace a dead pooled bb verifier, by letting the instance heal itself - #354
charlielye wants to merge 19 commits into
Conversation
|
fcarreiro
left a comment
There was a problem hiding this comment.
(Written by Claude on behalf of Facundo)
I read and reasoned through this; I didn't run it (CI fails at compile until bb.js is bumped, as the warning says). The verifier half and the stop() ordering look right to me, and letting the instance heal itself is simpler than #351's pool. The main issue is that going back to main's pool brings back a permanent wedge on a failed startup spawn (inline on bb_js_backend.ts:389). That wedge is also why Greptile's two test comments are right: "waits for a bb instance to start…" gets ProofVerifierUnavailableError, and "stops a queued verifier…" hangs on initPromise.
Other notes:
- On Greptile's P2 about the
ascast inisRetryableFailure: it is the same check asisRetryableinsimulator/src/public/avm_simulator_pool.ts:56. A shared type guard in foundation would satisfy the style rule and avoid a second copy. - 11 of the 12 commits are #351's, and the last one reverts most of them, so squash on merge.
…request (#25548) Addresses AztecProtocol/barretenberg-claude#4427 from the bb.js side. The aztec-node side is [aztec-node#354](aztec-labs-eng/aztec-node#354). A pooled bb verifier whose process dies is handed back to the pool and handed out again on every later borrow. The pool cannot tell, because the socket backend marks itself permanently unusable and every later call fails with a bare `Socket not connected`. The node therefore reports a dead helper process as an invalid transaction proof, persistently, and books it into the metric that means users are submitting bad proofs. This gives an owner the two things it needs, in the shape the AVM simulator pool already relies on. ### A failed call says whether retrying can help An environmental failure now carries `retry: true`: the bb process died, its connection broke, or it could not be started. Only a binary that cannot be executed stays non-retryable, because retrying cannot fix it. The bare property is the whole contract, feature-detected rather than imported: ```ts if (err instanceof Error && (err as Error & { retry?: unknown }).retry === true) { ... } ``` That is deliberately the same convention `ipc-runtime`'s transports use, so a caller written against one works unchanged against the other. It is also what lets a verifier distinguish a bad proof from a dead helper, which is the misattribution half of the issue. ### The socket backend can replace a dead process, on request `respawn` is a new `BackendOptions` flag, off by default. With it on, the bb process and its connection are one unit swapped as a whole: the next call starts a replacement, while calls already in flight still fail retryably. Concurrent callers that find the connection down share one replacement, so a single death costs a single process, and a replacement that arrives after `destroy()` is not left running. It is opt-in because a replacement has none of the state a command sequence establishes: no SRS loaded over the connection, no Chonk accumulation, no batch-verifier session with its registered keys. An owner that runs any such sequence must leave it off, so a death fails loudly instead of the next call quietly running against a process that has forgotten everything. The verifier pool qualifies because every verification carries its proof and key in the call. With this, a pool needs no liveness check and no maintenance loop: returning an instance unconditionally becomes correct, exactly as it already is for the AVM pool. ### Also `BarretenbergSync.initSingleton()` cached a failed initialization for the life of the process, so one bad spawn was permanent. It now clears the failure, as the asynchronous singleton already did. ### Testing `native_socket.test.ts`, against the fake bb the existing tests use: a killed bb fails the call retryably and keeps failing without the option; with it on the next call is served by a replacement process, a different pid; four concurrent calls that find the connection down start exactly one replacement; `destroy()` during a replacement leaves no process running; a replacement that cannot start fails retryably too; a connection that breaks while the process keeps running leaves no bb behind. `singleton.test.ts` covers the initialization fix and fails without it. Checked against a real bb as well: with the option off a killed bb gives a retryable error and keeps doing so, and with it on the next call transparently returns the same hash from a fresh process. ### What this does not cover `BarretenbergSync` runs on the shared-memory backend, which has no respawn and no way to report a death mid-call: the NAPI receive loop retries without a deadline, and because the call blocks the event loop the process exit is never even observed, so the caller wedges rather than fails. That path is untouched here and is deliberately left alone: the synchronous bb runs one thread doing hashes and signatures, so it is the least likely process on the machine to be killed, and the fix would mean threading a liveness check into the C++ client for a case that may never happen. #25546 covers the idle-death half of it by replacing a dead singleton, so the two PRs cover different backends rather than one superseding the other. ### Note on direction bb.js's hand-written backends are replaced by `ipc-runtime`'s in the codegen migration (#25362). Nothing above is lost in that move: `ipc-runtime`'s spawned backend already carries the same `retry` contract and the same opt-in respawn, so the callers written against this keep working and the implementation here is deleted. That is why this is expressed as the retry contract rather than as a liveness query, which would have to become part of the generated client's interface. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
8bfa68a to
d58a9da
Compare
| } | ||
| // Racing destruction as well, so a borrow does not outlive the factory when the pool is still | ||
| // starting; a bb that never comes up would otherwise block shutdown indefinitely. | ||
| await Promise.race([this.initPromise, this.destroyedSignal.promise]); |
There was a problem hiding this comment.
(Written by Claude on behalf of Facundo)
This leaks memory on every borrow. destroyedSignal stays pending until destroy(), and each Promise.race attaches a handler to it that is only released when it settles. The race runs on every getInstance(), long after the pool has started, so every verifyProof leaves one behind for the life of the node.
Measured on Node with forced GC: about 312 bytes retained per race on a pending promise, and 0 without it, over 1M iterations. A node that verifies 1M txs between restarts holds about 300 MB of dead handlers.
Suggest racing only while the pool is still starting, e.g. if (!this.pool) { await Promise.race([...]) }. Once the pool exists, destroy() already releases waiting borrowers through pool.cancel().
A pooled bb instance whose process died was returned to the pool and handed out again, so every later borrow of it failed. The pool now checks BBJsApi.isAlive(): a dead instance is destroyed when it is returned or found idle, and a replacement is spawned when a borrower finds no idle instance while the pool is below its size. A failed spawn fails that borrow and is retried by the next one, and a failed pool initialization is retried too. Requires a bb.js release with Barretenberg.isAlive().
…nvalid proof
BBCircuitVerifier.verifyProof turned every error into { valid: false }, so a
bb process that died was reported as an invalid transaction proof and counted
as a verification failure. A call that fails because its bb died is now
retried once on another pooled instance; if that bb dies too, or no instance
can be started, verifyProof throws ProofVerifierUnavailableError. RPC
admission then fails with that error instead of rejecting the tx as invalid,
and QueuedIVCVerifier does not record it as a failed verification. A call
that fails while bb is alive still means bb rejected the proof.
…p a partly started pool A borrower waiting on a full pool was stranded when a borrowed instance died, because the eviction freed room without spawning anything. Returning a dead instance now spawns its replacement, which goes straight to a waiting borrower. A borrow whose replacement spawn fails waits for a borrowed instance instead of failing, and only throws when the pool holds none. Pool initialization keeps the instances that started and spawns the rest on demand, rather than destroying them all when one fails.
…e checked ClientProtocolCircuitVerifier.verifyProof documents that it rejects, rather than reporting the proof invalid, when the proof could not be checked. BBCircuitVerifier's docs no longer claim that every failure while bb is alive is a rejection of the proof, and its failure log is structured.
…not strand them A borrower waiting on an empty pool counted on an in-flight spawn or a borrowed instance to come back, and was never woken when the spawn failed or the instance died without a replacement. Waiting borrowers now re-check every second: they spawn an instance themselves when the pool is below its size, and throw when that fails while no instance exists or is being spawned. A dead instance's replacement is spawned in the background, so returning it does not hold up the verification that found it dead.
…royed A borrower re-checking after destroy(), or an eviction just before it, spawned a bb only to destroy it. The pool test that waits for a borrowed instance also plans enough failed spawns for re-checks that land before that instance returns.
…tenance run The pool no longer spawns from the borrow path. Every second a maintenance run destroys the pooled instances whose bb died and spawns instances until the pool is back at poolSize, retrying failed spawns on the next run. A borrower skips dead instances and waits for a live one however long that takes, as it does while every instance is busy. destroy() does not wait for a spawn in flight; the spawn destroys its instance when it completes. QueuedIVCVerifier stops its verifier before draining its queue, so a verification waiting for a bb instance fails instead of blocking shutdown.
…n the factory is destroyed destroy() retires every dead member as well as the idle ones, so a dead instance a borrow dropped between maintenance runs is destroyed. Dispose returns an instance to the pool as before; the borrow path skips it if it is dead. QueuedIVCVerifier.stop() drains its queue even if stopping the verifier fails.
…r instead of a member list The idle queue is the only record of idle instances. Pool maintenance sweeps it, destroying dead instances and putting the live ones back, and spawns until idle, borrowed and spawning instances reach poolSize. A borrow destroys a dead instance it takes. Each instance is idle, borrowed or destroyed, so destroy() tears down the idle queue as on main.
The count covers idle, borrowed and spawning instances: a spawn adds one, and a failed spawn or a destroyed dead instance removes one. Pool maintenance spawns up to poolSize from it, and the dispose code is main's again.
… the pool Keeps the verifier half of this branch and replaces the pool half with one option. A pooled instance now spawns with bb.js's `respawn`, so an instance whose bb process dies replaces it and the next borrower gets a working one. The pool needs no liveness check, no eviction and no maintenance loop: returning an instance unconditionally is correct again, exactly as it already is for the AVM simulator pool. The verification retry keys off the failure rather than a liveness query. bb.js marks an environmental failure with `retry: true`, feature-detected rather than imported, so the same check works across the bb.js and ipc-runtime package boundaries. Asking an instance whether it is still alive can only ever be a guess, since the process can die between the answer and the next call, and it would not distinguish a dead helper from a bad proof, which is what makes the node report one as the other. The double models the same thing: a death fails the call in flight, retryably, and leaves the instance usable, because a replacement process serves the next call. Needs a bb.js release carrying aztec-packages#25548, which adds `respawn` and the `retry` property. Until the pin moves this does not build, as the branch it adjusts did not. The bb.js side is tested there, including against a real bb. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
`ProvingError` was the only retryable failure the prover recognised, so a bb process that died mid-call surfaced as permanent and the job was not retried. Both the prover and the verifier now ask the same question of the error, and the classifier lives next to the backend that raises it rather than once per caller. `ProvingError.retry` still answers true, since the check is on the property rather than the class. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
Reverting the pool to main's brought back main's permanent wedge, which #351 had also been fixing: initPromise caches its rejection, so one failed spawn made every later borrow fail for the life of the process. It is cleared on failure now, as the bb.js singleton's is, so the next borrow starts the pool again. A borrow also raced destruction, so a pool whose bb never comes up no longer holds up shutdown: the wait ends when the factory is destroyed and the borrow says so. The verifier spends its attempt budget on both halves. A bb that could not be started is worth another go, on the same terms a bb that died under the call gets; anything else, such as the factory being destroyed, will not improve by asking again. Either way no proof was checked, so neither is ever reported as the proof's fault. The double now fails a spawn retryably, as BBJsInstance.create does, rather than showing a test a permanent failure where production has a transient one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
… test A failed spawn is now retried, so failing it once let the second attempt succeed and the proof verify. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…into respawn The bb-prover and AVM pool copies of the retry check become one type guard in foundation, without a cast. Respawn is now an explicit BBJsFactory option the verifier sets, rather than implied by pooling, so a future pooled owner of stateful sessions does not get it by accident. The retry wording says attempts, since with respawn a retry usually lands on the same instance's new process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
… is starting Every borrow raced the init promise against destroyedSignal, which stays pending until destroy(). Each race leaves a handler on it, so every verification leaked one for the life of the node: about 312 bytes, or ~300 MB per million verifications. Once the pool exists, destroy() already releases waiting borrowers through pool.cancel(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
…their bb on demand The pool had a separate startup phase that spawned every instance at once, and each failure mode it introduced needed its own patch: a cached startup failure that wedged the factory, a shutdown that hung while it ran, and a race against destroy that leaked a handler per borrow. Without that phase there is nothing to patch. The pool is a queue of slots, created up front; a borrower takes one and starts its bb if it has none. A borrower now waits only for a slot, which destroy() releases by cancelling the queue, or for its own bb to start, which bb.js bounds with its startup deadline. A bb that fails to start costs only its slot, which goes back empty, so the instances that did start are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
ce51b30 to
faad42f
Compare
| let finishStart!: () => void; | ||
| factory.planNextInstance([], new Promise<void>(resolve => (finishStart = resolve))); |
There was a problem hiding this comment.
Escaped resolver in test This test stores a promise resolver in a mutable
let outside the executor. The repository’s TypeScript guide requires promiseWithResolvers for deferred gates instead. The shutdown test in bb_verifier.test.ts uses the same pattern. Please satisfy this repository requirement before merging.
Context Used: yarn-project/CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…seWithResolvers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NuUzj3qpkpJor6GMpWB4T4
fcarreiro
left a comment
There was a problem hiding this comment.
(Written by Claude on behalf of Facundo)
Approving. The slot pool is simpler than either earlier pool, the Promise.race leak is gone, and the retry and unavailable paths look right to me. I reasoned through the code; I didn't run it. The one inline comment is a suggestion, not a blocker.
The PR description is out of date against the current code:
- It doesn't describe the slot queue that
faad42f1b4introduced, where each slot starts its bb on first use and a slot whose bb failed to start goes back empty for the next borrower. - "Not run, for the reason in the warning above": the warning is gone, the bb.js pin carries aztec-packages#25548, and CI runs these tests.
- "Kept from #351" says a verification waiting for an instance "fails instead of blocking shutdown". That holds for one waiting for a slot, but one starting a bb holds
stop()until its start ends (inline comment). - "Testing" names "waiting for an instance to start", which is now "retries a failed spawn rather than rejecting the proof". It also leaves out the new pool tests in
bb_js_backend.test.tsand the "starts no more bb processes than the pool holds" case. - "Size" says the pool file is +19 and its test file is back to its original state. Against main they are now +66/−75 and +55.
| throw new Error('BBJsFactory has been destroyed'); | ||
| let instance: BBJsApi; | ||
| try { | ||
| instance = slot.instance ??= await this.createInstance(); |
There was a problem hiding this comment.
(Written by Claude on behalf of Facundo)
Ideally shutdown wouldn't wait here. Suppose a verification is starting this slot's bb when the node stops. QueuedIVCVerifier.stop() destroys the factory and then waits in queue.end() for that verification. The verification stays on this await until bb.js gives up the start, which takes up to STARTUP_TIMEOUT_MS, 60 s. The class comment documents this and the shutdown test pins it, so it's deliberate, but a node stuck on a slow bb start then stops slowly too.
To avoid the wait without bringing back the leak, race only the start against destruction, and detach the listener when the start settles. For example, once('destroyed') on an EventEmitter before the start, and off in a finally. Then nothing stays attached after a borrow, and a borrow that finds its slot started already doesn't race at all. On destroy the borrow releases the slot and throws. The abandoned start still has to destroy whatever it produces, e.g. void start.then(instance => instance.destroy(), () => {}).
Not blocking.
Builds on #351 and keeps its verifier half; supersedes it. Addresses https://github.com/AztecProtocol/barretenberg-claude/issues/4427 from the node side. Fixes A-2222. Relies on aztec-packages#25548, which main's bb.js pin carries.
The RPC proof verifier borrows long-lived
bbprocesses from theBBJsFactorypool. When one died, the pool put the dead handle back and handed it out for the life of the node, andverifyProofturned the resulting error into{ valid: false }, so valid transactions were rejected as having invalid proofs until restart. Because a dead instance failed instantly it also took far more than its share of the queue.The instance heals itself, so the pool does not have to
A pooled instance now spawns with bb.js's
respawn. When itsbbprocess dies it replaces it, and the next borrower gets a working one. That means the pool needs no liveness check, no eviction and no maintenance loop: returning an instance unconditionally is correct again, exactly as it already is for the AVM simulator pool, which has had this shape since aztec-packages#25073.respawnis an explicitBBJsFactoryOptionsfield, whichBBCircuitVerifiersets for its pool. A fresh-per-call instance does not get it, having nothing to heal, and neither does anything holding a Chonk accumulation or a batch-verifier session, where a replacement process would silently lose state.This is why the pool half of #351 is gone here.
BBJsApi.isAlive(), theRunningPromisemaintenance run and the eviction path are all replaced by one option.The pool is a queue of slots
The pool used to start every
bbat once in a separate startup phase, and each way that phase could fail needed its own handling: a cached startup failure that wedged the factory, a shutdown that hung while it ran, and a race against destruction that leaked a handler on every borrow.It is now a queue of
poolSizeslots, created up front. A borrower takes a slot and starts itsbbif it has none. A slot whosebbfails to start goes back empty for the next borrower to try again, so the instances that did start are kept.A borrower waits in two places only. Waiting for a slot ends at once when the factory is destroyed, because destroying it cancels the queue. Starting a slot's
bbends when the start does, which bb.js bounds at 60 s. So shutdown can wait up to that long for a verification that is starting abbwhen the node stops; this is deliberate, and the shutdown tests pin it.The retry keys off the failure, not a liveness query
bb.js marks an environmental failure with
retry: true, feature-detected rather than imported, so the same check holds across the bb.js and ipc-runtime package boundaries. It is one type guard,isRetryableErrorin@aztec-labs/foundation/error, shared with the AVM simulator pool:Asking an instance whether it is alive can only ever be a guess, because the process can die between the answer and the next call. More importantly it cannot tell a dead helper from a bad proof, which is the misattribution this issue is really about: the node reported an infrastructure fault as an invalid user transaction and booked it into
IVC_VERIFIER_FAILURE_COUNT.Kept from #351
Unchanged, and the better part of it:
verifyProofretries a verification whosebbdied or could not start, once, and then throwsProofVerifierUnavailableErrorrather than returning{ valid: false }. RPC admission fails with that instead of rejecting the transaction, andQueuedIVCVerifierdoes not count it as a verification failure.valid: false.QueuedIVCVerifier.stop()stops the verifier before draining its queue, so a verification waiting for a slot fails instead of blocking shutdown. One already starting abbholds shutdown until that start ends, as above.Testing
Run in CI. The double models a healing instance: a death fails the call in flight, retryably, and leaves the instance usable, because a replacement process serves the next call.
bb_verifier.test.tscovers valid, invalid, a bb error while healthy, a retry after a death that succeeds on the replacement without the pool growing, unavailable after two deaths, retrying a failed spawn rather than rejecting the proof, unavailable when a per-call instance cannot start, starting no morebbprocesses than the pool holds, andstop()while verifications wait on abbthat is still starting.bb_js_backend.test.tscovers the pool itself: reuse without exceeding the pool size, a slot whosebbfailed to start being started again by the next borrower,destroy()releasing a borrower waiting for a slot at once and one starting abbwhen its start ends, and an instance borrowed acrossdestroy()being destroyed on release rather than pooled.The
retrycontract these tests are written against is verified end to end in aztec-packages#25548, against a realbb.Size
Against main: +472/−94 across 10 files, most of it tests and the test double. The pool file is +66/−75.