fix(bb-prover): rebuild the batch verifier after its bb dies

[charlielye]

Follows #354 (merged), which introduces ProofVerifierUnavailableError.

The batch verifier at the p2p layer holds a long-lived bb process with a session on it: the protocol circuit verification keys, a worker pool, and a FIFO for results. When that process dies, the FIFO closes, the verifier latches a fatal error, and it stays dead for the life of the node. Nothing clears the latch and nothing rebuilds the verifier, which is created once at startup.

That is bad on its own, and worse than the equivalent on the RPC path. verifyProof turned any failure into valid: false, and this verifier's result feeds gossip validation as proofValidator at PeerErrorSeverity.LowToleranceError. So a dead helper process made every gossiped transaction look like a bad proof, and the node then worked through its peer set penalising honest peers for its own fault.

A failed verifier is replaced

SelfHealingChonkVerifier wraps the batch verifier and is what the node now creates. On the next verification after the verifier fails, it stops the failed one and creates a fresh one. Everything the verifier's session holds comes from configuration (ProtocolCircuitVks, the batch size, the core count), so a fresh one restores it exactly. Only proofs in flight are lost, and those were already rejected as unavailable.

This is safe here in a way it would not be for Chonk accumulation, where the state being lost is the half-built proof itself. Here the session is really just process initialisation: chonkBatchVerifierStart runs once inside new and is never restarted, so a new verifier restores exactly what was there.

Concurrent callers share one replacement. Attempts are paced at one a second, indefinitely rather than a bounded number of times, and flat rather than backing off; between attempts, verifications are rejected as unavailable. That is the convention avm_simulator_pool settled on, so a node recovers as soon as bb is healthy instead of waiting out a backoff it has already outlived.

BatchChonkVerifier itself stays one session that never rebuilds, as on main. It changes only to report the proofs it could not check as unavailable, and to say whether it has failed. An earlier version rebuilt it in place, swapping a new bb session into the fields of the dead one. That needed guards against the old session's FIFO reader, and a shutdown that waited for the rebuild, which could hang on a bb that never finished starting. Replacing the whole verifier needs neither.

It stops calling an unavailable verifier an invalid proof

A verifier that could not check a proof has not judged it. ProofVerifierUnavailableError now propagates rather than being flattened to valid: false, and that covers proofs in flight when the session fails and proofs bb fails to take, not only calls made while it is down.

Propagating it is not enough on its own, because gossip validation turns any thrown error into a rejection with a penalty. So the second (proof) stage of gossip validation catches it and returns Ignore: the tx is dropped without penalising the peer, as the pool pre-check already does. The error moves to stdlib/errors so p2p can recognise it. This is the same distinction the RPC verifier draws in #354, and here it is the difference between degraded verification and a self-inflicted partition.

Shutdown

stop() stops the verifier in use and never waits for one that is still being created: that one stops itself as soon as it is ready. So a bb that is slow to start cannot hold up shutdown.

Not done

The FIFO side channel stays. It exists because responses come back out of order, and bb.js's socket backend matches responses strictly in order by a callback queue with no ids, so there is nowhere to put them. ipc-runtime's clients do carry request ids, on both the socket and shared-memory paths, so this becomes removable once bb.js moves onto them in aztec-packages#25362 — but not before, and not cheaply.

Testing

• self_healing_chonk_verifier.test.ts, against a fake verifier, covers:
  • passing verifications through while the verifier is healthy;
  • replacing a failed verifier and stopping the failed one;
  • one replacement shared by concurrent verifications;
  • reporting unavailable, and pacing retries, when a replacement cannot start;
  • stop() stopping the verifier in use;
  • stop() not waiting for a replacement still being created, which then stops itself.
• libp2p_service.test.ts covers an unavailable proof verifier: the gossiped tx is ignored, the peer is not penalised and nothing reaches the pool.
• The existing batch verifier integration tests run against the main version of BatchChonkVerifier, including draining proofs accepted before stop().

Not covered: a replacement after a real bb death. That wants an integration test that kills the bb the verifier spawned and verifies again, which needs the process id exposed.

[greptile-apps]

Confidence Score: 4/5

[High risk] Replaces batch verifier with self-healing wrapper that recreates it on failure.

The PR is not yet safe to merge because shutdown can turn an unverified gossiped proof into a peer-penalizing invalid verdict.

Findings

1.  Shutdown can penalize peers ▶
2.  Shutdown can hang during rebuild ▶
3.  Shutdown skips retiring verifier ▶
4.  Recovery lacks regression coverage ▶
5.  Rebuild uses promise callbacks ▶
6.  Rebuild log lacks context ▶

Summary

The PR replaces the node’s peer batch verifier with a wrapper that recreates a failed bb session and distinguishes unavailable verification from an invalid proof.

• Failed verifications can now trigger a shared, paced replacement.
• Gossip ignores unavailable proof checks without penalizing the sender.
• Shutdown avoids waiting for a replacement that is still starting.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Gossiped transaction] --> B[Self-healing verifier]
  B --> C{Current batch verifier failed?}
  C -->|No| D[Verify proof]
  C -->|Yes| E[Create shared replacement]
  E --> D
  E -->|Unavailable| F[Ignore without peer penalty]
  D -->|Invalid proof| G[Reject and penalize]
  D -->|Unavailable| F

Loading

_{Reviews (3) · Last reviewed commit: "fix(txe): stub SelfHealingChonkVerifier ..."}

[greptile-apps]

Shutdown can penalize peers If node shutdown or a realProofs configuration change occurs after liveVerifier() selects a healthy verifier but before this call reaches it, stop() closes the verifier first. BatchChonkVerifier converts its “stopped” error into valid: false, so gossip can reject a valid transaction and penalize its sender instead of ignoring a proof it could not check.

Knowledge Base Used: P2P network implementation

[fcarreiro]

(Written by Claude on behalf of Facundo)

Partly fixed in 440939d. A proof sent to a stopped BatchChonkVerifier now rejects with ProofVerifierUnavailableError, so it's ignored rather than penalised. Proofs already in flight when it stops can still become valid: false, through the stop drain timing out or erroring. I've raised that in the review, and #383 converts those paths. Leaving this open until then.

[greptile-apps]

Shutdown skips retiring verifier During replacement, this clears started and begins stopping the failed verifier without waiting for it. If node shutdown starts then, the wrapper’s stop() returns without waiting for that verifier’s FIFO reader and backend teardown. Shutdown can proceed while those handles are still being released.

Knowledge Base Used: Aztec node service and APIs

[fcarreiro]

(Written by Claude on behalf of Facundo)

Not fixed yet. stop() still doesn't wait for a verifier that liveVerifier() is retiring. That teardown is bounded by the 5 s stop timeouts, so I've listed it as a nit in the review. Leaving this open.

[fcarreiro]

(Written by Claude on behalf of Facundo)

I reasoned through this; I didn't run it. CI is green. Replacing the whole verifier is the right shape: a bb death closes the results FIFO and the verifier latches as unavailable. The next verification then gets a fresh one, and a replacement that finishes after stop() stops itself, so shutdown never waits on one. Gossip ignoring ProofVerifierUnavailableError without a penalty is the important fix.

Two inline comments: a replacement that hangs has no deadline, and some paths still report an unavailable verifier as a verdict (#383 covers those).

Nits:

• stop() during a replacement doesn't wait for the failed verifier's teardown, which liveVerifier() starts with void this.retire(verifier) (Greptile's comment on line 84). It's bounded by the 5 s stop timeouts, and the exit handler cleans up the FIFO, so this is tidiness: keep the promise and await it, bounded, in stop().
• Recovery after a real bb death isn't tested. The description says why: the bb's pid isn't exposed.

[fcarreiro]

(Written by Claude on behalf of Facundo)

A replacement that hangs here stops peer-tx verification for good. create() is BatchChonkVerifier.new, which runs Barretenberg.new, then initSRSChonk(), then chonkBatchVerifierStart() (batch_chonk_verifier.ts:122-140). bb.js bounds only the first, with its 60 s startup timeout. The other two are calls on a running bb, with no timeout.

If either hangs, this.current never settles, and every later gossiped tx waits on it in liveVerifier() forever. It isn't rejected as unavailable, and no further replacement is tried. That's plausible right after a death caused by memory pressure.

Suggest a deadline on creation. Past it, treat the creation as failed, so the once-per-interval retry takes over, and stop the late verifier if it ever finishes.

[fcarreiro]

(Written by Claude on behalf of Facundo)

This turns the stopped path into unavailable, but three paths still reject in-flight proofs with plain Errors. verifyProof turns those into valid: false, so gossip still penalises the peer:

• the per-request result timeout (:194)
• the stop drain timing out (:258)
• an error during stop (:262)

Callers other than gossip also see the new rejection, where they used to get valid: false:

• The req/resp batch requester counts a rejected validation as invalid and penalises the peer (batch_tx_requester.ts:517-533).
• Block-proposal tx validation (validateTxsReceivedInBlockProposal) throws instead of reporting invalid txs.
• TxValidationCache keeps the rejected promise until the LRU evicts it, so the tx stays failed after the verifier recovers.

#383 covers all of these, so it's fine if the two land together. If #383 lags, the three conversions above are worth folding in here.