fix(providers): summarize per-provider errors instead of dumping err.stack - #1441
Conversation
When quorum is not reached and no fallback providers remain, the early-exit throwQuorumError() called getMismatchedProviders(), which closed over quorumResult - a const declared only after the fallback results are counted. That access lands in the temporal dead zone, so instead of the intended 'Not enough providers agreed to meet quorum' error the call site crashes with a ReferenceError, and the diagnostic warn log is skipped, hiding which providers disagreed. Pass the quorum result into getMismatchedProviders explicitly and skip the mismatch computation on the early-exit path, where no quorum result exists to compare against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stack
RetryProvider.send and the Solana QuorumFallback factory accumulate
per-provider failures via `err.stack || err.toString()`. For ethers
transport errors (SERVER_ERROR, etc.), `err.stack` and the stringified
form embed the failed RPC URL with its API key in cleartext, plus a
multi-line ethers stack trace. That accumulator then gets concatenated
into the aggregate "Not enough providers succeeded" error message and
into the `logQuorumMismatchOrFailureDetails` warn log — both reach
Slack/Datadog and leak the key.
Introduce `summarizeProviderError(err)` which produces a single-line,
log-safe summary. Preference order:
1. Parsed JSON-RPC error message (e.g. "execution reverted: ERC20:
burn amount exceeds balance"), drilling into `err.error` if
necessary — this is what callers actually want.
2. `err.reason` — ethers' short, URL-free reason string.
3. `err.message` for non-ethers errors (no `err.code` present).
4. `err.code` (e.g. "SERVER_ERROR", "NETWORK_ERROR") as a last resort.
`err.message` is deliberately skipped for ethers-shaped errors because
that's exactly where transport-layer errors splice in the failed URL.
The original error remains reachable via `error.cause` on the aggregate
thrown by `createSendErrorWithMessage`, so forensic detail isn't lost.
Wire it into both error-collection sites in `retryProvider.ts` and both
in `solana/quorumFallbackRpcFactory.ts`. No behavior change beyond the
shape of the recorded error string.
Rebase notes (onto #1483's branch): master's Solana path meanwhile
introduced formatRpcError(), which preserves the SolanaError JSON-RPC
code in the wrap message. Keep formatRpcError() at the Solana call
sites and replace only its non-Solana fallback (err.stack ||
err.toString()) with summarizeProviderError(), so both intents compose:
code preservation for SolanaErrors, no URL/API-key leakage for
everything else.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
725cc21 to
2d93130
Compare
|
Rebased onto #1483 and retargeted the base to its branch per Paul, so the two stack: merge #1483 first, then this one (GitHub retargets this PR to Conflict resolution note: No separate version bump here — this ships with the 4.4.12 bump from #1483 in the same release. 🤖 Generated with Claude Code |
Restore the original comment in quorumFallbackRpcFactory per review feedback; the note about non-Solana architectures didn't belong here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the review: reverted the comment in |
|
Thanks for the review and approval! No outstanding feedback to address — this is ready to merge whenever you are. |
Picks up across-protocol/sdk#1483 (quorum failures throw the intended error instead of a ReferenceError, fixing the recurring zion-across-proposer crash) and across-protocol/sdk#1441 (log-safe per-provider error summaries, no RPC URL/API-key leakage). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Picks up two RetryProvider fixes shipping together in 4.4.12: - across-protocol/sdk#1483 — quorum failures with no fallback providers left now throw the intended "Not enough providers agreed to meet quorum" error instead of crashing with `ReferenceError: Cannot access 'quorumResult' before initialization`. This is the error that has been killing `zion-across-proposer` runs (5 consecutive runs on 2026-07-15, previously 2026-06-22) and other dataworker/monitor bots since April. - across-protocol/sdk#1441 — per-provider failures are summarized into single-line, log-safe messages instead of `err.stack`, keeping RPC URLs (and embedded API keys) out of aggregate error messages and logs. Co-authored-by: droplet-rl <284132418+droplet-rl@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
RetryProvider.send(and the SolanaQuorumFallbackSolanaRpcFactorycounterpart) accumulate per-provider failures via(err as any)?.stack || err?.toString(). For ethers transport errors (SERVER_ERROR,NETWORK_ERROR, etc.), botherr.stackand the stringified form embed the failed RPC URL with its API key in cleartext, alongside a multi-line ethers stack trace. That accumulator then gets concatenated into the aggregateNot enough providers succeeded. Errors: …error message and into thelogQuorumMismatchOrFailureDetailswarn log — both reach Slack/Datadog. So in addition to being unreadable, this is leaking provider API keys to log destinations.Real example surfaced from the Across relayer warning logs (URL/key redacted here):
After this change, the same failure produces:
Approach
New
summarizeProviderError(err)helper insrc/providers/utils.tsthat produces a single-line, log-safe summary. Preference order:parseJsonRpcError(drilling intoerr.errorif needed) — this is the actual revert reason that callers want.err.reason— ethers' short, URL-free reason string (processing response error,execution reverted, etc.).err.message, only for non-ethers errors (noerr.codepresent) — safe because there's no transport-layer URL to leak.err.code(SERVER_ERROR,NETWORK_ERROR, …) as a last resort.err.messageis deliberately skipped for ethers-shaped errors because that's where ethers'Logger.makeErrorsplices the failed URL into the message. The original error remains reachable viaerror.causeon the aggregate thrown bycreateSendErrorWithMessage, so any forensic detail isn't lost — it just stops being on the user-visible log path.Wired into both
errors.push([provider, …])sites inretryProvider.tsand both insolana/quorumFallbackRpcFactory.ts. No other behavior change:failImmediatestill callsparseJsonRpcErroron the raw error; the originalerris still thrown/cascaded as before; classification downstream (e.g. relayer'sMultiCallerClient#canIgnoreRevertReason) is.includes()based and continues to match the cleaner string.Test plan
summarizeProviderErrorintest/providers/utils.test.tscovering: null/undefined, string inputs, JSON-RPC body extraction (top-level + nestederr.error),.reasonfallback,.codefallback, non-ethers.message, and an explicit URL-leak guard.createSendErrorWithMessage+providers.test.tssuites still pass.prettier --checkandeslintclean on touched files.tsc -p tsconfig.build.json --noEmitclean.Context
Found while investigating noisy
MultiCallerClient#LogSimulationFailureswarnings in the Across relayer. A relayer-side regex sanitizer was considered first (across-protocol/relayer#3411, now closed) but fixing it once at the source — here — is cleaner and benefits every consumer (relayer, dataworker, finalizer, monitor) with a single version bump.🤖 Generated with Claude Code