Skip to content

fix(providers): summarize per-provider errors instead of dumping err.stack - #1441

Merged
pxrl merged 3 commits into
masterfrom
droplet/T90K0AL22-C03GHT4RV42-1779566476-782239
Jul 15, 2026
Merged

fix(providers): summarize per-provider errors instead of dumping err.stack#1441
pxrl merged 3 commits into
masterfrom
droplet/T90K0AL22-C03GHT4RV42-1779566476-782239

Conversation

@droplet-rl

Copy link
Copy Markdown
Contributor

Summary

RetryProvider.send (and the Solana QuorumFallbackSolanaRpcFactory counterpart) accumulate per-provider failures via (err as any)?.stack || err?.toString(). For ethers transport errors (SERVER_ERROR, NETWORK_ERROR, etc.), both err.stack and the stringified form embed the failed RPC URL with its API key in cleartext, alongside a multi-line ethers stack trace. That accumulator then gets concatenated into the aggregate Not enough providers succeeded. Errors: … error message and into the logQuorumMismatchOrFailureDetails warn log — both reach Slack/Datadog. So in addition to being unreadable, this is leaking provider API keys to log destinations.

Real example surfaced from the Across relayer warning logs (URL/key redacted here):

reason: "Not enough providers succeeded. Errors:\nProvider https://arb-mainnet.g.alchemy.com failed with error: Error: processing response error (body=\"{...message:\\\"execution reverted: ERC20: burn amount exceeds balance\\\"...}\", url=\"https://arb-mainnet.g.alchemy.com/v2/<API_KEY>\", code=SERVER_ERROR, version=web/5.7.1)\n    at Logger.makeError (...)\n    at Logger.throwError (...)\n    ... <truncated, ~2KB total>"

After this change, the same failure produces:

Not enough providers succeeded. Errors:
Provider https://arb-mainnet.g.alchemy.com failed with error: execution reverted: ERC20: burn amount exceeds balance

Approach

New summarizeProviderError(err) helper in src/providers/utils.ts that produces a single-line, log-safe summary. Preference order:

  1. Parsed JSON-RPC error message via the existing parseJsonRpcError (drilling into err.error if needed) — this is the actual revert reason that callers want.
  2. err.reason — ethers' short, URL-free reason string (processing response error, execution reverted, etc.).
  3. err.message, only for non-ethers errors (no err.code present) — safe because there's no transport-layer URL to leak.
  4. err.code (SERVER_ERROR, NETWORK_ERROR, …) as a last resort.

err.message is deliberately skipped for ethers-shaped errors because that's where ethers' Logger.makeError splices the failed URL into the message. The original error remains reachable via error.cause on the aggregate thrown by createSendErrorWithMessage, so any forensic detail isn't lost — it just stops being on the user-visible log path.

Wired into both errors.push([provider, …]) sites in retryProvider.ts and both in solana/quorumFallbackRpcFactory.ts. No other behavior change: failImmediate still calls parseJsonRpcError on the raw error; the original err is still thrown/cascaded as before; classification downstream (e.g. relayer's MultiCallerClient#canIgnoreRevertReason) is .includes() based and continues to match the cleaner string.

Test plan

  • 8 new unit tests for summarizeProviderError in test/providers/utils.test.ts covering: null/undefined, string inputs, JSON-RPC body extraction (top-level + nested err.error), .reason fallback, .code fallback, non-ethers .message, and an explicit URL-leak guard.
  • Existing createSendErrorWithMessage + providers.test.ts suites still pass.
  • prettier --check and eslint clean on touched files.
  • tsc -p tsconfig.build.json --noEmit clean.

Context

Found while investigating noisy MultiCallerClient#LogSimulationFailures warnings in the Across relayer. A relayer-side regex sanitizer was considered first (across-protocol/relayer#3411, now closed) but fixing it once at the source — here — is cleaner and benefits every consumer (relayer, dataworker, finalizer, monitor) with a single version bump.

🤖 Generated with Claude Code

droplet-rl and others added 2 commits July 15, 2026 15:48
When quorum is not reached and no fallback providers remain, the
early-exit throwQuorumError() called getMismatchedProviders(), which
closed over quorumResult - a const declared only after the fallback
results are counted. That access lands in the temporal dead zone, so
instead of the intended 'Not enough providers agreed to meet quorum'
error the call site crashes with a ReferenceError, and the diagnostic
warn log is skipped, hiding which providers disagreed.

Pass the quorum result into getMismatchedProviders explicitly and skip
the mismatch computation on the early-exit path, where no quorum result
exists to compare against.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stack

RetryProvider.send and the Solana QuorumFallback factory accumulate
per-provider failures via `err.stack || err.toString()`. For ethers
transport errors (SERVER_ERROR, etc.), `err.stack` and the stringified
form embed the failed RPC URL with its API key in cleartext, plus a
multi-line ethers stack trace. That accumulator then gets concatenated
into the aggregate "Not enough providers succeeded" error message and
into the `logQuorumMismatchOrFailureDetails` warn log — both reach
Slack/Datadog and leak the key.

Introduce `summarizeProviderError(err)` which produces a single-line,
log-safe summary. Preference order:

  1. Parsed JSON-RPC error message (e.g. "execution reverted: ERC20:
     burn amount exceeds balance"), drilling into `err.error` if
     necessary — this is what callers actually want.
  2. `err.reason` — ethers' short, URL-free reason string.
  3. `err.message` for non-ethers errors (no `err.code` present).
  4. `err.code` (e.g. "SERVER_ERROR", "NETWORK_ERROR") as a last resort.

`err.message` is deliberately skipped for ethers-shaped errors because
that's exactly where transport-layer errors splice in the failed URL.
The original error remains reachable via `error.cause` on the aggregate
thrown by `createSendErrorWithMessage`, so forensic detail isn't lost.

Wire it into both error-collection sites in `retryProvider.ts` and both
in `solana/quorumFallbackRpcFactory.ts`. No behavior change beyond the
shape of the recorded error string.

Rebase notes (onto #1483's branch): master's Solana path meanwhile
introduced formatRpcError(), which preserves the SolanaError JSON-RPC
code in the wrap message. Keep formatRpcError() at the Solana call
sites and replace only its non-Solana fallback (err.stack ||
err.toString()) with summarizeProviderError(), so both intents compose:
code preservation for SolanaErrors, no URL/API-key leakage for
everything else.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@droplet-rl
droplet-rl changed the base branch from master to droplet/C03GHT4RV42-1784127850-691789 July 15, 2026 15:58
@droplet-rl
droplet-rl force-pushed the droplet/T90K0AL22-C03GHT4RV42-1779566476-782239 branch from 725cc21 to 2d93130 Compare July 15, 2026 15:58
@droplet-rl

Copy link
Copy Markdown
Contributor Author

Rebased onto #1483 and retargeted the base to its branch per Paul, so the two stack: merge #1483 first, then this one (GitHub retargets this PR to master automatically when #1483 merges).

Conflict resolution note: master meanwhile introduced formatRpcError() on the Solana path, which preserves the SolanaError JSON-RPC code in the wrap message. Kept formatRpcError() at the Solana call sites and replaced only its non-Solana fallback (err.stack || err.toString()) with summarizeProviderError(), so both intents compose: code preservation for SolanaErrors, no URL/API-key leakage for everything else. The formatRpcError fallback test was updated to match (undefined → "unknown error", no stack in the summary).

No separate version bump here — this ships with the 4.4.12 bump from #1483 in the same release.

🤖 Generated with Claude Code

Base automatically changed from droplet/C03GHT4RV42-1784127850-691789 to master July 15, 2026 16:02
Comment thread src/providers/solana/quorumFallbackRpcFactory.ts Outdated
Comment thread src/providers/retryProvider.ts
Restore the original comment in quorumFallbackRpcFactory per review
feedback; the note about non-Solana architectures didn't belong here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@droplet-rl

Copy link
Copy Markdown
Contributor Author

Addressed the review: reverted the comment in src/providers/solana/quorumFallbackRpcFactory.ts back to its original wording ("Preserve the underlying JSON-RPC error code in the wrap message.") in 739b395, and replied on the inline thread.

@droplet-rl

Copy link
Copy Markdown
Contributor Author

🔎 View trace

@droplet-rl

Copy link
Copy Markdown
Contributor Author

Thanks for the review and approval! No outstanding feedback to address — this is ready to merge whenever you are.

@pxrl
pxrl merged commit 56cca31 into master Jul 15, 2026
4 checks passed
@droplet-rl

Copy link
Copy Markdown
Contributor Author

🔎 View trace

@pxrl
pxrl deleted the droplet/T90K0AL22-C03GHT4RV42-1779566476-782239 branch July 15, 2026 16:14
droplet-rl added a commit to across-protocol/relayer that referenced this pull request Jul 15, 2026
Picks up across-protocol/sdk#1483 (quorum failures throw the intended
error instead of a ReferenceError, fixing the recurring
zion-across-proposer crash) and across-protocol/sdk#1441 (log-safe
per-provider error summaries, no RPC URL/API-key leakage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pxrl pushed a commit to across-protocol/relayer that referenced this pull request Jul 15, 2026
Picks up two RetryProvider fixes shipping together in 4.4.12:

- across-protocol/sdk#1483 — quorum failures with no fallback providers
left now throw the intended "Not enough providers agreed to meet quorum"
error instead of crashing with `ReferenceError: Cannot access
'quorumResult' before initialization`. This is the error that has been
killing `zion-across-proposer` runs (5 consecutive runs on 2026-07-15,
previously 2026-06-22) and other dataworker/monitor bots since April.
- across-protocol/sdk#1441 — per-provider failures are summarized into
single-line, log-safe messages instead of `err.stack`, keeping RPC URLs
(and embedded API keys) out of aggregate error messages and logs.

Co-authored-by: droplet-rl <284132418+droplet-rl@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants