Skip to content

Shape storage error messages and classify connection faults - #1784

Merged
RhysSullivan merged 3 commits into
mainfrom
fix/storage-error-shaping
Aug 27, 2026
Merged

Shape storage error messages and classify connection faults#1784
RhysSullivan merged 3 commits into
mainfrom
fix/storage-error-shaping

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Problem

When the database was unreachable or a query was issued on a pool that had already been torn down, the failure surfaced with the driver's raw text as its message: Failed query: <sql>\nparams: <bound values>. Two consequences in production:

  • Bound parameters — organization ids, user ids, user-chosen connection names — ended up in error-report titles.
  • One defect fanned out into a separate report per table and per WHERE-clause shape, because the statement text was the grouping key.

Nothing classified the underlying faults either: a lost socket, a pool that had already been ended, and a socket belonging to a different request context all melted into the same generic storage failure.

Fix

  • fumaFailureFromCause builds the message from stable inputs only — the call-site label plus the driver's error code, e.g. FumaDB plugin_storage.findFirst failed: CONNECTION_ENDED. The full driver error stays on cause, so nothing diagnostic is lost.
  • Adds StorageConnectionError to the StorageFailure union. A code-walk beside the existing unique-violation walk recognises CONNECTION_ENDED / CONNECTION_CLOSED / CONNECTION_DESTROYED / CONNECT_TIMEOUT / ECONNREFUSED / ECONNRESET and the workerd cross-request I/O rejection, and carries the fault code plus a retryable flag so a transient socket loss can be told apart from a pool-lifetime bug.
  • The HTTP edge translator handles the new tag exactly as it handles StorageError: capture the cause, fail with an opaque 500.

Testing

e2e/cloud/storage-error-report-shape.test.ts — "Storage · a rejected write is reported without its SQL or the caller's data". The failure is induced through the public typed API alone: PostgreSQL cannot store a NUL byte in a text column, so a connection whose description carries one is rejected with SQLSTATE 22021 after the statement and its bound parameters are assembled — the same class of failure the production reports came from, with no database or process manipulation. The scenario then asserts both public surfaces:

  • The caller gets an opaque InternalError carrying only a trace id, and no driver text in the payload.
  • The trace id joins to the report the server filed. Its headline — the exception type and message the reporter groups by — carries no statement text, no bound parameters, and none of the caller's identifiers, while still naming the failing operation and the database's error code. The driver's own text is still filed one level down, so the diagnosis survives.
  • Two rejected writes with different names, descriptions and secrets file the same headline: one defect, one report.

Verified red against the pre-fix code (headline is StorageError: Failed query: insert into "connection" … params: …, org_…) and green after; green three consecutive times.

Connection-fault classification stays unit-only. Inducing a real connection fault needs the shared database taken down mid-request, and a database that is down fails first in the auth layer, which never reaches this code — so there is no black-box path to it. packages/core/sdk/src/fuma-runtime.test.ts pins the code matrix against synthetic reconstructions of the driver error shapes, and packages/core/api/src/observability.test.ts pins that the edge translates a connection fault into the same opaque 500 as any other storage failure.

Repo format:check, lint and typecheck pass.

Deliberately not in this PR

The actual escape — a use() callback outliving its request-scoped pool — needs its own designed fix, as does any retry that consumes retryable. This PR only stops one cause from looking like six and stops customer identifiers reaching error-report titles.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 27, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing f7112f6 Commit Preview URL

Branch Preview URL
Aug 27 2026, 10:27 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 27, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud f7112f6 Aug 27 2026, 10:29 PM

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

@pkg-pr-new

pkg-pr-new Bot commented Aug 27, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1784

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1784

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1784

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1784

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1784

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1784

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1784

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1784

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1784

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1784

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1784

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1784

executor

npm i https://pkg.pr.new/executor@1784

commit: f7112f6

@RhysSullivan
RhysSullivan force-pushed the fix/storage-error-shaping branch from 901accb to bf5263f Compare August 27, 2026 20:32
@RhysSullivan
RhysSullivan marked this pull request as ready for review August 27, 2026 23:56
@RhysSullivan
RhysSullivan merged commit 55180cb into main Aug 27, 2026
44 checks passed
@RhysSullivan RhysSullivan mentioned this pull request Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant