Skip to content

Persist connection health verdicts on failure and always send ifStaleMs - #1777

Merged
RhysSullivan merged 4 commits into
mainfrom
fix/connection-health-verdicts
Aug 28, 2026
Merged

Persist connection health verdicts on failure and always send ifStaleMs#1777
RhysSullivan merged 4 commits into
mainfrom
fix/connection-health-verdicts

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Problem

When a connection's authorization server stopped re-minting its credential, the connection health endpoint answered with a bare 500 instead of a verdict, and the client kept asking. Two halves:

  • ServerconnectionCheckHealth persisted a verdict only on the success path, and a credential-resolution failure whose code was not invalid_grant was converted into a storage failure. So a connection with a broken credential never got a last_health row, and every probe failed the request.
  • ClientrevalidateQuery sent ifStaleMs only for healthy verdicts, so the server-side freshness gate was bypassed for exactly the connections that were failing. The client's once-per-mount guard is per mount, not per connection, so every surface rendering a broken connection sent its own refresh grant to an upstream that was already refusing.

In production a handful of broken connections produced hundreds of server errors on the health endpoint plus unbounded refresh traffic to third-party token endpoints.

Fix

Both health paths — credential-only and probing — now fold a credential-resolution failure through the single healthFromCredentialResolutionFailure mapping and persist the resulting expired/degraded verdict; the divergent healthFromCredentialResolutionError is deleted. The failure channel stays reserved for genuine storage faults, so an infra blip still fails the request rather than persisting as a verdict. The hook always sends ifStaleMs: the long window for a healthy verdict, a short 30s window for anything else.

Testing

Two e2e scenarios in e2e/scenarios/connection-health-verdict.test.ts (run on the cloud target), one per half of the symptom. Both complete a real authorization-code flow against a test authorization server that refuses every refresh, with a health check declared on the integration so the connection takes the probing path.

API scenario — the server half:

  • the health request returns a degraded verdict carrying the AS's reason, not a failed request;
  • the verdict is persisted on the connection;
  • three repeat checks inside the freshness window return that same verdict, and the AS's request ledger shows no further refresh grant;
  • a second connection whose grant is revoked (invalid_grant) reads expired and persists too.

Browser scenario — the client half, driven only by navigation between the two surfaces a user actually visits (integration page, integrations list, back again). After the first surface's probe is the baseline, the authorization server's ledger shows no further refresh grant, while the app is still asking — and every automatic health request carries an ifStaleMs window.

Red against pre-fix product code: the API scenario's health request fails with an InternalError; reverting only the hook leaves the browser scenario failing on the ledger (extra refresh grants) and on the missing ifStaleMs.

One unit test remains, packages/react/src/lib/use-connection-health.test.ts, pinning the exact window widths per verdict status — cheaper than a browser run per status. The server-side unit tests that duplicated the scenario were removed.

Out of scope, deliberately: refresh classification itself, the persistent refresh cool-off, and vault write policy.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 27, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud 295359f Aug 27 2026, 10:47 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 27, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing 295359f Commit Preview URL

Branch Preview URL
Aug 27 2026, 10:45 PM

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

@pkg-pr-new

pkg-pr-new Bot commented Aug 27, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1777

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1777

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1777

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1777

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1777

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1777

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1777

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1777

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1777

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1777

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1777

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1777

executor

npm i https://pkg.pr.new/executor@1777

commit: 295359f

@RhysSullivan
RhysSullivan force-pushed the fix/connection-health-verdicts branch from 8b0e52d to 5ed8ba4 Compare August 27, 2026 20:25
Sending a freshness window for a non-healthy verdict gated the probe
against the verdict the previous probe had just persisted, so an expired
or degraded connection could not turn green until the window elapsed.
That broke the recovery-on-next-load contract the health-checks-ui,
graphql-introspection-health and mcp-oauth-reconnect-health scenarios
are built on.

Restore the unconditional probe for non-healthy verdicts and keep the
server-side change, which is what actually fixes the reported symptom: a
refused refresh now folds into a persisted verdict instead of the
failure channel, so it is answered rather than raised.
The exact-count assertion sampled refresh grants at one instant while
probes were still in flight, so it read 3 where it expected 2. Assert
the two invariants that do not race instead: grants never outnumber the
health requests that caused them, and a settled page issues no further
probes.
@RhysSullivan
RhysSullivan marked this pull request as ready for review August 28, 2026 01:58
@RhysSullivan
RhysSullivan merged commit 5062eb5 into main Aug 28, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant