Skip to content

Trace and type the execution rate-limit counter - #1782

Merged
RhysSullivan merged 3 commits into
mainfrom
fix/rate-limit-counter-observability
Aug 28, 2026
Merged

Trace and type the execution rate-limit counter#1782
RhysSullivan merged 3 commits into
mainfrom
fix/rate-limit-counter-observability

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Problem

Every cloud execute makes a blocking Durable Object call to the per-org
hourly counter, under a 2s budget. That call had no span, so when it was slow
the only production evidence was a warning logged 2s after the fact: the
latency was invisible in traces, and each occurrence silently switched the
abuse backstop off for that request while adding a full 2s to a user-facing
execution.

When the counter itself faulted — a storage reset, an overloaded object — the
failure reached error reporting as one untyped UnknownError: An error occurred in Effect.tryPromise, with no organization, no window, and no application
frames: a group that identifies nothing and would bury a real misconfiguration
in the same bucket.

Separately, the counter rewrote its purge alarm on every increment: a second
durable write plus an alarm-manager update on the hot path of every execution,
with the object's input gate closed across all three.

Fix

  • A rate_limit.check span carrying the organization, window, cap, budget and
    outcome, with a rate_limit.increment child around the Durable Object call.
  • A typed RateLimitCounterError carrying a classified platform code
    (storage_reset / overloaded / exceeded_memory / network / unknown)
    in place of the generic wrapper.
  • The purge alarm is written once per window instead of once per increment; it
    is set two windows out, so one write per window is sufficient.
  • An EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS override alongside the existing
    per-hour cap override. Production leaves it unset.
  • Fail-open semantics are unchanged. A check that blows its own budget is
    deliberate degradation, so it is now recorded as rate_limit.check.timed_out
    on the span rather than reported as an error; every other counter failure
    still reports.

Testing

Cloud e2e, e2e/cloud/mcp-execution-limits.test.ts, drives real MCP
executions over the real workerd + Durable Object topology and asserts the
spans the worker actually EXPORTS. All three outcomes of the check are pinned,
one per scenario, so a regression in any single branch fails on its own:

  • allowed (the rate-limit counter check is visible in the exported spans):
    count, cap, blocked=false, failed_open=false, and a rate_limit.increment
    child for the same window with no fault classification.
  • blocked (the rate-limit backstop blocks runaway executions…): the check
    span carries blocked=true, the count that crossed the cap, exempt=false
    and failed_open=false — a real block, not a degraded check.
  • exempt (a paid org runs past the rate-limit backstop…): over the cap and
    allowed, with exempt=true and failed_open=false.

The budget on the span is asserted to be the boot's override rather than the
compiled-in 2000ms default, so the EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MS
plumbing is proved end to end instead of only in a unit test; the boot sets a
value LONGER than the default so no other scenario's backstop is disturbed.

Against the pre-fix code all three scenarios fail (no exported span matched {"operation":"rate_limit.check"…}) while the other three in the file pass;
with the fix the file is 6/6 green, three consecutive runs.

Unit tests are kept only for seams the e2e harness cannot observe: a faulting
counter RPC (the harness has no fault seam for a Durable Object call), fail-open
under a check that blows its budget (a tiny budget is a process-wide worker var,
and the cloud project shares one boot whose other scenarios need a working
backstop), and the alarm-write count (storage writes are not observable
black-box).

Notes

The platform-error classification is local to this file; a shared Durable
Object error classifier would be the right home for it once one exists.

Deliberately out of scope: moving the counter Durable Object out of the shared
app script so a cold counter isolate stops paying full app module evaluation,
and sharding or hedging the counter — the structural half, worth its own PR.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 27, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing cec1018 Commit Preview URL

Branch Preview URL
Aug 27 2026, 10:31 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 27, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud cec1018 Aug 27 2026, 10:31 PM

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Torn down — the PR is closed.

@pkg-pr-new

pkg-pr-new Bot commented Aug 27, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@1782

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@1782

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@1782

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@1782

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@1782

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@1782

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@1782

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@1782

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@1782

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@1782

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@1782

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@1782

executor

npm i https://pkg.pr.new/executor@1782

commit: cec1018

Wrap the counter DO increment in a named span and a typed, classified
error, cut the per-increment alarm write, and add a check-timeout env
override so the fail-open path is testable.
@RhysSullivan
RhysSullivan force-pushed the fix/rate-limit-counter-observability branch from ea9706c to 963968e Compare August 27, 2026 20:26
@RhysSullivan
RhysSullivan marked this pull request as ready for review August 28, 2026 00:19
@RhysSullivan
RhysSullivan merged commit b885436 into main Aug 28, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant