Trace and type the execution rate-limit counter - #1782
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | cec1018 | Commit Preview URL Branch Preview URL |
Aug 27 2026, 10:31 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | cec1018 | Aug 27 2026, 10:31 PM |
Contributor
Cloudflare previewTorn down — the PR is closed. |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
Wrap the counter DO increment in a named span and a typed, classified error, cut the per-increment alarm write, and add a check-timeout env override so the fail-open path is testable.
RhysSullivan
force-pushed
the
fix/rate-limit-counter-observability
branch
from
August 27, 2026 20:26
ea9706c to
963968e
Compare
RhysSullivan
marked this pull request as ready for review
August 28, 2026 00:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every cloud
executemakes a blocking Durable Object call to the per-orghourly counter, under a 2s budget. That call had no span, so when it was slow
the only production evidence was a warning logged 2s after the fact: the
latency was invisible in traces, and each occurrence silently switched the
abuse backstop off for that request while adding a full 2s to a user-facing
execution.
When the counter itself faulted — a storage reset, an overloaded object — the
failure reached error reporting as one untyped
UnknownError: An error occurred in Effect.tryPromise, with no organization, no window, and no applicationframes: a group that identifies nothing and would bury a real misconfiguration
in the same bucket.
Separately, the counter rewrote its purge alarm on every increment: a second
durable write plus an alarm-manager update on the hot path of every execution,
with the object's input gate closed across all three.
Fix
rate_limit.checkspan carrying the organization, window, cap, budget andoutcome, with a
rate_limit.incrementchild around the Durable Object call.RateLimitCounterErrorcarrying a classified platform code(
storage_reset/overloaded/exceeded_memory/network/unknown)in place of the generic wrapper.
is set two windows out, so one write per window is sufficient.
EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MSoverride alongside the existingper-hour cap override. Production leaves it unset.
deliberate degradation, so it is now recorded as
rate_limit.check.timed_outon the span rather than reported as an error; every other counter failure
still reports.
Testing
Cloud e2e,
e2e/cloud/mcp-execution-limits.test.ts, drives real MCPexecutions over the real workerd + Durable Object topology and asserts the
spans the worker actually EXPORTS. All three outcomes of the check are pinned,
one per scenario, so a regression in any single branch fails on its own:
the rate-limit counter check is visible in the exported spans):count, cap,
blocked=false,failed_open=false, and arate_limit.incrementchild for the same window with no fault classification.
the rate-limit backstop blocks runaway executions…): the checkspan carries
blocked=true, the count that crossed the cap,exempt=falseand
failed_open=false— a real block, not a degraded check.a paid org runs past the rate-limit backstop…): over the cap andallowed, with
exempt=trueandfailed_open=false.The budget on the span is asserted to be the boot's override rather than the
compiled-in 2000ms default, so the
EXECUTION_RATE_LIMIT_CHECK_TIMEOUT_MSplumbing is proved end to end instead of only in a unit test; the boot sets a
value LONGER than the default so no other scenario's backstop is disturbed.
Against the pre-fix code all three scenarios fail (
no exported span matched {"operation":"rate_limit.check"…}) while the other three in the file pass;with the fix the file is 6/6 green, three consecutive runs.
Unit tests are kept only for seams the e2e harness cannot observe: a faulting
counter RPC (the harness has no fault seam for a Durable Object call), fail-open
under a check that blows its budget (a tiny budget is a process-wide worker var,
and the cloud project shares one boot whose other scenarios need a working
backstop), and the alarm-write count (storage writes are not observable
black-box).
Notes
The platform-error classification is local to this file; a shared Durable
Object error classifier would be the right home for it once one exists.
Deliberately out of scope: moving the counter Durable Object out of the shared
app script so a cold counter isolate stops paying full app module evaluation,
and sharding or hedging the counter — the structural half, worth its own PR.