Skip to content

fix(platform): retry Cloudflare uploads after a 5xx, network error or timeout - #448

Merged
Brooooooklyn merged 6 commits into
mainfrom
fix/deploy-receipt-and-upload-retry
Oct 16, 2026
Merged

Brooooooklyn merged 6 commits into
mainfrom
fix/deploy-receipt-and-upload-retry

Conversation

@Brooooooklyn

Copy link
Copy Markdown
Member

Problem

One short Cloudflare error fails a whole deploy.

The platform sends two requests to Cloudflare for every deploy with a Worker: the asset upload session, then the Worker upload to the dispatch namespace. Both run inside retryWithBackoff, but both retried only a 2xx plain-text body. A 5xx or a dropped connection failed the deploy on the first try:

attempt 0 ──PUT──▶ Cloudflare 503 / connection reset / stalled
             │
             ▼
   throw ─▶ compensation deletes the script ─▶ deployment "failed"
            (the queue, recovery and the CLI never run the upload again)

A stalled connection was worse. Cloudflare sets no time limit on a subrequest, so the deploy waited until the Queue's 15-minute limit. Recovery then cancelled it with "Deployment execution was interrupted", about 18 minutes later. Until then, every new deploy got 409.

This affects every deploy with a Worker, on Void Cloud and on self-hosted installs, from old and new CLIs.

What this PR changes

Both requests keep the same limit: 4 attempts with 1 s, 2 s and 4 s sleeps (CF_RETRY, unchanged). The plain-text retry uses the same 4 attempts.

Cloudflare answer Before After
2xx done done
2xx plain text (Assets have…) retry retry
4xx, 429 included (even when the body drops) fail fail
5xx fail retry
network error, or the body drops after a 2xx or 5xx fail retry
no answer in time (120 s Worker upload, 60 s asset session) waits ≈ 18 min retry; the last attempt says "did not finish within … seconds"
a stored module fails its size or digest check fail fail at once, never retried

A retry after Cloudflare already saved the upload

A 5xx, a reset, a timeout or a dropped body can hide an upload that Cloudflare saved. The next attempt is then a replay. A live check on a Cloudflare account showed that Cloudflare rejects every such replay:

PUT #1 with assets.jwt        → 200, script saved
PUT #2, same body             → 400  code 100312  "Completion token has already been consumed"
PUT #2 with DO migrations     → 412  code 10079   "Actor migration tag precondition failed"
PUT that Cloudflare rejected  → the token is not used up; the next PUT succeeds

Without a check, the retry would end with "Completion token has already been consumed", which points to the wrong cause. The Worker upload now checks for this case:

earlier attempt failed unclearly ─┐
                                  ├─▶ GET the dispatch script ─▶ exists, same migration tag? ─▶ done
replay rejected 100312 / 10079 ───┘                             no, or the GET fails        ─▶ fail as before

last attempt fails unclearly ─────▶ same GET (this attempt may have been saved too)

The script name is unique to each deployment, and only this deployment writes it, always with the same bytes. So a saved script is this upload. The readiness check after the upload still runs.

Decisions

Question Decision Why
Retry 429? No The whole install shares one API token. Cloudflare blocks it for 5 minutes after the limit, which is longer than this retry window. platform/meta/operations.md already says that a rate limit fails deploys.
Which source failures stop at once Only WorkerModuleIntegrityError The same bytes fail every attempt. A short R2 read error can pass on the next attempt. durable-snapshot.ts uses the same rule.
Timeout value 120 s for each attempt A live upload of 50.5 MB from local workerd took 9.6 s, which is about 13 s for 64 MiB. 120 s is about 9 times that. The custom DO host upload already uses 60 s for the same modules. The timer also covers the module reads that feed the body, so the message names no system.
When to check for a saved upload A replay rejected with 100312 or 10079 after an unclear failure, or an unclear failure on the last attempt No extra request on the normal path, and at most one GET for each deploy. Any other rejection fails as before.
Asset session timeout 60 s for each attempt The same limit as the platform's other Cloudflare API calls (cfFetch).
Is a repeated asset session safe? Yes Production already repeats it on the plain-text path. Only the token from the attempt that succeeds is used.
Log level warn for an attempt that will be retried, error for the final one Error-level alerts then count only the deploys that fail.

Also in this PR:

  • The asset cleanup cap comment in deploy/shared.ts cited a subrequest limit of 1,000. That limit is out of date, and no code reads the number. The comment now gives the real reason for the cap. The cap stays at 200.
  • Two deploy tests slept 1 s for real on retry backoff. They no longer do.

Known limits

  • The GET for the dispatch script was not checked live: the test account has no Workers for Platforms. The response shape (result.script.migration_tag) comes from wrangler and from an earlier production fix in this repo. If the shape differs, the check finds nothing and the deploy fails as before.
  • The GET has two answers: saved, or not. A GET that fails (5xx, 429, network error) counts as "not saved", so the deploy fails and compensation deletes the candidate script, as before this PR. No traffic has reached that script yet, and the live deployment keeps serving. A third answer ("unknown") would need a new deployment state that recovery settles later. That is a larger design change, so this PR leaves it out.

Testing

Check Result
New tests on the old source timeout, saved-upload, 5xx, network-error and dropped-body cases fail; 4xx/429 and the integrity case pass
Same tests on the new source all pass
Mutation checks removing the integrity check, the replay-code check or the last-attempt rethrow each makes a new test fail
Live Cloudflare probe replay codes and token behavior as in the table above; probe scripts deleted
vp run test --base origin/main 217 files pass, 1 skipped; 4,404 tests pass
vp run check 33/33
vp test run (before the last two commits) 714 files pass, 1 skipped; 13,707 tests pass

🤖 Generated with Claude Code

Brooooooklyn and others added 6 commits October 16, 2026 12:14
…ork error

The Worker upload retried only when Cloudflare answered 2xx with a plain-text
body. A 5xx or a connection reset failed the deploy at once, and nothing
above it retries: compensation deletes the script and marks the deployment
failed.

5xx responses and network errors now use the same four attempts. A module
that fails its size or digest check still fails at once, and 4xx (429
included) stays final.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A connection that drops while the platform reads Cloudflare's answer failed
the deploy at once, even when the same drop before the headers was retried.
The request and its response body now retry together.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When Cloudflare saves the upload but the platform sees a 5xx, a network
error or a dropped body, the retry is a replay. Cloudflare rejects every
replay: the assets completion token is single-use (400, code 100312) and
the migration tag has already moved (412, code 10079). The deploy then
failed with a misleading message.

After an unclear failure, a replay rejected with one of those codes now
reads the dispatch script. If it exists with the expected migration tag,
the upload counts as done.

Each attempt also gets a 120-second timeout. Cloudflare sets no time limit
on a subrequest, so a stalled connection held the deployment until the
Queue's 15-minute limit. A 4xx stays final when its body drops.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…error

The asset upload session request failed the deploy on the first Cloudflare
5xx or network error. It now uses the same rule as the Worker upload. A
repeated session request is safe: only the token from the attempt that
succeeds is used.

Also correct the asset cleanup cap comment, which cited an old subrequest
limit, and stop two deploy tests from sleeping for real on retry backoff.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A timeout, network error or 5xx on the last attempt failed the deploy
without the saved-upload check, which ran only when a replay got a 100312
or 10079 answer. The last attempt now reads the dispatch script once
before it fails, because that attempt may itself have been saved.

The asset upload session also gets a 60-second timeout for each attempt,
like the platform's other Cloudflare API calls.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After an unclear failure, the replay is rejected with a 4xx. If that
response body dropped, the deploy failed before the saved-upload check
ran. It now runs the same check first.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T05:46:35.343144Z e5a160a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://fix-deploy-receipt-and-upload-retry-void-cloud.voidzero-docs.workers.dev (commit e5a160a)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://002fe5f6-void-cloud.voidzero-docs.workers.dev e5a160a 2026-10-16T05:44:17.002Z Visit the dashboard ↗

@Brooooooklyn
Brooooooklyn merged commit 6ef71cf into main Oct 16, 2026
20 checks passed
@Brooooooklyn
Brooooooklyn deleted the fix/deploy-receipt-and-upload-retry branch October 16, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant