Repository navigation
fix(platform): retry Cloudflare uploads after a 5xx, network error or timeout - #448
Merged
Merged
Conversation
…ork error The Worker upload retried only when Cloudflare answered 2xx with a plain-text body. A 5xx or a connection reset failed the deploy at once, and nothing above it retries: compensation deletes the script and marks the deployment failed. 5xx responses and network errors now use the same four attempts. A module that fails its size or digest check still fails at once, and 4xx (429 included) stays final. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A connection that drops while the platform reads Cloudflare's answer failed the deploy at once, even when the same drop before the headers was retried. The request and its response body now retry together. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When Cloudflare saves the upload but the platform sees a 5xx, a network error or a dropped body, the retry is a replay. Cloudflare rejects every replay: the assets completion token is single-use (400, code 100312) and the migration tag has already moved (412, code 10079). The deploy then failed with a misleading message. After an unclear failure, a replay rejected with one of those codes now reads the dispatch script. If it exists with the expected migration tag, the upload counts as done. Each attempt also gets a 120-second timeout. Cloudflare sets no time limit on a subrequest, so a stalled connection held the deployment until the Queue's 15-minute limit. A 4xx stays final when its body drops. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…error The asset upload session request failed the deploy on the first Cloudflare 5xx or network error. It now uses the same rule as the Worker upload. A repeated session request is safe: only the token from the attempt that succeeds is used. Also correct the asset cleanup cap comment, which cited an old subrequest limit, and stop two deploy tests from sleeping for real on retry backoff. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A timeout, network error or 5xx on the last attempt failed the deploy without the saved-upload check, which ran only when a replay got a 100312 or 10079 answer. The last attempt now reads the dispatch script once before it fails, because that attempt may itself have been saved. The asset upload session also gets a 60-second timeout for each attempt, like the platform's other Cloudflare API calls. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After an unclear failure, the replay is rejected with a 4xx. If that response body dropped, the deploy failed before the saved-upload check ran. It now runs the same check first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🚀 Deploying Preview to Cloudflare 🚀Preview URL: https://fix-deploy-receipt-and-upload-retry-void-cloud.voidzero-docs.workers.dev (commit e5a160a)This URL reflects your latest Preview deploymentPreview Deployments by commit
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
One short Cloudflare error fails a whole deploy.
The platform sends two requests to Cloudflare for every deploy with a Worker: the asset upload session, then the Worker upload to the dispatch namespace. Both run inside
retryWithBackoff, but both retried only a 2xx plain-text body. A 5xx or a dropped connection failed the deploy on the first try:A stalled connection was worse. Cloudflare sets no time limit on a subrequest, so the deploy waited until the Queue's 15-minute limit. Recovery then cancelled it with "Deployment execution was interrupted", about 18 minutes later. Until then, every new deploy got 409.
This affects every deploy with a Worker, on Void Cloud and on self-hosted installs, from old and new CLIs.
What this PR changes
Both requests keep the same limit: 4 attempts with 1 s, 2 s and 4 s sleeps (
CF_RETRY, unchanged). The plain-text retry uses the same 4 attempts.Assets have…)A retry after Cloudflare already saved the upload
A 5xx, a reset, a timeout or a dropped body can hide an upload that Cloudflare saved. The next attempt is then a replay. A live check on a Cloudflare account showed that Cloudflare rejects every such replay:
Without a check, the retry would end with "Completion token has already been consumed", which points to the wrong cause. The Worker upload now checks for this case:
The script name is unique to each deployment, and only this deployment writes it, always with the same bytes. So a saved script is this upload. The readiness check after the upload still runs.
Decisions
platform/meta/operations.mdalready says that a rate limit fails deploys.WorkerModuleIntegrityErrordurable-snapshot.tsuses the same rule.cfFetch).warnfor an attempt that will be retried,errorfor the final oneAlso in this PR:
deploy/shared.tscited a subrequest limit of 1,000. That limit is out of date, and no code reads the number. The comment now gives the real reason for the cap. The cap stays at 200.Known limits
result.script.migration_tag) comes from wrangler and from an earlier production fix in this repo. If the shape differs, the check finds nothing and the deploy fails as before.Testing
vp run test --base origin/mainvp run checkvp test run(before the last two commits)🤖 Generated with Claude Code