Skip to content

Stage deploy modules in parallel and explain deploy timeouts - #382

Merged
Brooooooklyn merged 3 commits into
mainfrom
fix/deploy-finalize-fail-fast
Oct 7, 2026
Merged

Brooooooklyn merged 3 commits into
mainfrom
fix/deploy-finalize-fail-fast

Conversation

@Brooooooklyn

@Brooooooklyn Brooooooklyn commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Problem

A hosted void deploy of an app with 421 worker modules failed after about 20 minutes:

■  deploy: Connection lost during deploy: 'fetch failed'.
     deploy_fetch_rejected attempts=4
     err=TypeError: fetch failed -> HeadersTimeoutError: Headers Timeout Error

Before it sends response headers, the deploy POST stages every worker module to R2, one at a time:

CLI ── POST /projects/:id/deploy ──► void-api
                                      stage module 1 ─► R2   (~0.7s each)
                                      stage module 2 ─► R2
                                      ...
                                      stage module 421 ─► R2
                                      admit + queue
CLI ◄──────────── headers ─────────── (270–320s later)

The production data for the failed run:

Try Server time before headers CLI limit R2 put rate
1 316s 300s ✗ 1.38/s
2 314s 300s ✗ 1.48/s
3 319s 300s ✗ 1.37/s
4 318s 300s ✗ 1.41/s
rerun next day 273s 300s ✓ 1.59/s

Each try used only about 1s of CPU, so the Worker was waiting on R2. A 15% change in R2 speed decided whether a deploy passed or failed.

  • Try 1 actually deployed. The server went on after the CLI disconnected, but the CLI reported "Deploy failed".
  • Tries 2–4 sent the same idempotency key. Each one staged all 421 modules again, then found try 1's deployment. They failed only because the staging again took more than 300s.

What this PR changes

Server  POST /deploy (before headers)
  before:  for each module: arrayBuffer → hash → put        (1 put at a time)
  after:   for each module: arrayBuffer → hash → drop buffer
           then put the original Blobs, 6 in flight          (Workers limit for pending R2 calls)
           421 modules: ~300s → about 50s (estimate)

CLI  deploy POST
  retries:  unchanged (4 tries, same idempotency key)
  message:  'fetch failed' → says what happened and what to do

Message when every try hit the headers timeout:

deploy: The Void platform did not respond while preparing the deploy. Tried 4 times; the last try waited 5 minutes.
It may still finish. Check `void project status`, then run `void deploy` again to retry.
If this keeps happening, share the detailed log below.
Detailed log: /home/runner/.void/logs/deploy-....jsonl

Other fetch failures now show their cause:

deploy: Connection lost during deploy: 'fetch failed' (ECONNREFUSED).

Decisions

Decision Why
6 puts in flight Workers allows at most 6 pending R2 calls. A bigger pool only queues inside the runtime and holds more memory.
Hash one at a time, then put the Blob Keeps at most one extra copy of module bytes in memory. A near-cap request already sits close to the 128 MiB isolate limit.
Stop after the first failed put Running puts finish before the prefix is deleted, so no object lands after cleanup. The receipt is still written last.
Keep retrying a headers timeout The server got the request and keeps working. A retry with the same idempotency key returns the running deployment's receipt, and the CLI follows it to the real result. Failing on the first timeout would report failure for a deploy that may land.
Keep the 300s timeout With parallel staging, the server should reply well within it.

Rollout

deploy platform (void-api) ──► publish CLI
        │                          │
        └ fixes the timeout        └ clear message if a
          for current CLIs           server is slow again

Known limits

  • A retry still stages everything again before the idempotency check. If R2 is slow enough to cause one timeout, the retries can time out too. A follow-up can look up the idempotency key before staging. On the failed run, that would have returned try 1's receipt at try 3.
  • Staging is still one R2 put per module. The expected time for 421 modules is about 50s (estimate). Staging as one object would change the queue payload.
  • Possible R2 region gap (inferred, not measured). The R2 bucket is in APAC and void-api runs in aws:us-east-2, which fits the ~0.7s per put. This is a separate infra change.
  • No progress line while the CLI retries. The client has no access to the spinner, so "Finalizing deploy..." stays on screen. The retries are in the detailed log.

Testing

Check Result
New staging unit tests At most 6 puts in flight, pool stays full, order and digest stable, receipt last, failure stops new puts and cleans up, put gets the original Blob
CLI tests Headers timeout retried 4 times with one idempotency key; a retry that gets an existing deployment's receipt follows it to done; exact message text for the timeout and for ENOTFOUND / ECONNREFUSED / UND_ERR_SOCKET
Old code against new tests Fails
vp test run Pass
api tsc --noEmit, void / isr typecheck, vp lint, vp fmt Pass

The second commit fixes two flaky platform tests found during verification. The native-form helper now starts Chromium once per file, not twice inside a 5s test. The email rate-limit test now freezes Date, so it cannot cross a minute boundary.

🤖 Generated with Claude Code

Brooooooklyn and others added 2 commits October 7, 2026 17:30
The deploy POST staged every worker module to R2 one at a time before it
sent response headers. For an app with 421 modules this took 270-320s,
past the CLI's 300s headers timeout. The CLI then retried three times,
and each retry staged everything again before it found the existing
idempotency key.

- Hash modules one at a time, then upload the original Blobs with up to
  six puts in flight (the Workers limit for pending R2 calls).
- Do not retry the deploy POST after a headers timeout, and explain that
  the deploy may still finish.
- Show the root cause of other fetch failures (e.g. ECONNREFUSED).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Launch Chromium once per test file and give each form submission its
  own context, so browser start and close no longer count against the
  test timeout.
- Freeze Date in the per-minute rate-limit test so it cannot cross a
  minute boundary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T09:46:03.032373Z 03dac88 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://fix-deploy-finalize-fail-fast-void-cloud.voidzero-docs.workers.dev (commit 03dac88)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://2dd57c3b-void-cloud.voidzero-docs.workers.dev 03dac88 2026-10-07T09:44:11.045Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://6387a999-void-cloud.voidzero-docs.workers.dev ec93e8c 2026-10-07T09:31:31.178Z Visit the dashboard ↗

A headers timeout means the server received the request and is still
working on it. Every retry sends the same idempotency key, so a retry
that reaches the server returns the running deployment's receipt and the
CLI follows it to the real result. Failing on the first timeout would
report a failure for a deploy that may still land.

When every attempt fails, the message now says how many tries were made
and that the deploy may still finish.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Brooooooklyn Brooooooklyn changed the title Stage deploy modules in parallel and fail fast on deploy timeouts Stage deploy modules in parallel and explain deploy timeouts Oct 7, 2026
@cpojer

cpojer commented Oct 7, 2026

Copy link
Copy Markdown
Member

Good to merge.

@Brooooooklyn
Brooooooklyn merged commit 44b931a into main Oct 7, 2026
20 checks passed
@Brooooooklyn
Brooooooklyn deleted the fix/deploy-finalize-fail-fast branch October 7, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants