Repository navigation
Commit 44b931a
Stage deploy modules in parallel and explain deploy timeouts (#382)
## Problem
A hosted `void deploy` of an app with 421 worker modules failed after
about 20 minutes:
```
■ deploy: Connection lost during deploy: 'fetch failed'.
deploy_fetch_rejected attempts=4
err=TypeError: fetch failed -> HeadersTimeoutError: Headers Timeout Error
```
Before it sends response headers, the deploy POST stages every worker
module to R2, **one at a time**:
```
CLI ── POST /projects/:id/deploy ──► void-api
stage module 1 ─► R2 (~0.7s each)
stage module 2 ─► R2
...
stage module 421 ─► R2
admit + queue
CLI ◄──────────── headers ─────────── (270–320s later)
```
The production data for the failed run:
| Try | Server time before headers | CLI limit | R2 put rate |
|---|---|---|---|
| 1 | 316s | 300s ✗ | 1.38/s |
| 2 | 314s | 300s ✗ | 1.48/s |
| 3 | 319s | 300s ✗ | 1.37/s |
| 4 | 318s | 300s ✗ | 1.41/s |
| rerun next day | 273s | 300s ✓ | 1.59/s |
Each try used only about 1s of CPU, so the Worker was waiting on R2. A
15% change in R2 speed decided whether a deploy passed or failed.
- **Try 1 actually deployed.** The server went on after the CLI
disconnected, but the CLI reported "Deploy failed".
- **Tries 2–4 sent the same idempotency key.** Each one staged all 421
modules again, then found try 1's deployment. They failed only because
the staging again took more than 300s.
## What this PR changes
```
Server POST /deploy (before headers)
before: for each module: arrayBuffer → hash → put (1 put at a time)
after: for each module: arrayBuffer → hash → drop buffer
then put the original Blobs, 6 in flight (Workers limit for pending R2 calls)
421 modules: ~300s → about 50s (estimate)
CLI deploy POST
retries: unchanged (4 tries, same idempotency key)
message: 'fetch failed' → says what happened and what to do
```
Message when every try hit the headers timeout:
```
deploy: The Void platform did not respond while preparing the deploy. Tried 4 times; the last try waited 5 minutes.
It may still finish. Check `void project status`, then run `void deploy` again to retry.
If this keeps happening, share the detailed log below.
Detailed log: /home/runner/.void/logs/deploy-....jsonl
```
Other fetch failures now show their cause:
```
deploy: Connection lost during deploy: 'fetch failed' (ECONNREFUSED).
```
## Decisions
| Decision | Why |
|---|---|
| 6 puts in flight | Workers allows at most 6 pending R2 calls. A bigger
pool only queues inside the runtime and holds more memory. |
| Hash one at a time, then put the Blob | Keeps at most one extra copy
of module bytes in memory. A near-cap request already sits close to the
128 MiB isolate limit. |
| Stop after the first failed put | Running puts finish before the
prefix is deleted, so no object lands after cleanup. The receipt is
still written last. |
| Keep retrying a headers timeout | The server got the request and keeps
working. A retry with the same idempotency key returns the running
deployment's receipt, and the CLI follows it to the real result. Failing
on the first timeout would report failure for a deploy that may land. |
| Keep the 300s timeout | With parallel staging, the server should reply
well within it. |
## Rollout
```
deploy platform (void-api) ──► publish CLI
│ │
└ fixes the timeout └ clear message if a
for current CLIs server is slow again
```
## Known limits
- **A retry still stages everything again before the idempotency
check.** If R2 is slow enough to cause one timeout, the retries can time
out too. A follow-up can look up the idempotency key before staging. On
the failed run, that would have returned try 1's receipt at try 3.
- **Staging is still one R2 put per module.** The expected time for 421
modules is about 50s (estimate). Staging as one object would change the
queue payload.
- **Possible R2 region gap (inferred, not measured).** The R2 bucket is
in APAC and `void-api` runs in `aws:us-east-2`, which fits the ~0.7s per
put. This is a separate infra change.
- **No progress line while the CLI retries.** The client has no access
to the spinner, so "Finalizing deploy..." stays on screen. The retries
are in the detailed log.
## Testing
| Check | Result |
|---|---|
| New staging unit tests | At most 6 puts in flight, pool stays full,
order and digest stable, receipt last, failure stops new puts and cleans
up, put gets the original Blob |
| CLI tests | Headers timeout retried 4 times with one idempotency key;
a retry that gets an existing deployment's receipt follows it to `done`;
exact message text for the timeout and for `ENOTFOUND` / `ECONNREFUSED`
/ `UND_ERR_SOCKET` |
| Old code against new tests | Fails |
| `vp test run` | Pass |
| api `tsc --noEmit`, `void` / `isr` typecheck, `vp lint`, `vp fmt` |
Pass |
The second commit fixes two flaky platform tests found during
verification. The native-form helper now starts Chromium once per file,
not twice inside a 5s test. The email rate-limit test now freezes
`Date`, so it cannot cross a minute boundary.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>1 parent f0bf5bc commit 44b931a
9 files changed
Lines changed: 470 additions & 30 deletions
File tree
- packages/void
- src/cli
- test/unit
- platform/packages/api
- src
- test
- integration
- unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
170 | 170 | | |
171 | 171 | | |
172 | 172 | | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
173 | 190 | | |
174 | 191 | | |
175 | 192 | | |
| |||
577 | 594 | | |
578 | 595 | | |
579 | 596 | | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
580 | 601 | | |
581 | 602 | | |
582 | 603 | | |
| |||
590 | 611 | | |
591 | 612 | | |
592 | 613 | | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
593 | 619 | | |
594 | 620 | | |
595 | 621 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
2 | 8 | | |
3 | 9 | | |
4 | 10 | | |
| |||
27 | 33 | | |
28 | 34 | | |
29 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
30 | 61 | | |
31 | 62 | | |
32 | 63 | | |
| |||
56 | 87 | | |
57 | 88 | | |
58 | 89 | | |
59 | | - | |
| 90 | + | |
60 | 91 | | |
61 | 92 | | |
62 | 93 | | |
63 | 94 | | |
64 | | - | |
65 | | - | |
66 | | - | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
67 | 102 | | |
68 | 103 | | |
69 | 104 | | |
| |||
98 | 133 | | |
99 | 134 | | |
100 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
101 | 147 | | |
| 148 | + | |
| 149 | + | |
102 | 150 | | |
103 | 151 | | |
104 | | - | |
| 152 | + | |
105 | 153 | | |
106 | 154 | | |
107 | 155 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
3 | 7 | | |
4 | 8 | | |
5 | 9 | | |
| |||
296 | 300 | | |
297 | 301 | | |
298 | 302 | | |
299 | | - | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
300 | 306 | | |
301 | 307 | | |
302 | 308 | | |
| |||
450 | 456 | | |
451 | 457 | | |
452 | 458 | | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
453 | 538 | | |
454 | 539 | | |
455 | 540 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
66 | 131 | | |
67 | 132 | | |
68 | 133 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
109 | 110 | | |
110 | 111 | | |
111 | 112 | | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
112 | 117 | | |
113 | 118 | | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
| 119 | + | |
| 120 | + | |
119 | 121 | | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
120 | 150 | | |
121 | 151 | | |
122 | 152 | | |
| |||
0 commit comments