Skip to content

Commit 44b931a

Browse files
Brooooooklynclaude
andauthored
Stage deploy modules in parallel and explain deploy timeouts (#382)
## Problem A hosted `void deploy` of an app with 421 worker modules failed after about 20 minutes: ``` ■ deploy: Connection lost during deploy: 'fetch failed'. deploy_fetch_rejected attempts=4 err=TypeError: fetch failed -> HeadersTimeoutError: Headers Timeout Error ``` Before it sends response headers, the deploy POST stages every worker module to R2, **one at a time**: ``` CLI ── POST /projects/:id/deploy ──► void-api stage module 1 ─► R2 (~0.7s each) stage module 2 ─► R2 ... stage module 421 ─► R2 admit + queue CLI ◄──────────── headers ─────────── (270–320s later) ``` The production data for the failed run: | Try | Server time before headers | CLI limit | R2 put rate | |---|---|---|---| | 1 | 316s | 300s ✗ | 1.38/s | | 2 | 314s | 300s ✗ | 1.48/s | | 3 | 319s | 300s ✗ | 1.37/s | | 4 | 318s | 300s ✗ | 1.41/s | | rerun next day | 273s | 300s ✓ | 1.59/s | Each try used only about 1s of CPU, so the Worker was waiting on R2. A 15% change in R2 speed decided whether a deploy passed or failed. - **Try 1 actually deployed.** The server went on after the CLI disconnected, but the CLI reported "Deploy failed". - **Tries 2–4 sent the same idempotency key.** Each one staged all 421 modules again, then found try 1's deployment. They failed only because the staging again took more than 300s. ## What this PR changes ``` Server POST /deploy (before headers) before: for each module: arrayBuffer → hash → put (1 put at a time) after: for each module: arrayBuffer → hash → drop buffer then put the original Blobs, 6 in flight (Workers limit for pending R2 calls) 421 modules: ~300s → about 50s (estimate) CLI deploy POST retries: unchanged (4 tries, same idempotency key) message: 'fetch failed' → says what happened and what to do ``` Message when every try hit the headers timeout: ``` deploy: The Void platform did not respond while preparing the deploy. Tried 4 times; the last try waited 5 minutes. It may still finish. Check `void project status`, then run `void deploy` again to retry. If this keeps happening, share the detailed log below. Detailed log: /home/runner/.void/logs/deploy-....jsonl ``` Other fetch failures now show their cause: ``` deploy: Connection lost during deploy: 'fetch failed' (ECONNREFUSED). ``` ## Decisions | Decision | Why | |---|---| | 6 puts in flight | Workers allows at most 6 pending R2 calls. A bigger pool only queues inside the runtime and holds more memory. | | Hash one at a time, then put the Blob | Keeps at most one extra copy of module bytes in memory. A near-cap request already sits close to the 128 MiB isolate limit. | | Stop after the first failed put | Running puts finish before the prefix is deleted, so no object lands after cleanup. The receipt is still written last. | | Keep retrying a headers timeout | The server got the request and keeps working. A retry with the same idempotency key returns the running deployment's receipt, and the CLI follows it to the real result. Failing on the first timeout would report failure for a deploy that may land. | | Keep the 300s timeout | With parallel staging, the server should reply well within it. | ## Rollout ``` deploy platform (void-api) ──► publish CLI │ │ └ fixes the timeout └ clear message if a for current CLIs server is slow again ``` ## Known limits - **A retry still stages everything again before the idempotency check.** If R2 is slow enough to cause one timeout, the retries can time out too. A follow-up can look up the idempotency key before staging. On the failed run, that would have returned try 1's receipt at try 3. - **Staging is still one R2 put per module.** The expected time for 421 modules is about 50s (estimate). Staging as one object would change the queue payload. - **Possible R2 region gap (inferred, not measured).** The R2 bucket is in APAC and `void-api` runs in `aws:us-east-2`, which fits the ~0.7s per put. This is a separate infra change. - **No progress line while the CLI retries.** The client has no access to the spinner, so "Finalizing deploy..." stays on screen. The retries are in the detailed log. ## Testing | Check | Result | |---|---| | New staging unit tests | At most 6 puts in flight, pool stays full, order and digest stable, receipt last, failure stops new puts and cleans up, put gets the original Blob | | CLI tests | Headers timeout retried 4 times with one idempotency key; a retry that gets an existing deployment's receipt follows it to `done`; exact message text for the timeout and for `ENOTFOUND` / `ECONNREFUSED` / `UND_ERR_SOCKET` | | Old code against new tests | Fails | | `vp test run` | Pass | | api `tsc --noEmit`, `void` / `isr` typecheck, `vp lint`, `vp fmt` | Pass | The second commit fixes two flaky platform tests found during verification. The native-form helper now starts Chromium once per file, not twice inside a 5s test. The email rate-limit test now freezes `Date`, so it cannot cross a minute boundary. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent f0bf5bc commit 44b931a

9 files changed

Lines changed: 470 additions & 30 deletions

File tree

‎packages/void/src/cli/client.ts‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,23 @@ export function isExpiredTokenError(error: unknown): boolean {
170170
return false;
171171
}
172172

173+
/**
174+
* True when `fetch` gave up waiting for response headers (undici's
175+
* `HeadersTimeoutError`, 300 seconds by default). The request reached the
176+
* server, which is still working on it.
177+
*/
178+
export function isHeadersTimeoutError(error: unknown): boolean {
179+
let current = error;
180+
for (let depth = 0; depth < 6 && current !== null && typeof current === 'object'; depth++) {
181+
const { code, name } = current as { code?: unknown; name?: unknown };
182+
if (code === 'UND_ERR_HEADERS_TIMEOUT' || name === 'HeadersTimeoutError') {
183+
return true;
184+
}
185+
current = (current as { cause?: unknown }).cause;
186+
}
187+
return false;
188+
}
189+
173190
function isPreflightRejection(error: unknown): boolean {
174191
return (
175192
error instanceof PlatformApiError &&
@@ -577,6 +594,10 @@ export type ManifestOnlyDeployPayload = DeployUploadPayload;
577594
* - Only `fetch` rejections (`TypeError` — no HTTP response) are retried.
578595
* A Response of any status means the server answered; surface it
579596
* verbatim rather than replay a request it may have accepted.
597+
* - A headers timeout is retried too. Every attempt sends the same
598+
* idempotency key, so once an earlier request has created its deployment,
599+
* a receipt-aware platform answers the retry with that deployment's
600+
* receipt and the CLI follows it to the real result.
580601
* - A rejected fetch does NOT prove the request never arrived — the
581602
* connection may have dropped after delivery. Arbitration is left to
582603
* the server's precheck, the only atomic (project-locked) verdict: if
@@ -590,6 +611,11 @@ export type ManifestOnlyDeployPayload = DeployUploadPayload;
590611
*/
591612
const DEPLOY_POST_MAX_RETRIES = 3;
592613
const DEPLOY_POST_BACKOFF_MS = [1000, 5000, 10000];
614+
/**
615+
* Every transport failure is retried, so a `TypeError` thrown by the deploy
616+
* POST always comes after this many attempts.
617+
*/
618+
export const DEPLOY_POST_ATTEMPTS: number = DEPLOY_POST_MAX_RETRIES + 1;
593619
const DEPLOY_OBSERVATION_TIMEOUT_MS = 30_000;
594620
const DEPLOY_OBSERVATION_DEADLINE_MS = 30 * 60_000;
595621
const DEPLOY_OBSERVATION_BACKOFF_MS = [1000, 2000, 5000, 10_000, 30_000];

‎packages/void/src/cli/deploy-errors.ts‎

Lines changed: 54 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1-
import { isCliOutdatedError, parsePlatformErrorBody, PlatformApiError } from './client.ts';
1+
import {
2+
DEPLOY_POST_ATTEMPTS,
3+
isCliOutdatedError,
4+
isHeadersTimeoutError,
5+
parsePlatformErrorBody,
6+
PlatformApiError,
7+
} from './client.ts';
28
import { platformErrorMessage } from './platform-errors.ts';
39

410
const BODY_SNIPPET_MAX = 200;
@@ -27,6 +33,31 @@ function appendTrailingLines(
2733
return appendLogLine(appendDeploymentLine(message, deploymentId), logPath);
2834
}
2935

36+
/**
37+
* Describe the deepest `cause` of an error in one short line, e.g.
38+
* `getaddrinfo ENOTFOUND api.example.com` under `fetch failed`.
39+
*/
40+
function describeRootCause(err: Error): string | null {
41+
let root: unknown = err;
42+
for (let depth = 0; depth < 6; depth++) {
43+
const cause = (root as { cause?: unknown }).cause;
44+
if (cause === null || typeof cause !== 'object') {
45+
break;
46+
}
47+
root = cause;
48+
}
49+
if (root === err) {
50+
return null;
51+
}
52+
const { code, message } = root as { code?: unknown; message?: unknown };
53+
const text = typeof message === 'string' ? message.trim() : '';
54+
const codeText = typeof code === 'string' ? code : '';
55+
if (!text) {
56+
return codeText || null;
57+
}
58+
return codeText && !text.includes(codeText) ? `${codeText}: ${text}` : text;
59+
}
60+
3061
/**
3162
* Format a server-emitted `event: 'error'` payload with an optional deployment id.
3263
*/
@@ -56,14 +87,18 @@ export function formatConnectionLostMessage(
5687
* Convert an exception thrown during the deploy NDJSON stream into a
5788
* user-facing `Error` with a readable message.
5889
*
59-
* Three cases:
90+
* Four cases:
6091
* 1. `PlatformApiError` with 409 + a JSON body carrying `{error, deployment}`
6192
* — surface the server-supplied error and prefer the body's deployment id.
6293
* 2. Any other `PlatformApiError` — retain the client's actionable message and
6394
* structured error fields; opaque provider failures use a trimmed body snippet.
64-
* 3. Anything else — preserve the original message under a `Connection lost
65-
* during deploy: <original>` prefix so the caller can still tell network,
66-
* parse, and transport errors apart.
95+
* 3. A headers timeout on the last deploy POST attempt — the platform sent
96+
* no response in time. Explain that the deploy may still finish and how
97+
* to check and retry.
98+
* 4. Anything else — preserve the original message, plus its root cause when
99+
* there is one, under a `Connection lost during deploy: <original>`
100+
* prefix so the caller can still tell network, parse, and transport
101+
* errors apart.
67102
*/
68103
export function formatStreamDeployError(
69104
err: unknown,
@@ -98,10 +133,23 @@ export function formatStreamDeployError(
98133
return err;
99134
}
100135

136+
if (isHeadersTimeoutError(err)) {
137+
const lines = [
138+
`deploy: The Void platform did not respond while preparing the deploy. Tried ${DEPLOY_POST_ATTEMPTS} times; the last try waited 5 minutes.`,
139+
'It may still finish. Check `void project status`, then run `void deploy` again to retry.',
140+
];
141+
if (logPath) {
142+
lines.push('If this keeps happening, share the detailed log below.');
143+
}
144+
return new Error(appendTrailingLines(lines.join('\n'), deploymentId, logPath));
145+
}
146+
101147
const original = err instanceof Error ? err.message : String(err);
148+
const rootCause = err instanceof Error ? describeRootCause(err) : null;
149+
const detail = rootCause && rootCause !== original ? ` (${rootCause})` : '';
102150
return new Error(
103151
appendTrailingLines(
104-
`deploy: Connection lost during deploy: '${original}'.`,
152+
`deploy: Connection lost during deploy: '${original}'${detail}.`,
105153
deploymentId,
106154
logPath,
107155
),

‎packages/void/test/unit/direct-r2-client.test.ts‎

Lines changed: 87 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
import { afterEach, beforeEach, describe, expect, it, vi } from 'vite-plus/test';
2-
import { PlatformClient, type ManifestOnlyDeployPayload } from '../../src/cli/client.ts';
2+
import {
3+
DEPLOY_POST_ATTEMPTS,
4+
PlatformClient,
5+
type ManifestOnlyDeployPayload,
6+
} from '../../src/cli/client.ts';
37
import type { DeployManifest } from '../../src/cli/types.ts';
48
import { withPlatformDiscovery } from '../helpers/platform-discovery.ts';
59

@@ -296,7 +300,9 @@ describe('PlatformClient.finalizeDeploy', () => {
296300
posts++;
297301
if (posts === 1) {
298302
return Promise.reject(
299-
new TypeError('fetch failed', { cause: new Error('read ECONNRESET') }),
303+
new TypeError('fetch failed', {
304+
cause: Object.assign(new Error('read ECONNRESET'), { code: 'ECONNRESET' }),
305+
}),
300306
);
301307
}
302308
return Promise.resolve(
@@ -450,6 +456,85 @@ describe('PlatformClient.finalizeDeploy', () => {
450456
expect(fetchMock).toHaveBeenCalledTimes(1);
451457
});
452458

459+
function headersTimeoutError() {
460+
const cause = Object.assign(new Error('Headers Timeout Error'), {
461+
name: 'HeadersTimeoutError',
462+
code: 'UND_ERR_HEADERS_TIMEOUT',
463+
});
464+
return { cause, error: new TypeError('fetch failed', { cause }) };
465+
}
466+
467+
const idempotencyKeys = () =>
468+
deployPostCalls().map(
469+
([, init]) => (init.headers as Record<string, string>)['x-void-deploy-idempotency-key'],
470+
);
471+
472+
it('retries a headers timeout with the same idempotency key, then rethrows it', async () => {
473+
const { cause, error } = headersTimeoutError();
474+
fetchMock.mockRejectedValue(error);
475+
476+
vi.useFakeTimers();
477+
const cliLog = makeCliLog();
478+
const client = new PlatformClient(token, { apiUrl, cliLog });
479+
try {
480+
const assertion = expect(
481+
client.finalizeDeploy(projectId, makePayload()).next(),
482+
).rejects.toMatchObject({ message: 'fetch failed', cause });
483+
await vi.runAllTimersAsync();
484+
await assertion;
485+
486+
expect(deployPostCalls()).toHaveLength(4);
487+
expect(DEPLOY_POST_ATTEMPTS).toBe(4);
488+
expect(new Set(idempotencyKeys()).size).toBe(1);
489+
const retries = cliLog.warn.mock.calls.filter(([msg]) => msg === 'deploy_fetch_retry');
490+
expect(retries.map(([, fields]) => fields?.delayMs)).toEqual([1000, 5000, 10000]);
491+
expect(cliLog.error).toHaveBeenCalledWith(
492+
'deploy_fetch_rejected',
493+
expect.any(TypeError),
494+
expect.objectContaining({ attempts: 4 }),
495+
);
496+
} finally {
497+
vi.useRealTimers();
498+
}
499+
});
500+
501+
it('follows the existing deployment receipt returned to a retry after a headers timeout', async () => {
502+
const done = {
503+
event: 'done',
504+
deployment: 'dpl_existing',
505+
url: 'https://x.void.app',
506+
urls: [],
507+
assets: 1,
508+
workers: 1,
509+
seq: 3,
510+
};
511+
fetchMock
512+
.mockRejectedValueOnce(headersTimeoutError().error)
513+
.mockResolvedValueOnce(
514+
ndjsonStreamResponse([
515+
{ event: 'start', deploymentId: 'dpl_existing', seq: 1 },
516+
{ event: 'worker_upload', seq: 2 },
517+
done,
518+
]),
519+
);
520+
521+
vi.useFakeTimers();
522+
try {
523+
const deploy = collectDeploy();
524+
await vi.runAllTimersAsync();
525+
expect(await deploy).toEqual([
526+
{ event: 'start', deploymentId: 'dpl_existing' },
527+
{ event: 'worker_upload', seq: 2 },
528+
done,
529+
]);
530+
expect(deployPostCalls()).toHaveLength(2);
531+
const [first, second] = idempotencyKeys();
532+
expect(second).toBe(first);
533+
} finally {
534+
vi.useRealTimers();
535+
}
536+
});
537+
453538
it('does not retry HTTP error responses — the server answered', async () => {
454539
fetchMock.mockResolvedValue(new Response('busted', { status: 503 }));
455540
const client = makeClient();

‎packages/void/test/unit/stream-deploy-errors.test.ts‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,71 @@ describe('formatStreamDeployError', () => {
6363
expect(formatted.message).toContain('fetch failed');
6464
});
6565

66+
it('explains a headers timeout after the last retry in plain words and keeps the log line', () => {
67+
const err = new TypeError('fetch failed', {
68+
cause: Object.assign(new Error('Headers Timeout Error'), {
69+
name: 'HeadersTimeoutError',
70+
code: 'UND_ERR_HEADERS_TIMEOUT',
71+
}),
72+
});
73+
74+
const formatted = formatStreamDeployError(err, null, '/tmp/deploy.jsonl');
75+
76+
expect(formatted.message).toBe(
77+
[
78+
'deploy: The Void platform did not respond while preparing the deploy. Tried 4 times; the last try waited 5 minutes.',
79+
'It may still finish. Check `void project status`, then run `void deploy` again to retry.',
80+
'If this keeps happening, share the detailed log below.',
81+
'Detailed log: /tmp/deploy.jsonl',
82+
].join('\n'),
83+
);
84+
});
85+
86+
it('omits the log hint for a headers timeout when no log was written', () => {
87+
const err = new TypeError('fetch failed', {
88+
cause: Object.assign(new Error('Headers Timeout Error'), {
89+
code: 'UND_ERR_HEADERS_TIMEOUT',
90+
}),
91+
});
92+
93+
const formatted = formatStreamDeployError(err, null, null);
94+
95+
expect(formatted.message).not.toContain('log');
96+
expect(formatted.message).toContain('void project status');
97+
});
98+
99+
it('adds the root cause of a fetch failure', () => {
100+
const err = new TypeError('fetch failed', {
101+
cause: Object.assign(new Error('getaddrinfo ENOTFOUND api.example.com'), {
102+
code: 'ENOTFOUND',
103+
}),
104+
});
105+
106+
const formatted = formatStreamDeployError(err, null, '/tmp/deploy.jsonl');
107+
108+
expect(formatted.message).toBe(
109+
"deploy: Connection lost during deploy: 'fetch failed' (getaddrinfo ENOTFOUND api.example.com).\n" +
110+
'Detailed log: /tmp/deploy.jsonl',
111+
);
112+
});
113+
114+
it('uses the cause code when the root cause has no message', () => {
115+
const refused = Object.assign(new AggregateError([], ''), { code: 'ECONNREFUSED' });
116+
const closed = Object.assign(new Error('other side closed'), {
117+
name: 'SocketError',
118+
code: 'UND_ERR_SOCKET',
119+
});
120+
121+
expect(
122+
formatStreamDeployError(new TypeError('fetch failed', { cause: refused }), null).message,
123+
).toBe("deploy: Connection lost during deploy: 'fetch failed' (ECONNREFUSED).");
124+
expect(
125+
formatStreamDeployError(new TypeError('fetch failed', { cause: closed }), null).message,
126+
).toBe(
127+
"deploy: Connection lost during deploy: 'fetch failed' (UND_ERR_SOCKET: other side closed).",
128+
);
129+
});
130+
66131
it('includes the deployment id for generic mid-stream errors when known', () => {
67132
const err = new Error('reader canceled');
68133

‎platform/packages/api/src/deployment-execution.ts‎

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { and, eq, getTableColumns, lte, ne, or, sql } from 'drizzle-orm';
22
import { paginatedR2Delete } from './assets.ts';
3+
import { R2_CONCURRENCY } from './concurrency.ts';
34
import { createDb, type Db, type DbOrTx } from './db.ts';
45
import type { DeploymentInput, DeploymentOperation } from './deployment-operation-types.ts';
56
import type { ApiEnv } from './env.ts';
@@ -109,14 +110,43 @@ export async function prepareDeploymentInput(
109110
};
110111
delete input.body.workerFiles;
111112
try {
113+
// Hash one module at a time so at most one extra copy of module bytes is
114+
// alive beside the decoded Blobs; a near-cap request already sits close
115+
// to the isolate memory limit.
116+
const modules: DeploymentInput['modules'] = [];
112117
for (let index = 0; index < files.length; index++) {
113118
const { name, content } = files[index]!;
114-
const bytes = await content.arrayBuffer();
115-
const digest = await sha256Hex(bytes);
116-
const key = `${prefix}${index}`;
117-
await r2.put(key, bytes);
118-
input.modules.push({ name, key, digest, type: content.type });
119+
const digest = await sha256Hex(await content.arrayBuffer());
120+
modules.push({ name, key: `${prefix}${index}`, digest, type: content.type });
119121
}
122+
// The request has not answered yet, so upload the Blobs themselves in
123+
// parallel. Workers let at most six R2 calls wait for a response at once
124+
// (R2_CONCURRENCY); a larger pool would only queue inside the runtime.
125+
// After the first failure no new put starts, and every started put
126+
// settles before cleanup so none lands under the prefix after it was
127+
// deleted.
128+
let next = 0;
129+
let failed = false;
130+
const workers = await Promise.allSettled(
131+
Array.from({ length: Math.min(R2_CONCURRENCY, files.length) }, async () => {
132+
while (!failed && next < files.length) {
133+
const index = next++;
134+
try {
135+
await r2.put(modules[index]!.key, files[index]!.content);
136+
} catch (error) {
137+
failed = true;
138+
throw error;
139+
}
140+
}
141+
}),
142+
);
143+
const rejected = workers.find(
144+
(result): result is PromiseRejectedResult => result.status === 'rejected',
145+
);
146+
if (rejected) {
147+
throw rejected.reason;
148+
}
149+
input.modules = modules;
120150
const identity = JSON.stringify({
121151
body: input.body,
122152
modules: input.modules.map(({ name, digest, type }) => ({ name, digest, type })),

0 commit comments

Comments
 (0)