Skip to content

feat(deploy): stage streamed payloads to a temp file for replication#536

Draft
kriszyp wants to merge 1 commit into
feat/deploy-component-progressfrom
feat/deploy-component-payload-staging
Draft

feat(deploy): stage streamed payloads to a temp file for replication#536
kriszyp wants to merge 1 commit into
feat/deploy-component-progressfrom
feat/deploy-component-payload-staging

Conversation

@kriszyp
Copy link
Copy Markdown
Member

@kriszyp kriszyp commented May 14, 2026

Stacked on #531 — please review #530 and #531 first; this PR will be retargeted to `main` after both land.

Summary

Origin-side payload staging needed for the direct-HTTPS peer relay (harper-pro slice 3b). When a `deploy_component` request will be replicated to peers, the origin now writes the streamed payload to a temp file before local extraction so it can re-stream the same bytes to each peer afterwards. Without this, the payload Readable is consumed once by local extract and gone by the time replication runs.

Third slice of #524.

Where to look

  • `components/payloadStaging.ts` — single-purpose utility that pipelines a Readable into a temp `payload.tar.gz` and returns a cleanup callback. The trade-off comment in the file header explains why we write-then-read instead of teeing in flight (simpler code, identical backpressure for extraction; disk speed isn't the bottleneck).
  • `components/operations.js` — staging gate in `deployComponent`. Skipped when:
    • the deploy is package-identifier-based (`req.package`, no `payload` stream),
    • `req.replicated === false` (the operation is itself a replicated deploy that the peer received, so no further fan-out),
    • or there are no peers (`server.nodes.length === 0` — single-node deploys keep their zero-disk-copy property).
  • Cleanup runs in a `finally` after `server.replication.replicateOperation(req)`, plus a defensive cleanup if `prepareApplication` fails before replication.

Backward compatibility

  • The staging is invisible to the operations API contract: `req.payload` is replaced internally with a `createReadStream` over the temp file before the `Application` is constructed, so every downstream code path (`Application.extractApplication`, the SSE phase emitter, anything reading `application.payload`) keeps seeing a `Readable` exactly as before.
  • Single-node and package-identifier deploys: zero change in behavior, no temp file written.

Test plan

  • `npx mocha unitTests/components/payloadStaging.test.js` — 5/5 pass (round-trip contents, cleanup removes file+dir, double-cleanup is safe, path-traversal sanitization keeps the staged dir under tmpdir, source-stream errors propagate).
  • `npx mocha unitTests/server/fastifyRoutes/operations.test.js` — 21/21 pass (existing deployComponent tests; the staging branch is only entered when nodes are present, which these tests don't simulate).
  • `npx mocha unitTests/components/packageComponent.test.js unitTests/server/serverHelpers/multipartParser.test.js unitTests/bin/multipartBuilder.test.js unitTests/server/serverHelpers/progressEmitter.test.js unitTests/bin/sseConsumer.test.js` — 32/32 pass (slice 1 + slice 2 unaffected).
  • End-to-end replicated deploy — exercised in harper-pro's slice 3b integration test, not here.

🤖 Generated with Claude Code

When a deploy_component is replicated to peers, the origin needs to keep
a copy of the payload so it can re-stream it to each peer. The payload
Readable is consumed once by local extraction; without staging, peer
replication would have nothing to send.

deployComponent now stages the payload to a temp file before extraction
when there are nodes to replicate to and the payload is a Readable
(non-package, streaming deploys). Local extraction re-sources from the
staged file, which keeps backpressure behavior identical to the
non-replicated case. The temp file is cleaned up in a finally block
after replication completes (success, failure, or skipped).

Local-only deploys keep their zero-disk-copy property; package-identifier
deploys are unaffected.

This is the core-side staging needed for harper-pro's direct-HTTPS deploy
relay (slice 3b, addressing #524).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kriszyp kriszyp requested a review from a team as a code owner May 14, 2026 15:15
@kriszyp kriszyp marked this pull request as draft May 14, 2026 15:18
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 14, 2026

Reviewed; no blockers found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant