[polylane]

The switchboard image build installed the coding harness with one npm command that failed whenever the package registry briefly did not serve a dependency tarball it had just advertised. Each such reply aborted the build, so staging deploys stalled and the production deploy waited behind them. The install now retries three times before it gives up, and a build that really has a bad pin still fails.

flowchart TD
  A["deploy workflow: deploy the captured commit to staging"] --> B["docker build: pi install layer"]
  B --> C["npm install -g, no lockfile"]
  C --> D["npm resolves the tree against the registry's live view"]
  D --> E["registry advertises bowser 2.15.0, credential-provider-process 3.972.76"]
  E --> F["tarball GET returns 404"]
  F --> G["npm does not retry a 404"]
  G --> H["layer exits 1, BuildKit fails the build"]
  H --> I["staging deploy blocked, production gated"]
  C -.->|"this change: three more tries, fifteen seconds apart"| D
  F -.->|"this change: retry before the build fails"| G

Loading

Detection evidence

Affected: int_14cf586860018m832vjyg5di

Why this fix

Two deploy builds stopped at the same install layer, five days apart, on a 404 for a tarball the registry was advertising: bowser 2.15.0 on 2026-10-26 and @aws-sdk/credential-provider-process 3.972.76 on 2026-10-21. Both tarballs answer HTTP 200 now, and neither version is the one this commit pins, so the reply was a window in a tree npm resolved at build time. npm never retries a 404, so the layer had no tolerance at all. The retry covers that window, and the pin proofs after it still end a build whose install never landed. The layer still resolves the tree from the registry's live view, which remains open on the issue.

This change is a declared stopgap: it contains the symptom while the producer's fix is tracked as a follow-up on the issue timeline.

Production impact: Pi install retry leaves image contents and deploy behaviour unchanged

The diff changes one build-time install step in three Dockerfiles: on a successful install the image is identical (same eight exact pins, same two proofs, same layer order), and the retry only executes after an install has already failed. The three workers these images deploy to were checked over 48 hours: switchboard 371 error-level lines, switchboard-sandbox 5,079, switchboard-resident 5, all from causes unrelated to this layer. The parser change is read-only and only widens what the pin fence sees.

Checked and not affected: switchboard, switchboard-sandbox, switchboard-resident.

Observability

• The build log of the pi install layer is the signal: a failed attempt prints npm's E404 line and the next attempt follows it, so three attempts are visible before a build fails.
• No runtime telemetry changes: the pins, the proofs and the layer's position are untouched.

Polylane assessed this pull request at 8e198f3.

Fix walkthrough

Watch the fix explained (1:20). Recorded for commit 8e198f317b35eb7f556fcddf8533eb32dccb79e5.

Causal chain

• Signal (log template): npm error code E404 — 404 Not Found - GET https://registry.npmjs.org//-/-.tgz - Not found
• Surfacing site: Depot job logs of coreplanelabs/switchboard deploy workflows (integration int_14cf586860018m832vjyg5di) at coreplanelabs/switchboard:Dockerfile#pi-install-layer, identical in deploy/cloudflare-resident/Dockerfile and deploy/cloudflare-sandbox/Dockerfile
• Mechanism: The layer runs npm install -g with no lockfile, so npm re-resolves pi's whole dependency tree against the registry's live view at build time and requests versions newer than the commit's pins (bowser 2.15.0 against a locked 2.14.1; @aws-sdk/credential-provider-process 3.972.76 against a locked 3.972.71). The registry advertised those versions but did not serve their tarballs, and npm never retries a 404, so the layer exited 1 and BuildKit aborted the build, failing the deploy step.
• Producer: switchboard image build — the pi install layer in all three images, at coreplanelabs/switchboard:Dockerfile#pi-install-layer
• Trigger: The registry publishing a version whose tarball it does not serve yet; both failing tarballs answer HTTP 200 now and both versions are still the registry's newest.
• What happens to the failed unit today: The layer deliberately fails the build on a wrong pin, so the build is the proof. npm's fetch retries cover connection errors and 5xx but not 404, so the layer's only tolerance today is none.
• Cadence check: Two build aborts in the 7 days to 2026-10-28, on different jobs, images and packages, 5 days apart.
• Blast radius: 2 other resource(s), 0 other tenant(s); data at risk: none named
• Producer evidence:
  • The layer had no retry, and npm's own fetch retries exclude 404, so one registry reply ended the build.
  • Both failing versions are absent from the commit's lockfile (bowser 2.14.1, @aws-sdk/credential-provider-process 3.972.71 are pinned), so the build resolved them itself at build time.
  • The identical layer is in the bot, resident and sandbox images, and the deploy runner builds each from its Dockerfile in build mode.

7 files changed (+60/-9)

• Dockerfile: modified, +10/-2
• deploy/cloudflare-resident/Dockerfile: modified, +10/-2
• deploy/cloudflare-sandbox/Dockerfile: modified, +10/-2
• docs/reference/specs/harness-pi.md: modified, +2/-2
• src/deploy/imagePiHarness.test.ts: modified, +13/-0
• src/deploy/imagePins.test.ts: modified, +7/-0
• src/deploy/imagePins.ts: modified, +8/-1

---

Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment.

[polylane]

This pull request is waiting on a decision: merge it if the change is still wanted, or close it if the fix is no longer needed. Either one settles it.

It was opened on 2026-10-28 and has had no new activity for a week. If nothing happens on it within a week, Polylane closes it and keeps the branch, so reopening it brings the change back exactly as it stands.

@justinhelmer and @boristane, the change touches what you own, so the review is with you.