Repository navigation
Cross-build from a bullseye snapshot: the live -security pool is gone - #195
Conversation
The staging deploy for #194's merge (Deploy main run 35719911932) failed in build.sh's Docker stage at apt-get install, before any compilation: thirteen packages apt chose from bullseye-security -- perl, libcurl4, libexpat1, libicu67, libxml2, dpkg-dev and friends -- answered 404 from the pool. The suite's index still advertises those versions (perl 5.32.1-4+deb11u5), but bullseye left LTS on 2026-08-31 and its security pool has since been pruned underneath the index, so every apt-get that consults it fails the same way. It is not this change's doing: the same stage passed on 2026-09-03 and the merge touched no Dockerfile, Cargo or apt input. Staying on bullseye is deliberate and documented (glibc 2.31 so the binary runs on Amazon Linux 2023's 2.34; bookworm's 2.36 would not), so the fix is to stop consulting the dead suite: the stage now writes a sources.list with the bullseye main suite alone before apt-get update. main is intact and carries every package this stage installs (verified against its current arm64 index and pool: perl 5.32.1-4+deb11u3, libcurl4 7.74.0-1.3+deb11u13, cmake 3.18.4-2+deb11u1, clang 1:11.0-51+nmu5, ...). The comment records the next step for when deb.debian.org drops bullseye entirely: archive.debian.org already mirrors the main suite. The Docker daemon is not available where this was prepared, so the stage was not rebuilt locally; the next push to main exercises it end to end in the staging deploy, which is exactly the run that failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
|
CI note:
No fix to port: the failure is the external origin's reachability from the runner. I've re-run the failed job once. If it fails the same way again, the test's dependence on a live third-party host is the thing to make robust — but that's Generated by Claude Code |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Preinstalled security-version packages can conflict with older exact-version dependencies available from main.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
What changed in this PR
Updates the Bullseye cross-build environment to avoid the removed security package pool.
Changes:
- Replaces APT sources with Bullseye
main. - Documents the eventual archive migration.
| File | Description |
|---|---|
deploy/native/build.sh |
Configures the inline build container’s APT source. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
… alone Copilot's review of the previous commit was right: restricting apt to bullseye main is unsatisfiable against this image. rust:1-slim-bullseye (built 2026-08-25) has its base packages at the final security versions -- perl-base 5.32.1-4+deb11u5, libc6 2.31-13+deb11u14, dpkg 1.20.14 and 25 more -- while main carries perl 5.32.1-4+deb11u3, which depends on perl-base (= 5.32.1-4+deb11u3), and libc6-dev 2.31-13+deb11u11, which depends on libc6 (= 2.31-13+deb11u11). apt will not downgrade the installed perl-base or libc6 to satisfy those, so the install fails. Verified by reading /var/lib/dpkg/status out of the image's base layer from the registry and comparing it with main's arm64 index. Install from snapshot.debian.org at 2026-08-30 instead, for both suites: the last state in which bullseye and bullseye-security were complete, after the image was built. Its security index carries exactly the versions the image has (perl-base +deb11u5, libc6 and libc6-dev +deb11u14, dpkg and dpkg-dev 1.20.14) and all thirteen packages that 404 on deb.debian.org (perl, libcurl4 7.74.0-1.3+deb11u16, libexpat1, libicu67, libxml2, libglib2.0-0, ...), and its pool serves them. Both suites are frozen, so the pin is the permanent final state, reproducible, with nothing to downgrade. check-valid-until=no because the snapshot's Release files expired on 2026-09-05; Acquire::Retries=3 for snapshot's throttling. Still not rebuilt locally (no Docker daemon here); the staging deploy on merge remains the end-to-end proof. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The PR metadata contradicts the implemented APT configuration, and the reproducibility claim is inaccurate.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 2
Open (2)
Resolved since last review (1)
The previous commit's comment said the snapshot made "the build reproducible". It does not: rust:1-slim-bullseye is a mutable tag and BUILD_TIME is stamped at build time, so the binary and its metadata still vary from build to build. What the dated snapshot fixes is apt's inputs, so the comment now says that package resolution is reproducible and names the two things that are not. Review nit from Copilot on 829622b. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Plain HTTP combined with disabled expiration permits replay of older signed package indexes.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (1)
Resolved since last review (2)
With check-valid-until=no, apt no longer rejects an index past its Valid-Until, so its remaining defence against a network attacker replaying an older, still validly signed snapshot index -- and thereby an older package set than the timestamp names -- is the transport. Fetch both snapshot lines over https so the dated path is authenticated in transit. The rust:1-slim-bullseye image installs ca-certificates in its own layer (alongside gcc, libc6-dev and wget), so https works before this stage installs anything; snapshot.debian.org serves https end to end, redirecting to same-origin content-hash paths. Review finding from Copilot on 69506cb. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
Resolves the conflict in deploy/native/deploy.sh between this branch's OAUTH_CIMD_ENABLED substitution on the unit_mcp line and #194's SERVE_STATUS block around the caddyfile line: both are kept, the CIMD comment and substitution first, then the status block as main has it. The other files main touched (#172, #194, #195) merged cleanly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xd7VT72Qt16qiAJynu9EKj


Summary
The staging deploy for #194's merge (Deploy main run 35719911932) failed in
build.sh's Docker stage atapt-get install, before any compilation. Thirteen packages apt chose frombullseye-security—perl,libcurl4,libexpat1,libicu67,libxml2,dpkg-devand friends — answered 404 from the pool. bullseye left LTS on 2026-08-31 and deb.debian.org has since pruned its security pool underneath an index that still advertises those versions, so everyapt-getthat consults the live suite fails the same way. Theshipjob never ran.Not #194's doing: the same stage passed on 2026-09-03, and that merge touched no Dockerfile, Cargo or apt input.
Staying on bullseye is deliberate and documented (
deploy/native/README.md§ Why cross-build against bullseye: glibc 2.31 so the binary runs on Amazon Linux 2023's 2.34; bookworm's 2.36 would not).Resolving from
bullseye/mainalone does not work either (Copilot's finding on the first revision, confirmed): the currentrust:1-slim-bullseye(built 2026-08-25) has its base packages at the final security versions —perl-base 5.32.1-4+deb11u5,libc6/libc-bin 2.31-13+deb11u14,dpkg 1.20.14, 28 packages in all — while main'sperl 5.32.1-4+deb11u3depends onperl-base (= …+deb11u3)andlibc6-dev 2.31-13+deb11u11onlibc6 (= …+deb11u11). apt will not downgrade the installed packages to satisfy those.So the stage installs from
snapshot.debian.orgat20260830T000000Z, for bothbullseyeandbullseye-security: the last state in which both suites were complete, after the image was built. Its security index carries exactly the image's versions and all thirteen packages that 404 today, and its pool serves them. Both suites are frozen, so the pin is the permanent final state of bullseye: package resolution is deterministic and nothing is downgraded. (The build as a whole is not bit-reproducible —rust:1-slim-bullseyeis a mutable tag andBUILD_TIMEis stamped per build; the snapshot fixes apt's inputs only.)check-valid-until=nobecause the snapshot'sReleasefiles expired on 2026-09-05;Acquire::Retries=3for snapshot's occasional throttling. Both lines use https: withValid-Untilunchecked, transport authentication is what stops a network attacker replaying an older, still-signed index (Copilot's finding on the second revision); the rust image preinstallsca-certificatesin its own layer, so https works before this stage installs anything.Follow-up worth its own issue: building on
amazonlinux:2023would match the host's glibc exactly and remove the dependency on an end-of-life Debian suite altogether. Out of scope here — this is the minimal unblock.Related issues
Unblocks the staging deploy after #194. Follow-up to #194.
Changes
deploy/native/build.sh(inline Dockerfile): beforeapt-get update, write asources.listpointing both suites, over https, atsnapshot.debian.org/archive/{debian,debian-security}/20260830T000000Zwith[check-valid-until=no], clearsources.list.d, and run apt withAcquire::Retries=3. The comment records why main alone is unsatisfiable, why https, what the snapshot does and does not make reproducible, and what the durable fix is.Testing
bash -n deploy/native/build.sh/var/lib/dpkg/statusfrom the currentrust:1-slim-bullseyearm64 image via the registry — both the Debian base layer and the rust toolchain layer: 28 base packages are at security-only versions (perl-base +deb11u5,libc6 +deb11u14,dpkg 1.20.14, …); main'sperlandlibc6-devcarry exact-version dependencies on older versions of two of them — so the first revision (main only) would not have resolved. The toolchain layer preinstallsca-certificates 20250419~deb12u1~deb11u1,gcc,libc6-dev 2.31-13+deb11u14andwget(its history records theapt-get install ca-certificates gcc libc6-dev wgetline), so https apt works from the firstapt-get update20260830T000000Zsnapshot'sbullseye-securityarm64 index lists exactly the image's versions (perl-base +deb11u5,libc6/libc6-dev +deb11u14,dpkg/dpkg-dev 1.20.14) and every package that 404s on deb.debian.org (libcurl4 7.74.0-1.3+deb11u16,libexpat1,libicu67,libxml2,libglib2.0-0,libnghttp2-14,libarchive13,xz-utils, …); its pool serves them (perl_5.32.1-4+deb11u5_arm64.deb→ 200); https is served end to end for both suites'InRelease(302 to a same-origin content-hash path, then 200);ReleaseValid-Untilwas 2026-09-05, hencecheck-valid-until=no.github/scripts/scan-internal-identifiers.sh origin/main...HEAD— cleanbuild.sheither (it runs only from the deploy workflows), so the end-to-end proof is the staging deploy triggered by merging this — exactly the run that failed.cargo build/cargo test/cargo fmt/cargo clippy— n/a, no Rust changednpm test --prefix monitoring/mcp-status— n/a, dashboard unchangedChecklist
🤖 Generated with Claude Code
https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw