Skip to content

Cross-build from a bullseye snapshot: the live -security pool is gone - #195

Merged
aterga merged 4 commits into
mainfrom
claude/connector-marketplace-publish-p4el4n
Sep 22, 2026
Merged

aterga merged 4 commits into
mainfrom
claude/connector-marketplace-publish-p4el4n

Conversation

@aterga

@aterga aterga commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The staging deploy for #194's merge (Deploy main run 35719911932) failed in build.sh's Docker stage at apt-get install, before any compilation. Thirteen packages apt chose from bullseye-security — perl, libcurl4, libexpat1, libicu67, libxml2, dpkg-dev and friends — answered 404 from the pool. bullseye left LTS on 2026-08-31 and deb.debian.org has since pruned its security pool underneath an index that still advertises those versions, so every apt-get that consults the live suite fails the same way. The ship job never ran.

Not #194's doing: the same stage passed on 2026-09-03, and that merge touched no Dockerfile, Cargo or apt input.

Staying on bullseye is deliberate and documented (deploy/native/README.md § Why cross-build against bullseye: glibc 2.31 so the binary runs on Amazon Linux 2023's 2.34; bookworm's 2.36 would not).

Resolving from bullseye/main alone does not work either (Copilot's finding on the first revision, confirmed): the current rust:1-slim-bullseye (built 2026-08-25) has its base packages at the final security versions — perl-base 5.32.1-4+deb11u5, libc6/libc-bin 2.31-13+deb11u14, dpkg 1.20.14, 28 packages in all — while main's perl 5.32.1-4+deb11u3 depends on perl-base (= …+deb11u3) and libc6-dev 2.31-13+deb11u11 on libc6 (= …+deb11u11). apt will not downgrade the installed packages to satisfy those.

So the stage installs from snapshot.debian.org at 20260830T000000Z, for both bullseye and bullseye-security: the last state in which both suites were complete, after the image was built. Its security index carries exactly the image's versions and all thirteen packages that 404 today, and its pool serves them. Both suites are frozen, so the pin is the permanent final state of bullseye: package resolution is deterministic and nothing is downgraded. (The build as a whole is not bit-reproducible — rust:1-slim-bullseye is a mutable tag and BUILD_TIME is stamped per build; the snapshot fixes apt's inputs only.) check-valid-until=no because the snapshot's Release files expired on 2026-09-05; Acquire::Retries=3 for snapshot's occasional throttling. Both lines use https: with Valid-Until unchecked, transport authentication is what stops a network attacker replaying an older, still-signed index (Copilot's finding on the second revision); the rust image preinstalls ca-certificates in its own layer, so https works before this stage installs anything.

Follow-up worth its own issue: building on amazonlinux:2023 would match the host's glibc exactly and remove the dependency on an end-of-life Debian suite altogether. Out of scope here — this is the minimal unblock.

Related issues

Unblocks the staging deploy after #194. Follow-up to #194.

Changes

  • deploy/native/build.sh (inline Dockerfile): before apt-get update, write a sources.list pointing both suites, over https, at snapshot.debian.org/archive/{debian,debian-security}/20260830T000000Z with [check-valid-until=no], clear sources.list.d, and run apt with Acquire::Retries=3. The comment records why main alone is unsatisfiable, why https, what the snapshot does and does not make reproducible, and what the durable fix is.

Testing

  • bash -n deploy/native/build.sh
  • Read /var/lib/dpkg/status from the current rust:1-slim-bullseye arm64 image via the registry — both the Debian base layer and the rust toolchain layer: 28 base packages are at security-only versions (perl-base +deb11u5, libc6 +deb11u14, dpkg 1.20.14, …); main's perl and libc6-dev carry exact-version dependencies on older versions of two of them — so the first revision (main only) would not have resolved. The toolchain layer preinstalls ca-certificates 20250419~deb12u1~deb11u1, gcc, libc6-dev 2.31-13+deb11u14 and wget (its history records the apt-get install ca-certificates gcc libc6-dev wget line), so https apt works from the first apt-get update
  • Verified the 20260830T000000Z snapshot's bullseye-security arm64 index lists exactly the image's versions (perl-base +deb11u5, libc6/libc6-dev +deb11u14, dpkg/dpkg-dev 1.20.14) and every package that 404s on deb.debian.org (libcurl4 7.74.0-1.3+deb11u16, libexpat1, libicu67, libxml2, libglib2.0-0, libnghttp2-14, libarchive13, xz-utils, …); its pool serves them (perl_5.32.1-4+deb11u5_arm64.deb → 200); https is served end to end for both suites' InRelease (302 to a same-origin content-hash path, then 200); Release Valid-Until was 2026-09-05, hence check-valid-until=no
  • .github/scripts/scan-internal-identifiers.sh origin/main...HEAD — clean
  • Docker rebuild of the stage — not run: no Docker daemon where this was prepared. PR CI does not exercise build.sh either (it runs only from the deploy workflows), so the end-to-end proof is the staging deploy triggered by merging this — exactly the run that failed.
  • cargo build / cargo test / cargo fmt / cargo clippy — n/a, no Rust changed
  • npm test --prefix monitoring/mcp-status — n/a, dashboard unchanged

Checklist

  • I have read the Contributing guidelines.
  • Docs (README / comments) updated for any user-visible change.
  • No secrets, credentials, or internal-only information are included.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw

The staging deploy for #194's merge (Deploy main run 35719911932) failed in
build.sh's Docker stage at apt-get install, before any compilation: thirteen
packages apt chose from bullseye-security -- perl, libcurl4, libexpat1,
libicu67, libxml2, dpkg-dev and friends -- answered 404 from the pool. The
suite's index still advertises those versions (perl 5.32.1-4+deb11u5), but
bullseye left LTS on 2026-08-31 and its security pool has since been pruned
underneath the index, so every apt-get that consults it fails the same way.
It is not this change's doing: the same stage passed on 2026-09-03 and the
merge touched no Dockerfile, Cargo or apt input.

Staying on bullseye is deliberate and documented (glibc 2.31 so the binary
runs on Amazon Linux 2023's 2.34; bookworm's 2.36 would not), so the fix is
to stop consulting the dead suite: the stage now writes a sources.list with
the bullseye main suite alone before apt-get update. main is intact and
carries every package this stage installs (verified against its current
arm64 index and pool: perl 5.32.1-4+deb11u3, libcurl4 7.74.0-1.3+deb11u13,
cmake 3.18.4-2+deb11u1, clang 1:11.0-51+nmu5, ...). The comment records the
next step for when deb.debian.org drops bullseye entirely: archive.debian.org
already mirrors the main suite.

The Docker daemon is not available where this was prepared, so the stage was
not rebuilt locally; the next push to main exercises it end to end in the
staging deploy, which is exactly the run that failed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw

aterga commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator Author

CI note: test failed on e42406e — one test, and not this PR's.

discoverability::tests::a_legacy_only_origin_is_refused_live fetches the real https://multidex.ai/.well-known/ic-architecture over the network and got a transport error (error sending request for url …), so the refusal it received was the "likely transient — retry" one rather than the legacy-only-origin refusal it asserts at discoverability.rs:1432. The other 178 tests passed, End-to-end (PocketIC), Formatting and the identifier scan are green on this head, and CI on main at 6a67958 passed nine minutes earlier. This PR's diff is deploy/native/build.sh alone — a shell script cargo test never reads — so it cannot have changed what that test does.

No fix to port: the failure is the external origin's reachability from the runner. I've re-run the failed job once. If it fails the same way again, the test's dependence on a live third-party host is the thing to make robust — but that's imcp2-core test code, out of this PR's scope, so I'd propose it separately rather than widen this change.


Generated by Claude Code

@aterga
aterga requested a balanced review from Copilot September 22, 2026 12:17
@aterga
aterga marked this pull request as ready for review September 22, 2026 12:17
@aterga
aterga requested a review from a team September 22, 2026 12:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Preinstalled security-version packages can conflict with older exact-version dependencies available from main.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Updates the Bullseye cross-build environment to avoid the removed security package pool.

Changes:

  • Replaces APT sources with Bullseye main.
  • Documents the eventual archive migration.
File Description
deploy/​native/​build.sh Configures the inline build container’s APT source.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deploy/native/build.sh Resolved
sea-snake
sea-snake previously approved these changes Sep 22, 2026
… alone

Copilot's review of the previous commit was right: restricting apt to
bullseye main is unsatisfiable against this image. rust:1-slim-bullseye
(built 2026-08-25) has its base packages at the final security versions --
perl-base 5.32.1-4+deb11u5, libc6 2.31-13+deb11u14, dpkg 1.20.14 and 25
more -- while main carries perl 5.32.1-4+deb11u3, which depends on
perl-base (= 5.32.1-4+deb11u3), and libc6-dev 2.31-13+deb11u11, which
depends on libc6 (= 2.31-13+deb11u11). apt will not downgrade the
installed perl-base or libc6 to satisfy those, so the install fails.
Verified by reading /var/lib/dpkg/status out of the image's base layer
from the registry and comparing it with main's arm64 index.

Install from snapshot.debian.org at 2026-08-30 instead, for both suites:
the last state in which bullseye and bullseye-security were complete, after
the image was built. Its security index carries exactly the versions the
image has (perl-base +deb11u5, libc6 and libc6-dev +deb11u14, dpkg and
dpkg-dev 1.20.14) and all thirteen packages that 404 on deb.debian.org
(perl, libcurl4 7.74.0-1.3+deb11u16, libexpat1, libicu67, libxml2,
libglib2.0-0, ...), and its pool serves them. Both suites are frozen, so
the pin is the permanent final state, reproducible, with nothing to
downgrade. check-valid-until=no because the snapshot's Release files
expired on 2026-09-05; Acquire::Retries=3 for snapshot's throttling.

Still not rebuilt locally (no Docker daemon here); the staging deploy on
merge remains the end-to-end proof.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
Copilot AI review requested due to automatic review settings September 22, 2026 12:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The PR metadata contradicts the implemented APT configuration, and the reproducibility claim is inaccurate.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Low severity

Open (2)
Resolved since last review (1)

Comment thread deploy/native/build.sh Resolved
Comment thread deploy/native/build.sh Resolved
@aterga aterga changed the title Cross-build from bullseye main only: the -security pool is gone Cross-build from a bullseye snapshot: the live -security pool is gone Sep 22, 2026
@aterga
aterga requested a review from sea-snake September 22, 2026 12:25
The previous commit's comment said the snapshot made "the build
reproducible". It does not: rust:1-slim-bullseye is a mutable tag and
BUILD_TIME is stamped at build time, so the binary and its metadata still
vary from build to build. What the dated snapshot fixes is apt's inputs,
so the comment now says that package resolution is reproducible and names
the two things that are not. Review nit from Copilot on 829622b.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
Copilot AI review requested due to automatic review settings September 22, 2026 12:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Plain HTTP combined with disabled expiration permits replay of older signed package indexes.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (2)

Comment thread deploy/native/build.sh Resolved
With check-valid-until=no, apt no longer rejects an index past its
Valid-Until, so its remaining defence against a network attacker replaying
an older, still validly signed snapshot index -- and thereby an older
package set than the timestamp names -- is the transport. Fetch both
snapshot lines over https so the dated path is authenticated in transit.
The rust:1-slim-bullseye image installs ca-certificates in its own layer
(alongside gcc, libc6-dev and wget), so https works before this stage
installs anything; snapshot.debian.org serves https end to end, redirecting
to same-origin content-hash paths. Review finding from Copilot on 69506cb.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TGhwpdTbUqQbR5AiiN8Ljw
Copilot AI review requested due to automatic review settings September 22, 2026 12:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The deployment-critical Docker stage has not been rebuilt end to end.

Review effort: Balanced
Findings: None

Resolved since last review (1)

@aterga
aterga merged commit ab35deb into main Sep 22, 2026
15 checks passed
aterga pushed a commit that referenced this pull request Sep 22, 2026
Resolves the conflict in deploy/native/deploy.sh between this branch's
OAUTH_CIMD_ENABLED substitution on the unit_mcp line and #194's
SERVE_STATUS block around the caddyfile line: both are kept, the CIMD
comment and substitution first, then the status block as main has it.
The other files main touched (#172, #194, #195) merged cleanly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xd7VT72Qt16qiAJynu9EKj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants