ci(images): publish :dev and :stg ingestor channels

[LukasWodka]

Part 1 of tracebloc/backend#1360. @saadqbal flagging you as the release-pipeline owner.

What this fixes

The ingestor image is the only component without per-environment channels — client-runtime (publish-images.yml) and tracebloc-engine (docker-build.yml) both do develop -> :dev, staging -> :stg. The ingestor image only ever existed as a byproduct of a prod release, so validating a change on a staging edge required a production release. Today that meant a prod PyPI publish + a skip-fr-gate override + a manual dispatch, to test one change (#421 and the failed v0.8.0 tag-build are the receipts).

Note publish-dev.yml does not already cover this: it publishes the Python package to GitHub Packages, and edges never pip-install the ingestor — jobs-manager spawns the image.

Design notes

release-image.yml is untouched. Prod remains vX.Y.Z-triggered, PyPI-gated, cosign-signed with the Release digest. A separate file (rather than adding branch triggers there) keeps the customer-facing cosign identity release-image.yml@refs/tags/v.* meaning exactly what it says, and makes it structurally impossible for a branch build to reach the semver tags, the GitHub Release, or the PyPI gate.

No PyPI gate here. verify-published is a release-policy lockstep ("never ship an image for a version that did not also publish to PyPI"), not a build dependency — the Dockerfile builds from source (pip wheel ., commented "ships the exact code being released, not whatever is on PyPI"). :dev/:stg aren't releases.

Unsigned on purpose. One signed trust root (the prod tags) is clearer than two with different identities. Documented in the file header; don't pin :dev/:stg as a chart default.

Native runners, not QEMU — this is the part that removes the cost objection in #1360. client-runtime's own comment measures the emulated arm64 leg as the bulk of its ~13 min build; today's QEMU-based release-image run took ~25 min. Building each arch on ubuntu-24.04 / ubuntu-24.04-arm in parallel and merging with imagetools makes multi-arch cost about amd64-alone wall-clock — so arm64 edges (Apple Silicon laptops, arm64 nodes) get a pullable image without an emulation tax on every push. Path filters keep docs/workflow-only pushes from spending two runners.

Safety properties

• Refuses any ref that isn't develop/staging, and any channel that isn't dev/stg — verified against master, main, v0.8.0, and forced inputs prod/latest, all correctly refused.
• Refuses to publish a partial index: if fewer than 2 per-arch digests arrive, the merge fails rather than pushing an amd64-only channel tag (the ImagePullBackOff class from client#186). Verified with 0/1/2 digests.
• concurrency per channel with cancel-in-progress — a channel tag is a moving pointer, so superseding an in-flight build is correct.
• Refs passed via env:, never interpolated into shell (R8).

Verification

actionlint 1.7.12 -shellcheck shellcheck 0.11.0 (the versions CI pins): 0 findings. Channel-mapping and partial-index guard logic exercised locally across all branches shown above.

Still needed to make this usable (Part 2, not in this PR)

Publishing the tags is necessary but not sufficient: client/values.yaml has a single images.ingestor.tag (default "0.7") shared by dev and staging, so the two can't sit on different channels. prodDigest is env-aware but tag isn't. #1360 covers making the effective tag env-aware (dev -> dev, stg -> stg, prod unchanged). Until then a test edge can adopt a channel with an explicit images.ingestor.tag: dev|stg override.

The first develop push after this merges will publish :dev — worth watching that run to confirm both arches land.

---

Note

Low Risk
CI-only change publishing unsigned internal images to GHCR; prod release signing and PyPI-gated flow stay in release-image.yml.

Overview
Adds .github/workflows/publish-images.yml so the ingestor Docker image can ship on internal :dev / :stg channels from develop and staging, without touching prod release-image.yml.

The workflow maps branch → tag only (no dispatch channel input), path-filters ingestor-related changes, and uses native amd64/arm64 runners with digest push + manifest merge to ghcr.io/tracebloc/ingestor. It deliberately skips PyPI verification, cosign, and provenance for these channels. Per-arch smoke tests run against the digest before the channel tag moves; merge refuses a single-arch index and verifies both linux/amd64 and linux/arm64 on the published tag.

^{Reviewed by Cursor Bugbot for commit 936d864. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

👋 Heads-up — Code review queue is at 37 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#222 — fix(sklearn): unwrap Pipeline to terminal estimator before merge dispatch (docs(yaml): time_series_forecasting timestamp must be TIMESTAMP not VARCHAR (#200) #209) · author: @divyasinghds · no reviewer assigned
• averaging-service#245 — chore(ci): record gitleaks history baseline (triaged false positives) · author: @LukasWodka · no reviewer assigned
• averaging-service#246 — feat(chart): ServiceAccount + IRSA + blobStore env for the S3 backend (#1162) · author: @saadqbal · no reviewer assigned
• backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
• backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
• backend#1293 — fix(heartbeat): tolerate unknown experiment_key in training_info (#1287) · author: @divyasinghds · no reviewer assigned
• backend#1332 — feat(check-model): allow KNN/GaussianProcess on a single-edge owned dataset (#923 follow-up) · author: @divyasinghds · no reviewer assigned
• backend#1352 — fix(tasks): route reaper beat logs to App Insights (backend#1132) · author: @saqlainsyed007 · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cfc7ca7. Configure here.}

[LukasWodka]

All three findings were valid — fixed in 936d864.

Missing image schema smoke probe (High) — the important one. You're right that this was the outlier: release-image.yml, publish-master.yml and publish-dev.yml all probe _load_schema() against what they ship, and that probe exists for a real regression (v0.3.0-rc1 shipped without tracebloc_ingestor/schema/ because the directory had no __init__.py and find_packages() silently dropped it). Shipping that class of break to the very edges whose job is to test the ingestor would defeat the point of these channels.

Added it per-arch, natively, against the just-pushed digest — and placed deliberately: that digest is unreferenced until the merge job creates the channel tag, so a packaging failure leaves :dev/:stg pointing at the last good image rather than a broken one. It also invokes the console script and asserts it names INGEST_CONFIG, mirroring release-image.yml. Worth noting this is stronger than the prod path, which probes a single arch after a QEMU build — here each arch is probed on its own runner.

Dispatch bypasses branch allowlist (Medium) — the sharpest catch. Correct on both counts: with INPUT_CHANNEL set the ref check was skipped entirely, so a dispatch from master could publish master code to :dev; and the choice preselecting dev meant a dispatch from staging silently defaulted to the wrong channel. I removed the input outright rather than patching around it — the ref is now the only source of the channel, so the code built and the tag published cannot disagree. Dispatch still works: you pick the ref in the Run-workflow dialog.

Concurrency groups split by trigger (Medium) — valid. A push landed in publish-images-develop while a dispatch for the same channel landed in publish-images-dev, so the two could race the same moving tag with cancel-in-progress unable to help across groups. Now keyed on the ref alone — which removing the input makes exact rather than approximate.

Neatly, one change fixed two: dropping the input eliminated both the allowlist bypass and the group split.

Verified: actionlint + shellcheck clean at the CI-pinned versions; channel derivation re-tested — develop→:dev, staging→:stg, and master/main/v0.8.0/hotfix/x/release-train/to-master all refused, with no input able to override the ref; and push-vs-dispatch for the same ref now collapse into one concurrency group.

All three threads resolved.

bugbot run