Repository navigation
🤖 ci: diagnose and restore verified main image publishing #98
Description
Problem
The latest CI run for current main failed while publishing ghcr.io/coder/coder-k8s:main:
- Commit:
b1d48e3a64b42a23cb0cfaee8c30f3f21b761424. - CI run 22223215157.
- Failed publishing job.
- The check annotation reports GoReleaser 2.13.3 exiting with code 1.
- Historical logs now return HTTP 410. The annotation does not establish the underlying cause.
Other validation jobs were skipped in that run. This is not evidence that the current source fails its tests, nor proof that its current image is broken. Image provenance and the publishing path need verification.
Decision and scope
Accepted for bounded diagnosis and the smallest evidence-backed repair, if needed. Maintainer-owned delivery will first reproduce the relevant publishing configuration in an isolated, non-publishing environment. Distinguish configuration/build failure from infrastructure or registry failure. Do not change unrelated dependencies or CI settings merely to obtain a green run.
Do not rerun the publisher just to recover logs: that can overwrite the mutable :main image. Do not create a release, change branch protections, or deploy to a shared cluster for diagnosis.
Acceptance criteria
- Record source SHA, tool versions, exact commands, native exit codes, and current image provenance where accessible. Explain any inaccessible historical evidence.
- Reproduce the failure or document the precise remaining blocker; do not invent a root cause from the expired logs.
- If a source fix is necessary, add focused regression coverage and make the smallest coherent change. Preserve release and main-channel behavior.
- Run the required local test, build, lint, vendor and relevant workflow/docs checks. Validate non-publishing release/main-channel behavior with screenshots and video of the hands-on check.
- Bind CI, normal Codex review, security Codex review, and an independent final recommendation to the candidate head. Keep the lifetime review ledger within six completed assessments, reserving the final slot.
- After an authorized merge, verify main CI and the resulting image revision/digests. Until then, report readiness separately from restored publishing.
Execution and verification
First investigate and record a minimal plan. Prove the reproduction before editing. Then validate locally, complete review, and hand off the ready PR for the repository's human-only merge step. Post-merge verification remains maintainer-owned.
Generated with xum • Model: coder:openai/gpt-6-astra • Thinking: xhigh
Activity
ThomasK33 commentedon Sep 18, 2026
Readiness: blocked
PR #99, exact head
8aa69887a61ccca0d21964eebc98d552c0a764bc, implements the demonstrated main-channel preparation repair. Publication is not yet verified.make test,make build,make verify-vendor,make lint, and actionlint v1.7.10. Pinned Go 1.25.7 / GoReleaser 2.13.3 main and release-fixture non-publishing packaging both exited 0. Screenshots and recordings are attached to the PR and their downloaded hashes match the original evidence.8aa69887a6. There are zero review threads. The fresh independent assessor found no source defect and recommends blocked because required CI is not green. Lifetime count: 4/6 assessments (automatic normal, explicit normal, explicit security, independent final); two slots remain. No failed review requests or retries were needed.lintfailsgovulncheckwith 55 called vulnerability IDs. The exact same 55 IDs reproduce on unchanged mainb1d48e3a64b42a23cb0cfaee8c30f3f21b761424, exit 3.Terraform (fmt/validate/tflint/trivy)fails while installing Trivy v0.65.0, before scanning; one targeted PR-job retry failed at the same point. A read-only request for that upstream release returns HTTP 404. The logs do not establish why it is unavailable. Filesystem and image scans fail at the same installer boundary.Codex Commentstreats the bot's completed summary and explicit clean security verdict as unresolved regular comments. A fresh local helper run returns 1 while reporting zero unresolved review threads. These comments were not hidden or deleted. Requiredtest, workflow lint, and changed-path checks passed. The successful PR E2E check follows the workflow's documented skip; it is not full cluster E2E evidence.Decision and stopping reason: the source repair is validated, but PR #99 is not ready to merge. Do not bypass the failing checks, broaden this repair into unrelated dependency upgrades, or reset the review budget. The maintainer desk owns follow-up coordination under issue #98: scope the baseline security and scanner/helper compatibility work, then revalidate this PR within its remaining review capacity. Any head change invalidates head-bound evidence and must be reconciled within that capacity.
Issue #98 remains open. Human owns every merge/enqueue/auto-merge operation. After an authorized merge, the delivery workspace and maintainer desk retain responsibility for main CI and image revision/digest verification. No publisher rerun, GHCR push, release, deployment, infrastructure change, or merge was performed for this diagnosis.
Generated with
xum• Model:coder:openai/gpt-6-astra• Thinking:xhighThomasK33 commentedon Sep 21, 2026
Verified recovery — completed
PR #99 landed as
cd378781a116dac8c68f908bdc74d8425413677fafter required merge-group CI, including full Kind/CNPG/template E2E. The normal main CI publisher run 35605551335 succeeded with GoReleaser 2.18.2. Its published-image HIGH/CRITICAL Trivy gate passed.The maintainer independently fetched and SHA256-verified the registry index, each platform manifest, and each image config. The index digest matches the publisher's output:
ghcr.io/coder/coder-k8s@sha256:3e70115126612f8c3a283705f691dface651a4ddad3675c6aaeb860161418135sha256:e50d6a90348d7f366b9b5a5e8f076b8c35b4b1ec3e2a8ef2bacefcae2df60fb4cd378781a116dac8c68f908bdc74d8425413677fsha256:b377e06e6f733c03daa18b6106d74a90c82a0287a0aaefa7eb871fde71574e08cd378781a116dac8c68f908bdc74d8425413677fThe mutable
:maintag was unchanged across the verification reads. This replaces the previously observed stale indexsha256:90b505ac03d7f5862d25dd955a74957c8fcc9d344f69c42ebd1f841111a2fa18.All six acceptance criteria are satisfied. PR #99 used five of six lifetime assessments. Required validation ran in the merge group; skipped jobs in the subsequent push run are not represented as reruns. Actual publication and registry provenance are verified, not inferred from private packaging alone.
Limits: this registry check did not execute the published containers or independently scan both architectures. The expired original failure logs remain unavailable; the later reproduced and observed synthetic-tag failure is not retroactive proof of every historical cause. Tagged release publication was not performed.
Generated with
xum• Model:coder:bedrock-mantle-us-west-2/openai.gpt-6-astra• Thinking:xhigh