Release pipeline from #304: four defects, and publishing now requires a manual CLI step #311

Context

#304 landed as fix(lookup): attribute providers to the entries they declare, a fix for #294. Riding along in the same PR was a full migration of CI and the release pipeline: four workflows deleted, publish.yml added, every test and build step moved onto blacklanternsecurity/CLA's shared workflows, the publish trigger moved from a push to stable to a pushed v* tag, and PyPI auth moved from a long-lived token to OIDC trusted publishing.

For the record on how it got in: the PR opened Oct 2 06:22 UTC as a bugfix and the title never changed. The CI and deployment commits landed Oct 3 between 08:28 and 15:13 UTC, after both first-round reviews had already been given on bugfix-only code. Nine of its twelve commits are CI/release work, roughly 700 of ~1,000 changed lines.

That happened, and this issue isn't about reversing it. The new pipeline is genuinely better than what it replaced — the old one let any push to stable, including the 06:00 signature bot, run the entire publish path, and it uploaded to PyPI in two stages with the git tag pushed in between, so a wheel failure left a tagged release with one wheel live.

The problem is that publish.yml only fires on a tag, so no part of the release path has ever executed. Every green check on #304 and since is tests.yml. The defects below are all in the half CI cannot exercise.

Problems

1. No release can run at all — helm/Chart.yaml

The shared release-check requires every git-tracked Chart.yaml to carry the release version. Ours has said version: 1.0.0 since long before this change, independent of the package version.

$ manifest.py --root . tag v11.2.0
tag v11.2.0 does not match helm/Chart.yaml version 1.0.0     → exit 1

check is the first job in publish.yml and every other job needs: it, so pushing v11.2.0 publishes nothing. It fails closed, which is the good news. release.sh runs the same check locally and aborts with merge the version bump first, which describes a different failure entirely and would cost real time to diagnose cold.

Fix: set the chart to the release version. Note the ongoing cost — the Helm chart version is now permanently coupled to the package version and must be bumped every release.

2. Docker tags silently lose their v prefix

release-check emits the manifest version (11.2.0), not the tag (v11.2.0), and the shared docker.yml interpolates it directly:

type=raw,value=${{ inputs.version }}                    # 11.2.0
type=raw,value=${{ inputs.major }}.${{ inputs.minor }}  # 11.2
type=raw,value=${{ inputs.major }}                      # 11

Confirmed by running the planner against dev:

images=["blacklanternsecurity/cloudcheck:11.2.0"]

Every tag on Docker Hub today is v11.1.0 / v11.1 / v11. Nothing errors. latest and stable keep working, so this is invisible until someone pinned to :v11 or :v11.1 — the normal thing to put in a compose file or FROM line — reports a bug that was fixed months earlier. They never get a failed pull, they just stop receiving updates.

Fix, local, no upstream change:

version: v${{ needs.check.outputs.version }}
major: v${{ needs.check.outputs.major }}

Or decide deliberately to move to unprefixed tags, with a period where both exist.

3. pyo3 0.27 against Python 3.15 — deadline Oct 9 — RESOLVED

Resolved by #310, merged into dev 2026-10-06 19:03 UTC. Cargo.toml is now on pyo3 = "0.29" and pyo3-async-runtimes = "0.29" (0.29.3 / 0.29.0 in the lockfile), which supports Python 3.15. No source changes were needed — python.rs was already on the Bound / Python::attach API. Original text kept below for the record.

Raised by @en0f on #304 and still open. The test matrix is no longer a literal list; manifest.py matrix returns every released CPython satisfying requires-python = ">=3.10", which is unbounded. Cargo.toml pins pyo3 = "0.27", which refuses to build against an unknown interpreter.

On 3.15.0 release day, python / test (3.15) fails at maturin develop, which fails the required python / passed check, which blocks every PR in the repo — and the 3.15 wheel leg kills any release. No commit to this repo is needed to trigger it.

Fix: #310, open and green. pyo3 0.29.3, no source changes required.

4. PyPI trusted publishing, and the order it fails in

The pypi job uses OIDC and needs a trusted publisher registered on PyPI for workflow publish.yml, environment pypi. PYPI_API_TOKEN and PYPI_TOKEN are both still present in repo secrets with nothing referencing them, which suggests the migration step is outstanding — worth confirming on pypi.org, which can't be checked from the repo side.

The ordering makes this worse than it sounds. crates and pypi both only needs: wheels, so they run concurrently. If the publisher isn't registered, crates.io publishes successfully while PyPI fails on auth — and crates.io has no unpublish, so that version is permanently burned. Because crates.yml runs a bare cargo publish with no skip-existing guard, re-running the workflow then dies at crates.io on "already published."

Fix: register the publisher before tagging anything and delete the two dead secrets. Separately, consider making crates depend on pypi so the irreversible step runs last.

Behavior change: publishing is now a manual CLI task

This is a change in kind, not just in configuration, and it deserves to be an explicit decision rather than a side effect of a bugfix PR.

Before: merging to stable published. One human action — a button in the GitHub UI.

Now: merging publishes nothing. A person must open a terminal and push a tag:

git checkout stable && git pull
git tag -a v11.2.0 -m v11.2.0
git push origin refs/tags/v11.2.0

The documented path is the org's release.sh, which validates trunk, a clean tree, remote sync, tag shape and the manifest version before prompting. But that script is not in this repository. It lives in blacklanternsecurity/CLA/scripts/, linked from our README at a blob URL pinned to e532142 — a commit on the unmerged feat/shared-workflows branch. It also sources its sibling manifest.py, so it can't be curled standalone, and it requires uv and jq locally. In practice:

git clone https://github.com/blacklanternsecurity/CLA ~/src/CLA
~/src/CLA/scripts/release.sh v11.2.0

Consequences worth naming:

• New opportunities for error. Tagging the wrong commit, tagging from a feature branch, typo'ing the version, forgetting the v, or tagging before the version bump merges. release-check catches the shape and version mismatches; it does not verify the tagged commit is on stable or anywhere near it.
• The guard rails are optional. git tag && git push takes the same four seconds and skips every check in release.sh. There is no tag ruleset and no tag protection on this repo — the single active ruleset targets ~DEFAULT_BRANCH only — so anyone who can push a tag can publish from an arbitrary commit that never saw review. Tests re-run on the tag, but tests are not review. The old trigger required a merge to stable, which the ruleset gated; the new one is gated by nothing.
• A maintenance task that lives outside the repo. Releasing now depends on a developer workstation having uv, jq and a clone of another repository at the right revision. Onboarding anyone to cut a release means onboarding them to all of that, and the README is the only thing holding the convention together.
• No manual re-trigger. publish.yml has no workflow_dispatch. A failed release can only be retried by re-running jobs in the UI or deleting and re-pushing the tag, which combines badly with the non-idempotent cargo publish and the missing --skip-existing on PyPI.

Proposal: restore merge-as-release

Merge-triggered releases and the new pipeline aren't in conflict. Only the old trigger's looseness was the problem, and a version check fixes that in about twenty lines:

name: Tag release
on:
  push:
    branches: [stable]
jobs:
  tag:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/create-github-app-token@...
        id: app-token
        with:
          app-id: ${{ secrets.CLOUDCHECK_APP_ID }}
          private-key: ${{ secrets.CLOUDCHECK_APP_PRIVATE_KEY }}
      - uses: actions/checkout@...
        with: { fetch-depth: 0, token: '${{ steps.app-token.outputs.token }}' }
      - run: |
          version=$(grep -m1 '^version' Cargo.toml | sed 's/.*"\(.*\)".*/\1/')
          git rev-parse -q --verify "refs/tags/v$version" >/dev/null && exit 0
          git tag -a "v$version" -m "v$version"
          git push origin "refs/tags/v$version"

Merge → version differs from every existing tag → tag pushed → publish.yml runs normally. The daily signature commit never touches Cargo.toml, so it tags nothing — which is the behavior the old system lacked. The app token matters: tags pushed with GITHUB_TOKEN do not trigger workflows. CLOUDCHECK_APP_* is already in the repo and is what the deleted publish job used for exactly this.

This keeps everything the new design fixed — validation before anything publishes, all wheels built before the first upload, one ordered DAG, no daily republish — and returns the trigger to a button.

Secondary issues

• Pinned to an unmerged branch. Every workflow entry point depends on blacklanternsecurity/CLA@e532142, on feat/shared-workflows (CLA#3, still open). The SHA makes the content immutable but not reachable — if that branch is deleted or force-pushed and the commit is collected, every workflow here stops resolving at once. Re-pin to a default-branch SHA once CLA#3 merges.
• At the workflow nesting ceiling. On a tag, publish.yml → tests.yml → python-tests.yml → python-versions.yml is four levels, GitHub's documented maximum, and has never executed. PRs only reach three. If the shared python-tests.yml ever adds a nested call, releases break while PRs stay green.
• No rust-toolchain.toml. The shared Rust jobs run rustup show active-toolchain || rustup toolchain install, which is designed to honor a pinned toolchain file. Without one, the toolchain is whatever the runner image ships that week.
• 156 stale Docker Hub tags. The old cleanup step is gone. The new scheme stops creating sha-* tags but nothing prunes the backlog.
• Daily job interpreter is now implicit. daily-update.yml lost its setup-python 3.12 so uv follows requires-python. With no .python-version in the repo, the interpreter is whatever uv resolves from the runner image rather than what we declared.
• docker test job rebuilds the Rust extension unnecessarily. uv run --with requests python test_docker.py syncs the project first; --no-project would cut several minutes and decouple the Docker test from the Rust build.

Suggested order

1. helm/Chart.yaml → 11.2.0 — nothing releases until this lands
2. Merge build(deps): bump pyo3 to 0.29 for Python 3.15 #310 before Oct 9 — done, merged 2026-10-06
3. Register the PyPI trusted publisher, delete PYPI_API_TOKEN and PYPI_TOKEN
4. Restore the v prefix on Docker tags, or decide against it explicitly
5. Decide on merge-as-release vs. the manual tag step
6. Cut v11.2.0-rc.1 as a dry run — release-check understands -rc.N and marks it prerelease, so it exercises the whole path, including the untested nesting depth, on a version nobody minds burning
7. Re-pin the shared workflows once CLA#3 merges

Considered and accepted, not listed above: the manylinux floor moving from 2.17 to 2.28 (drops CentOS 7, RHEL 7, Ubuntu 18.04, Amazon Linux 2 — not officially supported). Required-check names in the ruleset are known and being handled separately.