Skip to content

Three merges in 40s raced the release version plan; PyPI 0.4.30 was built from a commit two merges behind its tag #1016

Closed

Description

@thekevinbot

On 2026-08-21 between 15:05:37 and 15:06:13 UTC, three PRs merged to main (#1011 → 5611aea, #1009 → a94d6c1, #1013 → e2692cd). Their three release.yml runs overlapped and came out inconsistent.

run head outcome
32495716890 5611aea success — published PyPI 0.4.30, tagged only dirsql-npm-v0.4.28 / dirsql-rust-v0.4.24
32495747184 a94d6c1 cancelled by the concurrency group before plan finished
32495773488 e2692cd failure — pypi-publish got 400 File already exists, yet still tagged dirsql-py-v0.4.30 at its own commit

Two separate defects fall out of that.

The version plan is not serialized against tag pushes. The first run published dirsql-py 0.4.30 but its py tag never landed. The third run therefore read the latest py tag as 0.4.29, planned 0.4.30 a second time, built five wheels plus an sdist for it, and PyPI rejected the upload as a duplicate. Any two merges close enough together can do this again; the 40-second window here was not unusual.

A run whose publish failed still wrote the tag. dirsql-py-v0.4.30 now points at e2692cd, but the artifact on PyPI under that version was built from 5611aea, two merges earlier. Tagging is not gated on the publish it describes, so the tag records a release that never happened from that tree.

Impact this time: provenance only

The wheels on PyPI are functionally correct, which is why this did not surface as a user-visible break.

So a wheel built from e2692cd would behave identically to the one built from 5611aea. crates.io 0.4.25 and npm 0.4.29 both came from e2692cd and are current. Nothing needs republishing — but the next occurrence will not necessarily be this benign, since the same race can drop a real source change.

Suggested fix

Both defects are in the release lane rather than in dirsql source, so this likely wants an upstream putitoutthere change (and possibly its own issue there):

  1. Serialize the plan against the tag push, or compute the next version from a source that a concurrent run cannot observe stale — a concurrency group on the publish lane keyed to the package rather than the ref would be the blunt version.
  2. Push a package's tag only after that package's publish succeeds, so a failed publish leaves no tag to mislead the next plan.
  3. Treat 400 File already exists as a hard error worth naming in the log — it currently reads as a generic upload failure, and the duplicate-version cause took a log dig to find.

Leaving dirsql-py-v0.4.30 in place for now: retagging rewrites published history for no functional gain. Reconsider if the audit trail matters more than the churn.

Activity

  1. thekevinbot commented on Sep 2, 2026

    @thekevinbot
    CollaboratorAuthor

    Recurred today, 2026-09-02, same shape and same lane.

    Two merges six minutes apart produced two racing release runs:

    run head outcome
    33655137821 73787b8 (#1000) success — published PyPI 0.4.31, npm 0.4.31, crates 0.4.27
    33655717704 01c3c669 (#1014) failure — pypi-publish got 400 File already exists for dirsql-0.4.31-cp310-abi3-macosx_10_12_x86_64.whl

    Run 2 re-planned 0.4.31 because run 1's version was already live but the plan had been computed before that landed. Its release / publish job then succeeded — isPublished correctly skipped npm and crates — and cut all three tags anyway:

    • dirsql-py-v0.4.31 -> 01c3c669
    • dirsql-npm-v0.4.31 -> 01c3c669
    • dirsql-rust-v0.4.27 -> 01c3c669

    Every one of those artifacts was built from 73787b8. So all three tags name a commit that was never published, and the failing pypi-publish job is downstream of a publish job that reported success — the run is red, but the damage was done by its green half.

    No republish needed. git diff 73787b8..01c3c669 is 5 files, all packages/rust/tests/ plus agents/reference/testing-gates.md — no shipped source in any package. The artifacts are functionally identical to what 01c3c669 would have produced. Same conclusion as the 0.4.30 case: leave the tags, retagging rewrites published history for no gain.

    Worth noting this recurrence is not the 40-seconds-apart edge case. These merges were six minutes apart and it still raced, so the window is wider than the original report suggested. The tagging half is addressed by #1022 (adopt the tag-follows-upload template); the racing half still has no upstream issue filed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions