Skip to content

Publish reproducible Vision v8 ONNX release artifacts - #16

Merged
Complexity-ML merged 2 commits into
Complexity-ML:mainfrom
IIIllllIlIlllII:feat/onnx-release-workflow
Aug 27, 2026
Merged

Complexity-ML merged 2 commits into
Complexity-ML:mainfrom
IIIllllIlIlllII:feat/onnx-release-workflow

Conversation

@IIIllllIlIlllII

Copy link
Copy Markdown
Contributor

Closes #11.

Adds an ONNX release workflow — manual dispatch or an onnx-v8-* tag — that
downloads a pinned checkpoint revision, exports both prediction branches, runs
the parity gates from #12, writes a machine-readable manifest and verifies every
checksum before it uploads anything. Any failure in that chain aborts
publication.

All logic lives in scripts/build_onnx_release.py, so the release path runs and
is tested outside CI. The workflow only installs the pinned toolchain and calls
the script.

Pinning the toolchain is required, not defensive

The acceptance criteria ask that published assets be reproducible from the
documented commit and checkpoint revision, and that artifact hashes match the
manifest. Those two together are only true if the export toolchain is pinned,
and the repository is the natural place to pin it — which is what
docs/onnx/release.json does.

The demonstration is cheap. Re-exporting the checkpoint documented in the
validation report, at the same revision and opset, under torch 2.13.0 instead
of 2.6.0:

Artifact Report (torch 2.6.0) Rebuild (torch 2.13.0)
tr_hash_v8_o2m.onnx 11,104,476 bytes 11,104,477 bytes
tr_hash_v8_nms_free.onnx 11,108,683 bytes 11,108,684 bytes

Exactly one byte larger on both branches, and therefore completely different
digests. torch.onnx.export stamps its own version into the model's
producer_version field, and "2.13.0" is one character longer than "2.6.0"
(verified: onnx.load(...).producer_version == "2.13.0"). The graph is
otherwise unchanged — the parity gates pass with identical numbers — but the
bytes are not. A commit plus a checkpoint revision does not determine a SHA-256.

Consequences in the design:

  • The build refuses to run when installed versions differ from the pin.
    Without that gate the manifest would advertise digests nobody could reproduce.
    --allow-toolchain-drift downgrades it to a warning for local dry runs; CI
    never passes it.
  • The workflow reads its version pins from release.json rather than
    repeating them in YAML, so the manifest cannot drift from what CI installs.
  • The checkpoint revision must be a full 40-character sha. main is rejected
    with an explicit message, since a moving ref would make reproducibility depend
    on when the build ran.

Manifest

manifest.json is published alongside the binaries and carries every field the
issue asks for — checkpoint revision, framework commit, opset, input/output
contract, sizes, SHA-256 — plus the toolchain versions, for the reason above.
Verify a downloaded release with:

python scripts/build_onnx_release.py --verify manifest.json

Release notes are generated from the manifest and keep the branches distinct:
O2M is decode plus NMS, NMS-free is decode plus confidence filtering.

Binaries stay out of source

*.onnx, *.engine and /dist/ are now ignored. They were not before — a
local export left tr_hash_v8_o2m.onnx sitting untracked at the repository root,
one git add -A away from being committed.

Verification

Full local run against AETHORIA-AI/TR-HASH-Vision-v8-2M-COCO-SFT at
f3b3e659612e543ca9ff91892c0662d38dc1a1d6: both exports succeed, all six parity
gates pass, manifest written and re-verified. tests/test_onnx_release.py adds
12 tests covering config validation, toolchain drift, the output contract,
manifest round-trip, and detection of an altered, truncated or missing artifact
— none of which need network or torch. The release tooling is now linted and
tested by the existing Detector export workflow.

The first workflow run will produce the first reference digests; the historical
ones in the validation report predate this path and are labelled as such.

Add an ONNX release workflow, triggered manually or by an onnx-v8-* tag, that
downloads a pinned checkpoint revision, exports both prediction branches, runs
the parity gates, writes a manifest and verifies every checksum before it
uploads anything. Any failure in that chain aborts before publication.

All logic lives in scripts/build_onnx_release.py so it can run and be tested
outside CI; the workflow only installs the pinned toolchain and calls it.

docs/onnx/release.json pins the checkpoint repository and revision, the opset,
and the export toolchain, and the workflow reads its version pins from that same
file so the manifest cannot drift from what CI installs. Pinning the toolchain
is what makes a repository commit sufficient to reproduce a digest: re-exporting
the same checkpoint under torch 2.13.0 instead of 2.6.0 produces binaries one
byte larger on both branches, because torch.onnx.export stamps its own version
into producer_version. The graph is unchanged and the parity gates pass
identically, but the SHA-256 does not match.

A moving checkpoint ref is rejected: the revision must be a full 40-character
sha, otherwise reproducibility would silently depend on when the build ran.

Also ignore *.onnx, *.engine and /dist/ so exported graphs cannot reach a source
commit, and link the release path from the validation report.

Closes Complexity-ML#11

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@Complexity-ML Complexity-ML left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two changes before merge:

  1. Blocking — bind the release tag to the manifest commit. The workflow can create/upload a release for TAG while manifest.json records framework_commit from GITHUB_SHA, but I don't see a guard that the Git tag actually resolves to that same commit. For reproducibility, please either create the tag explicitly at ${GITHUB_SHA} or verify git rev-list -n 1 "$TAG" == "$GITHUB_SHA" (or equivalent) before any upload, and abort publication on mismatch.

  2. Release parity depth — increase parity_num_tests. release.json currently uses 5 seeds, while #15 showed that 5 seeds materially underestimate the observed maxima and that healthy exports exceed the old thresholds by 20–50 seeds. For an official release gate, please use at least 20 seeds, ideally 50. This workflow runs only for releases, so the additional cost is justified.

Everything else in the PR looks well structured: pinned checkpoint/toolchain, manifest + SHA-256 verification, pre-upload failure gates, and dedicated release tests.

@Complexity-ML Complexity-ML left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two changes before merge:

  1. Blocking — bind the release tag to the manifest commit. The workflow can create/upload a release for TAG while manifest.json records framework_commit from GITHUB_SHA, but I don't see a guard that the Git tag actually resolves to that same commit. For reproducibility, please either create the tag explicitly at ${GITHUB_SHA} or verify git rev-list -n 1 "$TAG" == "$GITHUB_SHA" (or equivalent) before any upload, and abort publication on mismatch.

  2. Release parity depth — increase parity_num_tests. release.json currently uses 5 seeds, while #15 showed that 5 seeds materially underestimate the observed maxima and that healthy exports exceed the old thresholds by 20–50 seeds. For an official release gate, please use at least 20 seeds, ideally 50. This workflow runs only for releases, so the additional cost is justified.

Everything else in the PR looks well structured: pinned checkpoint/toolchain, manifest + SHA-256 verification, pre-upload failure gates, and dedicated release tests.

Nothing tied the published tag to the commit the manifest records. gh release
create without --target tags the default branch head, so a dispatch run could
publish a manifest whose framework_commit cannot be checked out from the release
itself, which defeats the point of pinning anything.

The release is now created with --target "$GITHUB_SHA"; an existing tag is
resolved through the API and publication aborts when it points elsewhere; and
--verify gained --expect-commit so the manifest is bound to the publishing
commit by tested code rather than by shell in the workflow.

Raise parity_num_tests from 5 to 50. Five seeds underestimate the observed
maximum by about 60%, so a release gate has to be deeper than the development
default; a test keeps the committed value at 20 or above. At 50 seeds the maxima
are 2.79e-03 (o2m) and 4.79e-03 (nms-free) against thresholds of 6.0e-03 and
1.0e-02, so the deeper gate keeps roughly a 2x margin. Job timeout raised to 60
minutes to cover two exports plus 100 parity runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@IIIllllIlIlllII

Copy link
Copy Markdown
Contributor Author

Tag binding.
You were right that nothing tied the two together — gh release create without --target tags the default branch head, not the commit that built the artifacts.

Now guarded in three places:

  • the release is created with --target "$GITHUB_SHA";
  • if the tag already exists, a step resolves it through the API and aborts when
    it points anywhere other than $GITHUB_SHA;
  • --verify gained --expect-commit, so the manifest's framework_commit is
    checked against the publishing commit by the script rather than by shell
    buried in YAML. That part is unit-tested.

Release parity depth.
parity_num_tests is now 50, and a test asserts the committed config stays at 20 or above so it cannot quietly drift back down to speed CI up. Job timeout raised to 60 minutes.

Verified locally end to end at the new depth: both exports, all six gates, manifest written and re-verified in 3m9s wall (17m25s CPU across 12 threads, so expect roughly 10–15 min on a 4-core runner). The maxima at 50 seeds are 2.79e-03 (o2m) and 4.79e-03 (nms-free) against thresholds of 6.0e-03 and 1.0e-02 — the release gate keeps about a 2x margin at its new depth, and reproduces the calibration figures from #15 digit for digit.

@Complexity-ML
Complexity-ML merged commit 7870fda into Complexity-ML:main Aug 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish reproducible Vision v8 ONNX release artifacts

2 participants