Repository navigation
Publish reproducible Vision v8 ONNX release artifacts - #16
Complexity-ML merged 2 commits into
Conversation
Add an ONNX release workflow, triggered manually or by an onnx-v8-* tag, that downloads a pinned checkpoint revision, exports both prediction branches, runs the parity gates, writes a manifest and verifies every checksum before it uploads anything. Any failure in that chain aborts before publication. All logic lives in scripts/build_onnx_release.py so it can run and be tested outside CI; the workflow only installs the pinned toolchain and calls it. docs/onnx/release.json pins the checkpoint repository and revision, the opset, and the export toolchain, and the workflow reads its version pins from that same file so the manifest cannot drift from what CI installs. Pinning the toolchain is what makes a repository commit sufficient to reproduce a digest: re-exporting the same checkpoint under torch 2.13.0 instead of 2.6.0 produces binaries one byte larger on both branches, because torch.onnx.export stamps its own version into producer_version. The graph is unchanged and the parity gates pass identically, but the SHA-256 does not match. A moving checkpoint ref is rejected: the revision must be a full 40-character sha, otherwise reproducibility would silently depend on when the build ran. Also ignore *.onnx, *.engine and /dist/ so exported graphs cannot reach a source commit, and link the release path from the validation report. Closes Complexity-ML#11 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Complexity-ML
left a comment
There was a problem hiding this comment.
Two changes before merge:
-
Blocking — bind the release tag to the manifest commit. The workflow can create/upload a release for
TAGwhilemanifest.jsonrecordsframework_commitfromGITHUB_SHA, but I don't see a guard that the Git tag actually resolves to that same commit. For reproducibility, please either create the tag explicitly at${GITHUB_SHA}or verifygit rev-list -n 1 "$TAG" == "$GITHUB_SHA"(or equivalent) before any upload, and abort publication on mismatch. -
Release parity depth — increase
parity_num_tests.release.jsoncurrently uses 5 seeds, while #15 showed that 5 seeds materially underestimate the observed maxima and that healthy exports exceed the old thresholds by 20–50 seeds. For an official release gate, please use at least 20 seeds, ideally 50. This workflow runs only for releases, so the additional cost is justified.
Everything else in the PR looks well structured: pinned checkpoint/toolchain, manifest + SHA-256 verification, pre-upload failure gates, and dedicated release tests.
Complexity-ML
left a comment
There was a problem hiding this comment.
Two changes before merge:
-
Blocking — bind the release tag to the manifest commit. The workflow can create/upload a release for
TAGwhilemanifest.jsonrecordsframework_commitfromGITHUB_SHA, but I don't see a guard that the Git tag actually resolves to that same commit. For reproducibility, please either create the tag explicitly at${GITHUB_SHA}or verifygit rev-list -n 1 "$TAG" == "$GITHUB_SHA"(or equivalent) before any upload, and abort publication on mismatch. -
Release parity depth — increase
parity_num_tests.release.jsoncurrently uses 5 seeds, while #15 showed that 5 seeds materially underestimate the observed maxima and that healthy exports exceed the old thresholds by 20–50 seeds. For an official release gate, please use at least 20 seeds, ideally 50. This workflow runs only for releases, so the additional cost is justified.
Everything else in the PR looks well structured: pinned checkpoint/toolchain, manifest + SHA-256 verification, pre-upload failure gates, and dedicated release tests.
Nothing tied the published tag to the commit the manifest records. gh release create without --target tags the default branch head, so a dispatch run could publish a manifest whose framework_commit cannot be checked out from the release itself, which defeats the point of pinning anything. The release is now created with --target "$GITHUB_SHA"; an existing tag is resolved through the API and publication aborts when it points elsewhere; and --verify gained --expect-commit so the manifest is bound to the publishing commit by tested code rather than by shell in the workflow. Raise parity_num_tests from 5 to 50. Five seeds underestimate the observed maximum by about 60%, so a release gate has to be deeper than the development default; a test keeps the committed value at 20 or above. At 50 seeds the maxima are 2.79e-03 (o2m) and 4.79e-03 (nms-free) against thresholds of 6.0e-03 and 1.0e-02, so the deeper gate keeps roughly a 2x margin. Job timeout raised to 60 minutes to cover two exports plus 100 parity runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Tag binding. Now guarded in three places:
Release parity depth. Verified locally end to end at the new depth: both exports, all six gates, manifest written and re-verified in |
Closes #11.
Adds an
ONNX releaseworkflow — manual dispatch or anonnx-v8-*tag — thatdownloads a pinned checkpoint revision, exports both prediction branches, runs
the parity gates from #12, writes a machine-readable manifest and verifies every
checksum before it uploads anything. Any failure in that chain aborts
publication.
All logic lives in
scripts/build_onnx_release.py, so the release path runs andis tested outside CI. The workflow only installs the pinned toolchain and calls
the script.
Pinning the toolchain is required, not defensive
The acceptance criteria ask that published assets be reproducible from the
documented commit and checkpoint revision, and that artifact hashes match the
manifest. Those two together are only true if the export toolchain is pinned,
and the repository is the natural place to pin it — which is what
docs/onnx/release.jsondoes.The demonstration is cheap. Re-exporting the checkpoint documented in the
validation report, at the same revision and opset, under torch
2.13.0insteadof
2.6.0:tr_hash_v8_o2m.onnx11,104,476bytes11,104,477bytestr_hash_v8_nms_free.onnx11,108,683bytes11,108,684bytesExactly one byte larger on both branches, and therefore completely different
digests.
torch.onnx.exportstamps its own version into the model'sproducer_versionfield, and"2.13.0"is one character longer than"2.6.0"(verified:
onnx.load(...).producer_version == "2.13.0"). The graph isotherwise unchanged — the parity gates pass with identical numbers — but the
bytes are not. A commit plus a checkpoint revision does not determine a SHA-256.
Consequences in the design:
Without that gate the manifest would advertise digests nobody could reproduce.
--allow-toolchain-driftdowngrades it to a warning for local dry runs; CInever passes it.
release.jsonrather thanrepeating them in YAML, so the manifest cannot drift from what CI installs.
mainis rejectedwith an explicit message, since a moving ref would make reproducibility depend
on when the build ran.
Manifest
manifest.jsonis published alongside the binaries and carries every field theissue asks for — checkpoint revision, framework commit, opset, input/output
contract, sizes, SHA-256 — plus the toolchain versions, for the reason above.
Verify a downloaded release with:
Release notes are generated from the manifest and keep the branches distinct:
O2M is decode plus NMS, NMS-free is decode plus confidence filtering.
Binaries stay out of source
*.onnx,*.engineand/dist/are now ignored. They were not before — alocal export left
tr_hash_v8_o2m.onnxsitting untracked at the repository root,one
git add -Aaway from being committed.Verification
Full local run against
AETHORIA-AI/TR-HASH-Vision-v8-2M-COCO-SFTatf3b3e659612e543ca9ff91892c0662d38dc1a1d6: both exports succeed, all six paritygates pass, manifest written and re-verified.
tests/test_onnx_release.pyadds12 tests covering config validation, toolchain drift, the output contract,
manifest round-trip, and detection of an altered, truncated or missing artifact
— none of which need network or torch. The release tooling is now linted and
tested by the existing
Detector exportworkflow.The first workflow run will produce the first reference digests; the historical
ones in the validation report predate this path and are labelled as such.