ci(ENG-CI-ORACLE-GOLDENS): run the oracle-goldens digests on a lane, not only on a developer box - #2555
Merged
Merged
Conversation
…not only on a developer box `tests/scripts/test_ltx2_oracle_goldens.py` was in `scripts/agent-preflight.sh`'s `SUITES` array and in no step of `.github/workflows/ci.yml`. Read with the checker's own `_active_ci_commands` at `beafb46df`, which parses the file the way a lane does rather than the way a grep does: 110 active commands, the goldens suite absent, `test_ltx2_latent_dump` present as the positive control. #2520 made that gap expensive. The suite now also covers `tests/parity/goldens/ltx2_oracle_latent/` -- the Phase A latents a future LTX-2 cross-decode reads INSTEAD of taking a GPU lease. Their whole value is that they are the bytes the leased run produced, and until this step a pull request could overwrite `video_latent_bfloat16.raw` and merge green. Four mutations on a scratch copy, each red, the unmutated copy green: one byte flipped in `video_latent_bfloat16.raw` -> 2 FAILs, rc 1 that byte flipped AND `SHA256SUMS` rewritten -> 1 FAIL, rc 1 an unrecorded artefact added to the directory -> 1 FAIL, rc 1 `upstream-render.mp4` truncated by one byte -> 2 FAILs, rc 1 The second is the one worth naming: the digest file beside the artefacts cannot anchor itself, and what catches it is the third anchor, the sha256 the dump manifest carries because the worker computed it before anything was copied out. The step is unconditional in `agent-record`, which carries no job-level `if:`. The suite is standard library only -- `hashlib`, `json`, `re`, `pathlib` -- over four committed files, 0.02 s, no numpy, no build, no GPU, no lease, no checkpoint, no network. It needs no guard, and a guard is what would have turned an honest absence into false coverage. `REQUIRED_SUITE_REGISTRATIONS` gains the suite in the same change, because that is the only moment the widening is free: seeded with one entry by #1833, it is the mechanism that spec says a later row widens rather than replaces. Deleting the new CI step now prints `pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane`; deleting the preflight entry prints the preflight half. Without it the step this commit adds would itself have been deletable at rc=0, which is the #1833 finding restored. This is not the symmetric population rule, which still belongs to #408 and #1509. The synthetic fixtures in `test_check_test_registration.py` gain the second pinned suite in both lanes so `test_complete_preflight_and_ci_wiring_passes` still reads `[]`. No `test_M*` method name changes, so the digest-pinned mutation manifest is untouched. 70 tests OK. FOLLOWING_AGENTS_PROTOCOL Closes #2553 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…against the base checker `pr-size` was FAILURE on #2555 (job 100149327523): ERROR: BASE checker stayed green for 'scripts/check-test-registration.py'; changed test is not semantic evidence `scripts/check-pr-size.py:616` runs the HEAD test module against the BASE checker and requires it to FAIL. The previous commit's only test edit added `test_ltx2_oracle_goldens` to `PASSING_CI` and `PASSING_PREFLIGHT`. That edit is additive and symmetric, so the base checker -- which pins only `test_sglang_lease_identity` -- is indifferent to it, and nothing in the module exercised the new pin at all. Its job was to KEEP `test_complete_preflight_and_ci_wiring_passes` green, which is exactly what the contract rejects: a test that cannot fail for the intended reason is not evidence. Reproduced before the fix, the way the gate reproduces it: install `beafb46df:scripts/check-test-registration.py` over the head checker in a detached worktree at c2595e1 and run the head module. Ran 70 tests in 6.196s OK rc=0 Two behavioural cases now assert the checker's OUTPUT on a mutated input, in the shape M56/M57 already use for the first pinned suite: drop the ltx2 step from the CI fixture while preflight keeps it, and require `pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane`; then the preflight mirror. Both carry the `assertNotEqual` guard, so a `replace` that stopped matching reds instead of reading as a pass. They do not assert `"test_ltx2_oracle_goldens" in REQUIRED_SUITE_REGISTRATIONS`. Asserting against the constant is a tautology that would let the constant change freely. They are also deliberately not named `test_M*`: `check-test-registration.py:1007` requires the `test_M*` set to equal the digest-pinned manifest exactly, so an M-number would drag `check_test_registration_mutations.txt` and `MUTATION_MANIFEST_SHA256` into a change that is not about the manifest. All 57 `test_M*` names are byte-identical to `beafb46df`, and the manifest still hashes to 4589e7a96d839a8309fd0e5799cc7a652b22a9d568d56151e2a1b8feea5d02ee. Same worktree, same invocation, after the fix: BASE checker -> Ran 72 tests, FAILED (failures=2), rc=1 HEAD checker -> Ran 72 tests, OK, rc=0 Both failures read `AssertionError: False is not true : []` -- the base checker emits no error at all for the missing registration, which is the finding stated as a test. Note the local gates could not have caught this: `check-pr-size` is invoked from neither `scripts/agent-preflight.sh` nor `scripts/agent-ready.py`. It is CI-only, so `preflight rc=0` is not sufficient evidence for a checker change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… whose build-test-cpu red this PR inherited CI builds the merge commit, so every verdict on #2555 was measured against a main whose build-test-cpu was red repo-wide on the rank guard in `dense_device_glue.h:52`. That defect is fixed on main by 33b08e4, in the test fixture's two rank-5 paged-KV scratch buffers, so this merge re-bases the pull request onto a green trunk and the next push measures this change rather than the trunk it sat on. Nothing this pull request touches moved on main across the merged range: `.github/workflows/ci.yml`, `scripts/check-test-registration.py`, `tests/scripts/test_check_test_registration.py`, `tests/scripts/test_ltx2_oracle_goldens.py` and `scripts/agent-preflight.sh` are byte-identical to the merge base, and so are the goldens the pinned suite digests. The merge is therefore a fast-forward of the base only, and it falsifies none of the prose the change carries. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci(ENG-CI-ORACLE-GOLDENS): run the oracle-goldens digests on a lane, not only on a developer box
tests/scripts/test_ltx2_oracle_goldens.pywas inscripts/agent-preflight.sh'sSUITESarray and in no step of.github/workflows/ci.yml. Read with thechecker's own
_active_ci_commandsatbeafb46df, which parses the file the waya lane does rather than the way a grep does: 110 active commands, the goldens
suite absent,
test_ltx2_latent_dumppresent as the positive control.#2520 made that gap expensive. The suite now also covers
tests/parity/goldens/ltx2_oracle_latent/-- the Phase A latents a futureLTX-2 cross-decode reads INSTEAD of taking a GPU lease. Their whole value is
that they are the bytes the leased run produced, and until this step a pull
request could overwrite
video_latent_bfloat16.rawand merge green.Four mutations on a scratch copy, each red, the unmutated copy green:
The second is the one worth naming: the digest file beside the artefacts cannot
anchor itself, and what catches it is the third anchor, the sha256 the dump
manifest carries because the worker computed it before anything was copied out.
The mutations ran in a scratch skeleton outside the worktree;
git statusisempty and every golden's sha256 is unchanged.
The step is unconditional in
agent-record, which carries no job-levelif:.The suite is standard library only --
hashlib,json,re,pathlib-- overfour committed files, 0.02 s, no numpy, no build, no GPU, no lease, no
checkpoint, no network. It needs no guard, and a guard is what would have turned
an honest absence into false coverage.
REQUIRED_SUITE_REGISTRATIONSgains the suite in the same change, because thatis the only moment the widening is free: seeded with one entry by #1833, it is
the mechanism that spec says a later row widens rather than replaces. Deleting
the new CI step now prints
pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane; deleting the preflight entry prints the preflight half.Without it the step this change adds would itself have been deletable at rc=0,
which is the #1833 finding restored. This is not the symmetric population rule,
which still belongs to #408 and #1509.
The checker widening now carries its own red-before
The first revision of this pull request failed
pr-size(job 100149327523):scripts/check-pr-size.py:616runs the HEAD test module against the BASEchecker and requires it to FAIL. The only test edit that revision made was
adding
test_ltx2_oracle_goldensto thePASSING_CIandPASSING_PREFLIGHTfixtures. That edit is additive and symmetric, so the base checker -- which pins
only
test_sglang_lease_identity-- is indifferent to it, and no test in themodule exercised the new pin at all. The edit existed to KEEP
test_complete_preflight_and_ci_wiring_passesgreen, which is precisely whatthe contract rejects: a test that cannot fail for the intended reason is not
evidence. Reproduced the way the gate reproduces it, by installing
beafb46df:scripts/check-test-registration.pyover the head checker in adetached worktree at
c2595e186and running the head module:Ran 70 tests,OK, rc 0.Two behavioural cases now assert the checker's OUTPUT on a mutated input, in the
shape
M56/M57already use for the first pinned suite. They drop the ltx2step from the CI fixture while preflight keeps it and require
pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane, thenthe preflight mirror. Both carry an
assertNotEqualguard, so areplacethatstopped matching reds instead of reading as a pass.
They do not assert
"test_ltx2_oracle_goldens" in REQUIRED_SUITE_REGISTRATIONS.Asserting against the constant is a tautology that would let the constant change
freely. They are also deliberately not named
test_M*, becausecheck-test-registration.py:1007requires thetest_M*set to equal thedigest-pinned manifest exactly: an M-number would drag
check_test_registration_mutations.txtandMUTATION_MANIFEST_SHA256into achange that is not about the manifest. All 57
test_M*names are byte-identicalto
beafb46df, and the manifest still hashes to4589e7a96d839a8309fd0e5799cc7a652b22a9d568d56151e2a1b8feea5d02ee.One worktree, one invocation, only the checker swapped:
Both failures read
AssertionError: False is not true : []. The base checkeremits no error at all for the missing registration, which is the finding stated
as a test.
check-pr-size.py, run asagent-recordruns it, on the same base and the twoheads:
The old head is the positive control: the same command, the same base, and the
red is still there, so the green is the change and not the invocation.
Why the local gates were silent
check-pr-size.pyis named in neitherscripts/agent-preflight.shnorscripts/agent-ready.py.agent-ready.pydiscovers it and reportsSKIP check-pr-size.py -- needs arguments preflight does not supply, so a localpreflight rc 0says nothing whatever about a checker change. The firstrevision's body read that skip as "for want of ... an open pull request"; the
reason is the missing
--base/--head, and a skip is never a pass. It has tobe run by hand, with both revisions, which is what the block above does.
The row ID names nothing, and that is correct
ENG-CI-ORACLE-GOLDENSappears nowhere under.agents/ordocs/-- no spec,no matrix row -- verified with a positive control (
ENG-RECORD-CONFLICT-SURFACESreturns three files from the same command). It is a branch and commit-scope
label, not a roadmap row, and #2553's
Row: \-`is the accurate record: the issue is fixed in this same flow and closed by this pull request, so it owes no owning row and no spec## Owed` entry. Opening a row and a spec for a CI stepthat lands complete would add a record surface with no future, which is the
shape "Records" tells us not to create. No gate is violated either way; the
label and the issue now agree in writing rather than only in fact.
Lane evidence
The new step has been observed green on the forge, not only argued to be
runnable:
agent-recordjob 100149357298 atc2595e186reportsThe committed LTX-2 oracle goldens still hash to what records them (#2553) -> success, beside the four LTX steps that already lived in that job. This headmoves it, so the step is re-run on the merge commit before this lands.
scripts/agent-preflight.shrc 0 on the working tree and rc 0--staged.agent-ready.pyreports no FAIL and exits 1 only on--fail-on-skip, forcheck-arm-isa-build,check-cpu-isa-build,check-cuda-fat-gencode,check-pr-sizeandcheck-triton-aot-multiarch.check-tree-compilescompiled158 of 158 translation units.
FOLLOWING_AGENTS_PROTOCOL
Closes #2553
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]