Skip to content

ci(ENG-CI-ORACLE-GOLDENS): run the oracle-goldens digests on a lane, not only on a developer box - #2555

Merged
mudler merged 3 commits into
mainfrom
row/ENG-CI-ORACLE-GOLDENS
Sep 2, 2026
Merged

ci(ENG-CI-ORACLE-GOLDENS): run the oracle-goldens digests on a lane, not only on a developer box#2555
mudler merged 3 commits into
mainfrom
row/ENG-CI-ORACLE-GOLDENS

Conversation

@localai-org-maint-bot

@localai-org-maint-bot localai-org-maint-bot commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

ci(ENG-CI-ORACLE-GOLDENS): run the oracle-goldens digests on a lane, not only on a developer box

tests/scripts/test_ltx2_oracle_goldens.py was in scripts/agent-preflight.sh's
SUITES array and in no step of .github/workflows/ci.yml. Read with the
checker's own _active_ci_commands at beafb46df, which parses the file the way
a lane does rather than the way a grep does: 110 active commands, the goldens
suite absent, test_ltx2_latent_dump present as the positive control.

#2520 made that gap expensive. The suite now also covers
tests/parity/goldens/ltx2_oracle_latent/ -- the Phase A latents a future
LTX-2 cross-decode reads INSTEAD of taking a GPU lease. Their whole value is
that they are the bytes the leased run produced, and until this step a pull
request could overwrite video_latent_bfloat16.raw and merge green.

Four mutations on a scratch copy, each red, the unmutated copy green:

one byte flipped in video_latent_bfloat16.raw   -> 2 FAILs, rc 1
that byte flipped AND SHA256SUMS rewritten      -> 1 FAIL,  rc 1
an unrecorded artefact added to the directory   -> 1 FAIL,  rc 1
upstream-render.mp4 truncated by one byte       -> 2 FAILs, rc 1

The second is the one worth naming: the digest file beside the artefacts cannot
anchor itself, and what catches it is the third anchor, the sha256 the dump
manifest carries because the worker computed it before anything was copied out.
The mutations ran in a scratch skeleton outside the worktree; git status is
empty and every golden's sha256 is unchanged.

The step is unconditional in agent-record, which carries no job-level if:.
The suite is standard library only -- hashlib, json, re, pathlib -- over
four committed files, 0.02 s, no numpy, no build, no GPU, no lease, no
checkpoint, no network. It needs no guard, and a guard is what would have turned
an honest absence into false coverage.

REQUIRED_SUITE_REGISTRATIONS gains the suite in the same change, because that
is the only moment the widening is free: seeded with one entry by #1833, it is
the mechanism that spec says a later row widens rather than replaces. Deleting
the new CI step now prints pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane; deleting the preflight entry prints the preflight half.
Without it the step this change adds would itself have been deletable at rc=0,
which is the #1833 finding restored. This is not the symmetric population rule,
which still belongs to #408 and #1509.

The checker widening now carries its own red-before

The first revision of this pull request failed pr-size (job 100149327523):

ERROR: BASE checker stayed green for 'scripts/check-test-registration.py';
changed test is not semantic evidence

scripts/check-pr-size.py:616 runs the HEAD test module against the BASE
checker and requires it to FAIL. The only test edit that revision made was
adding test_ltx2_oracle_goldens to the PASSING_CI and PASSING_PREFLIGHT
fixtures. That edit is additive and symmetric, so the base checker -- which pins
only test_sglang_lease_identity -- is indifferent to it, and no test in the
module exercised the new pin at all. The edit existed to KEEP
test_complete_preflight_and_ci_wiring_passes green, which is precisely what
the contract rejects: a test that cannot fail for the intended reason is not
evidence. Reproduced the way the gate reproduces it, by installing
beafb46df:scripts/check-test-registration.py over the head checker in a
detached worktree at c2595e186 and running the head module: Ran 70 tests,
OK, rc 0.

Two behavioural cases now assert the checker's OUTPUT on a mutated input, in the
shape M56/M57 already use for the first pinned suite. They drop the ltx2
step from the CI fixture while preflight keeps it and require
pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane, then
the preflight mirror. Both carry an assertNotEqual guard, so a replace that
stopped matching reds instead of reading as a pass.

They do not assert "test_ltx2_oracle_goldens" in REQUIRED_SUITE_REGISTRATIONS.
Asserting against the constant is a tautology that would let the constant change
freely. They are also deliberately not named test_M*, because
check-test-registration.py:1007 requires the test_M* set to equal the
digest-pinned manifest exactly: an M-number would drag
check_test_registration_mutations.txt and MUTATION_MANIFEST_SHA256 into a
change that is not about the manifest. All 57 test_M* names are byte-identical
to beafb46df, and the manifest still hashes to
4589e7a96d839a8309fd0e5799cc7a652b22a9d568d56151e2a1b8feea5d02ee.

One worktree, one invocation, only the checker swapped:

BASE checker (beafb46df) + head module -> Ran 72 tests, FAILED (failures=2), rc 1
HEAD checker             + head module -> Ran 72 tests, OK,                  rc 0

Both failures read AssertionError: False is not true : []. The base checker
emits no error at all for the missing registration, which is the finding stated
as a test.

check-pr-size.py, run as agent-record runs it, on the same base and the two
heads:

--head c2595e186 (before) -> rc 1, BASE checker stayed green ...
--head 00c57e3c4 (after)  -> rc 0, OK: every explicit path class is within its review budget.

The old head is the positive control: the same command, the same base, and the
red is still there, so the green is the change and not the invocation.

Why the local gates were silent

check-pr-size.py is named in neither scripts/agent-preflight.sh nor
scripts/agent-ready.py. agent-ready.py discovers it and reports
SKIP check-pr-size.py -- needs arguments preflight does not supply, so a local
preflight rc 0 says nothing whatever about a checker change. The first
revision's body read that skip as "for want of ... an open pull request"; the
reason is the missing --base/--head, and a skip is never a pass. It has to
be run by hand, with both revisions, which is what the block above does.

The row ID names nothing, and that is correct

ENG-CI-ORACLE-GOLDENS appears nowhere under .agents/ or docs/ -- no spec,
no matrix row -- verified with a positive control (ENG-RECORD-CONFLICT-SURFACES
returns three files from the same command). It is a branch and commit-scope
label, not a roadmap row, and #2553's Row: \-`is the accurate record: the issue is fixed in this same flow and closed by this pull request, so it owes no owning row and no spec## Owed` entry. Opening a row and a spec for a CI step
that lands complete would add a record surface with no future, which is the
shape "Records" tells us not to create. No gate is violated either way; the
label and the issue now agree in writing rather than only in fact.

Lane evidence

The new step has been observed green on the forge, not only argued to be
runnable: agent-record job 100149357298 at c2595e186 reports
The committed LTX-2 oracle goldens still hash to what records them (#2553) -> success, beside the four LTX steps that already lived in that job. This head
moves it, so the step is re-run on the merge commit before this lands.

scripts/agent-preflight.sh rc 0 on the working tree and rc 0 --staged.
agent-ready.py reports no FAIL and exits 1 only on --fail-on-skip, for
check-arm-isa-build, check-cpu-isa-build, check-cuda-fat-gencode,
check-pr-size and check-triton-aot-multiarch. check-tree-compiles compiled
158 of 158 translation units.

FOLLOWING_AGENTS_PROTOCOL

Closes #2553

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…not only on a developer box

`tests/scripts/test_ltx2_oracle_goldens.py` was in `scripts/agent-preflight.sh`'s
`SUITES` array and in no step of `.github/workflows/ci.yml`. Read with the
checker's own `_active_ci_commands` at `beafb46df`, which parses the file the way
a lane does rather than the way a grep does: 110 active commands, the goldens
suite absent, `test_ltx2_latent_dump` present as the positive control.

#2520 made that gap expensive. The suite now also covers
`tests/parity/goldens/ltx2_oracle_latent/` -- the Phase A latents a future
LTX-2 cross-decode reads INSTEAD of taking a GPU lease. Their whole value is
that they are the bytes the leased run produced, and until this step a pull
request could overwrite `video_latent_bfloat16.raw` and merge green.

Four mutations on a scratch copy, each red, the unmutated copy green:

  one byte flipped in `video_latent_bfloat16.raw`  -> 2 FAILs, rc 1
  that byte flipped AND `SHA256SUMS` rewritten     -> 1 FAIL,  rc 1
  an unrecorded artefact added to the directory    -> 1 FAIL,  rc 1
  `upstream-render.mp4` truncated by one byte      -> 2 FAILs, rc 1

The second is the one worth naming: the digest file beside the artefacts cannot
anchor itself, and what catches it is the third anchor, the sha256 the dump
manifest carries because the worker computed it before anything was copied out.

The step is unconditional in `agent-record`, which carries no job-level `if:`.
The suite is standard library only -- `hashlib`, `json`, `re`, `pathlib` -- over
four committed files, 0.02 s, no numpy, no build, no GPU, no lease, no
checkpoint, no network. It needs no guard, and a guard is what would have turned
an honest absence into false coverage.

`REQUIRED_SUITE_REGISTRATIONS` gains the suite in the same change, because that
is the only moment the widening is free: seeded with one entry by #1833, it is
the mechanism that spec says a later row widens rather than replaces. Deleting
the new CI step now prints `pinned suite test_ltx2_oracle_goldens is missing
from the CI suite lane`; deleting the preflight entry prints the preflight half.
Without it the step this commit adds would itself have been deletable at rc=0,
which is the #1833 finding restored. This is not the symmetric population rule,
which still belongs to #408 and #1509.

The synthetic fixtures in `test_check_test_registration.py` gain the second
pinned suite in both lanes so `test_complete_preflight_and_ci_wiring_passes`
still reads `[]`. No `test_M*` method name changes, so the digest-pinned
mutation manifest is untouched. 70 tests OK.

FOLLOWING_AGENTS_PROTOCOL

Closes #2553

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…against the base checker

`pr-size` was FAILURE on #2555 (job 100149327523):

    ERROR: BASE checker stayed green for 'scripts/check-test-registration.py';
    changed test is not semantic evidence

`scripts/check-pr-size.py:616` runs the HEAD test module against the BASE
checker and requires it to FAIL. The previous commit's only test edit added
`test_ltx2_oracle_goldens` to `PASSING_CI` and `PASSING_PREFLIGHT`. That edit is
additive and symmetric, so the base checker -- which pins only
`test_sglang_lease_identity` -- is indifferent to it, and nothing in the module
exercised the new pin at all. Its job was to KEEP
`test_complete_preflight_and_ci_wiring_passes` green, which is exactly what the
contract rejects: a test that cannot fail for the intended reason is not
evidence.

Reproduced before the fix, the way the gate reproduces it: install
`beafb46df:scripts/check-test-registration.py` over the head checker in a
detached worktree at c2595e1 and run the head module.

    Ran 70 tests in 6.196s
    OK
    rc=0

Two behavioural cases now assert the checker's OUTPUT on a mutated input, in the
shape M56/M57 already use for the first pinned suite: drop the ltx2 step from the
CI fixture while preflight keeps it, and require
`pinned suite test_ltx2_oracle_goldens is missing from the CI suite lane`; then
the preflight mirror. Both carry the `assertNotEqual` guard, so a `replace` that
stopped matching reds instead of reading as a pass.

They do not assert `"test_ltx2_oracle_goldens" in REQUIRED_SUITE_REGISTRATIONS`.
Asserting against the constant is a tautology that would let the constant change
freely. They are also deliberately not named `test_M*`:
`check-test-registration.py:1007` requires the `test_M*` set to equal the
digest-pinned manifest exactly, so an M-number would drag
`check_test_registration_mutations.txt` and `MUTATION_MANIFEST_SHA256` into a
change that is not about the manifest. All 57 `test_M*` names are byte-identical
to `beafb46df`, and the manifest still hashes to
4589e7a96d839a8309fd0e5799cc7a652b22a9d568d56151e2a1b8feea5d02ee.

Same worktree, same invocation, after the fix:

    BASE checker  -> Ran 72 tests, FAILED (failures=2), rc=1
    HEAD checker  -> Ran 72 tests, OK, rc=0

Both failures read `AssertionError: False is not true : []` -- the base checker
emits no error at all for the missing registration, which is the finding stated
as a test.

Note the local gates could not have caught this: `check-pr-size` is invoked from
neither `scripts/agent-preflight.sh` nor `scripts/agent-ready.py`. It is CI-only,
so `preflight rc=0` is not sufficient evidence for a checker change.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
… whose build-test-cpu red this PR inherited

CI builds the merge commit, so every verdict on #2555 was measured against a
main whose build-test-cpu was red repo-wide on the rank guard in
`dense_device_glue.h:52`. That defect is fixed on main by 33b08e4, in the
test fixture's two rank-5 paged-KV scratch buffers, so this merge re-bases the
pull request onto a green trunk and the next push measures this change rather
than the trunk it sat on.

Nothing this pull request touches moved on main across the merged range:
`.github/workflows/ci.yml`, `scripts/check-test-registration.py`,
`tests/scripts/test_check_test_registration.py`,
`tests/scripts/test_ltx2_oracle_goldens.py` and `scripts/agent-preflight.sh`
are byte-identical to the merge base, and so are the goldens the pinned suite
digests. The merge is therefore a fast-forward of the base only, and it
falsifies none of the prose the change carries.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@mudler
mudler merged commit 6974557 into main Sep 2, 2026
24 of 27 checks passed
@mudler
mudler deleted the row/ENG-CI-ORACLE-GOLDENS branch September 2, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_ltx2_oracle_goldens gates in preflight and on NO CI lane, so a corrupted oracle golden is green on the forge

2 participants