Symmetry CI passes green when the other language's artifacts are missing (a fourth "skip is a silent pass") #34

test-symmetry.yml can report success when the cross-language comparison never ran. This is the same failure AGENTS.md describes under "A skip is a silent pass", which lists three earlier instances in the bridge tests. This would be the fourth, in the symmetry tests, which tests/conftest.py's strict-mode backstop does not cover.

What happened

On #33 (the video reader port), the NDR-python branch was pushed a few seconds before the matching NDR-matlab branch (VH-Lab/NDR-matlab#144). The first symmetry run, 39476490054 attempt 1, resolved NDR-matlab before that branch existed:

ref: main

NDR-matlab main had no readVideo symmetry tests yet. So:

• MATLAB: Stage 1 wrote no video artifacts.
• Python: all 10 test_read_video.py read checks skipped.
• Result: the job was green.

This was caught only by reading the log. After a re-run (attempt 2, ref: claude/clever-bardeen-hf5jgx), every check really ran and passed.

Why it can't be seen from the result

There are three separate ways a check that never ran looks the same as one that passed:

1. The branch fallback is silent. "Resolve NDR-matlab branch" falls back to main when the matching branch is missing, and nothing in the job summary says which it picked. That fallback is right for a Python-only change, but for a paired change it is a race against the order the two branches are pushed.
2. Python read checks skip when artifacts are missing, using pytest.skip(f"No artifacts from {source_type}"). Stage 2 also always skips the pythonArtifacts cases, because Python makes its artifacts in Stage 3, after Stage 2 reads. So every green run already carries a standing set of skips: 16 on main today, including in the post-merge run 39599603916. A real missing-artifact skip hides among them, which is the second instance AGENTS.md warns about.
3. MATLAB read checks report passed, not skipped. +readArtifacts/+reader/*.m (readTiffstack, readPrairieview, readVld, readVideo, …) do disp('... does not exist. Skipping.'); return; when the artifact directory is absent. A return in a matlab.unittest test counts as a pass, so Stage 4's "14 passed, 0 failed" would read the same if Python had produced nothing.

Suggested fix

Follow the bridge tests' pattern: in CI, a check that could not run fails, and it says why.

• A strict flag in CI: set e.g. NDR_SYMMETRY_CHECK_STRICT=1 in test-symmetry.yml. When it is set, a missing artifact directory fails a read check in both languages:
  • in Python, condition the pytest.skip on the flag, or extend the tests/conftest.py backstop to tests/symmetry/read_artifacts/;
  • in MATLAB, use testCase.assertTrue(isfolder(artifactDir), ...) under the flag, instead of disp + return.
• Expected skips only: the Stage 2 pythonArtifacts cases are expected, so deselect them in that stage (for example -k matlabArtifacts) rather than skip them, as was done for the needs_matlab tests. Any skip that remains then means something.
• Make the branch choice visible: print the resolved NDR-matlab ref and why (matching branch found, or fallback to main) to $GITHUB_STEP_SUMMARY. Optionally warn when a PR adds a tests/symmetry/*/test_read_*.py while NDR-matlab resolved to main.
• Write it down: add this as the fourth entry in AGENTS.md's "A skip is a silent pass" list.

Meanwhile

For paired changes, push the NDR-matlab branch first and merge the NDR-matlab PR first. NDR-matlab's main then carries its half before NDR-python's runs depend on it. After any paired change, check the symmetry log for the ref: line rather than trusting the green check.