Repository navigation
symmetry: a missing artifact makes a read test Incomplete, never Passed - #145
Merged
Merged
Conversation
The readArtifacts tests ended with disp(...) + return when the other suite's artifact directory or a JSON file inside it was missing. A return makes matlab.unittest report the test as Passed, so a comparison that never ran read exactly like one that ran and matched. That is how a missing counterpart could hide behind a green build (VH-Lab/NDR-python#34). - Every such path is now an assumption: assumeTrue(isfolder(artifactDir)) for the directory guard (7 tests), assumeFail in the "file not found" else branches (10), assumeTrue for SpikeGadgets' readData.json early return. A machine with only one port installed still sees no failures; the result is Incomplete instead of Passed. - test-symmetry.yml deselects, by name, the pythonArtifacts cases (Python never runs in this workflow) and then fails on ANY Incomplete result, so an Incomplete in CI always means a comparison that should have run. - readArtifacts/INSTRUCTIONS.md and readOmeZarr's header no longer tell authors to "skip silently". Paired with the NDR-python change on the same branch name, which applies the same rule to its four symmetry stages. Written without access to MATLAB; the symmetry workflows run it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjkvVXsdCVw6NNe5SnGNyX
Contributor
Test Results415 tests ±0 405 ✅ - 7 34s ⏱️ +8s Results for commit 48c7625. ± Comparison against base commit a45a4e6. This pull request skips 7 tests. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #145 +/- ##
=======================================
Coverage 54.94% 54.94%
=======================================
Files 164 164
Lines 9598 9598
=======================================
Hits 5274 5274
Misses 4324 4324 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MATLAB half of the fix for VH-Lab/NDR-python#34. The NDR-python half is VH-Lab/NDR-python#35, on the same branch name,
claude/clever-bardeen-hf5jgx, so its symmetry workflow pairs with this one. Merge this one first.The problem
When the other suite's artifact directory, or a JSON file inside it, was missing, the
+readArtifactstests ended withdisp(...)+return. Areturnmakesmatlab.unittestreport the test as Passed, so a comparison that never ran looked exactly like one that ran and matched. That is one of the three ways VH-Lab/NDR-python#33's first symmetry run went green with every video comparison unrun.Changes
All 7 read tests: every silent path is now an assumption, which
matlab.unittestrecords as Incomplete:assumeTrue(isfolder(artifactDir), ...);metadata.json/readFrames.json/readData.jsonnot found"elsebranches (10 places) are nowassumeFail(...);readData.jsonis missing is nowassumeTrue(isfile(...)).On a machine with only one language port, nothing fails. The result simply isn't Passed.
.github/workflows/test-symmetry.yml:pythonArtifactscases are deselected by name, and the log says how many.+readArtifacts/INSTRUCTIONS.mdand thereadOmeZarr.mheader: both said to "skip silently". They now give the rule above, and explain why areturnis wrong.NDR-python's workflow applies the same rule to its four stages. It also declares one known gap:
readOmeZarrhas no Python half, so itspythonArtifactscase is deselected there and logged.Verification
I wrote this without access to MATLAB. The edits are mechanical, and no
disp/returnpath remains in+readArtifacts. Two CI runs check them:If an Incomplete shows up in either run, the log lists it as
INCOMPLETE: <test name>.🤖 Generated with Claude Code
https://claude.ai/code/session_01QjkvVXsdCVw6NNe5SnGNyX