Skip to content

symmetry: a missing artifact makes a read test Incomplete, never Passed - #145

Merged
stevevanhooser merged 1 commit into
mainfrom
claude/clever-bardeen-hf5jgx
Oct 21, 2026
Merged

stevevanhooser merged 1 commit into
mainfrom
claude/clever-bardeen-hf5jgx

Conversation

@stevevanhooser

@stevevanhooser stevevanhooser commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

MATLAB half of the fix for VH-Lab/NDR-python#34. The NDR-python half is VH-Lab/NDR-python#35, on the same branch name, claude/clever-bardeen-hf5jgx, so its symmetry workflow pairs with this one. Merge this one first.

The problem

When the other suite's artifact directory, or a JSON file inside it, was missing, the +readArtifacts tests ended with disp(...) + return. A return makes matlab.unittest report the test as Passed, so a comparison that never ran looked exactly like one that ran and matched. That is one of the three ways VH-Lab/NDR-python#33's first symmetry run went green with every video comparison unrun.

Changes

  • All 7 read tests: every silent path is now an assumption, which matlab.unittest records as Incomplete:

    • the directory guard (7 places) is now assumeTrue(isfolder(artifactDir), ...);
    • the "metadata.json / readFrames.json / readData.json not found" else branches (10 places) are now assumeFail(...);
    • SpikeGadgets' early return when readData.json is missing is now assumeTrue(isfile(...)).

    On a machine with only one language port, nothing fails. The result simply isn't Passed.

  • .github/workflows/test-symmetry.yml:

    • Python never runs in this workflow, so the pythonArtifacts cases are deselected by name, and the log says how many.
    • Then the workflow fails on any Incomplete result, so an Incomplete here always means a comparison that should have run.
  • +readArtifacts/INSTRUCTIONS.md and the readOmeZarr.m header: both said to "skip silently". They now give the rule above, and explain why a return is wrong.

NDR-python's workflow applies the same rule to its four stages. It also declares one known gap: readOmeZarr has no Python half, so its pythonArtifacts case is deselected there and logged.

Verification

I wrote this without access to MATLAB. The edits are mechanical, and no disp/return path remains in +readArtifacts. Two CI runs check them:

If an Incomplete shows up in either run, the log lists it as INCOMPLETE: <test name>.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QjkvVXsdCVw6NNe5SnGNyX

The readArtifacts tests ended with disp(...) + return when the other
suite's artifact directory or a JSON file inside it was missing. A return
makes matlab.unittest report the test as Passed, so a comparison that
never ran read exactly like one that ran and matched. That is how a
missing counterpart could hide behind a green build (VH-Lab/NDR-python#34).

- Every such path is now an assumption: assumeTrue(isfolder(artifactDir))
  for the directory guard (7 tests), assumeFail in the "file not found"
  else branches (10), assumeTrue for SpikeGadgets' readData.json early
  return. A machine with only one port installed still sees no failures;
  the result is Incomplete instead of Passed.
- test-symmetry.yml deselects, by name, the pythonArtifacts cases (Python
  never runs in this workflow) and then fails on ANY Incomplete result, so
  an Incomplete in CI always means a comparison that should have run.
- readArtifacts/INSTRUCTIONS.md and readOmeZarr's header no longer tell
  authors to "skip silently".

Paired with the NDR-python change on the same branch name, which applies
the same rule to its four symmetry stages.

Written without access to MATLAB; the symmetry workflows run it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QjkvVXsdCVw6NNe5SnGNyX
@github-actions

Copy link
Copy Markdown
Contributor

Test Results

415 tests  ±0   405 ✅  - 7   34s ⏱️ +8s
 54 suites ±0    10 💤 +7 
  1 files   ±0     0 ❌ ±0 

Results for commit 48c7625. ± Comparison against base commit a45a4e6.

This pull request skips 7 tests.
ndr.symmetry.readArtifacts.format.readOmeZarr ‑ testReadOmeZarrArtifacts(SourceType=pythonArtifacts)
ndr.symmetry.readArtifacts.format.readSpikeGadgets ‑ testReadSpikeGadgetsArtifacts(SourceType=pythonArtifacts)
ndr.symmetry.readArtifacts.reader.readData ‑ testReadDataArtifacts(SourceType=pythonArtifacts)
ndr.symmetry.readArtifacts.reader.readPrairieview ‑ testReadPrairieviewArtifacts(SourceType=pythonArtifacts)
ndr.symmetry.readArtifacts.reader.readTiffstack ‑ testReadTiffstackArtifacts(SourceType=pythonArtifacts)
ndr.symmetry.readArtifacts.reader.readVideo ‑ testReadVideoArtifacts(SourceType=pythonArtifacts)
ndr.symmetry.readArtifacts.reader.readVld ‑ testReadVldArtifacts(SourceType=pythonArtifacts)

@codecov

codecov Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.94%. Comparing base (a45a4e6) to head (48c7625).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #145   +/-   ##
=======================================
  Coverage   54.94%   54.94%           
=======================================
  Files         164      164           
  Lines        9598     9598           
=======================================
  Hits         5274     5274           
  Misses       4324     4324           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stevevanhooser
stevevanhooser merged commit e2e96be into main Oct 21, 2026
8 checks passed
@stevevanhooser
stevevanhooser deleted the claude/clever-bardeen-hf5jgx branch October 21, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants