Skip to content

symmetry: port the OME-Zarr symmetry test (readOmeZarr) to Python - #37

Merged
stevevanhooser merged 1 commit into
mainfrom
claude/clever-bardeen-hf5jgx
Oct 21, 2026
Merged

stevevanhooser merged 1 commit into
mainfrom
claude/clever-bardeen-hf5jgx

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Fixes #36. NDR-matlab's readOmeZarr symmetry test now has a Python half. The MATLAB half already exists, so there is no NDR-matlab counterpart to merge.

Changes

  • tests/_omezarr_fixture.py: a new codec="blosc-zstd" option writes every array through zarr-python: zarr v2, numcodecs.Blosc(cname="zstd", clevel=5, shuffle=SHUFFLE). That is the codec the lab's data uses, and the one makeExampleFixture.m writes. Writing via zarr-python means MATLAB's hand-rolled Blosc/Zstd reader is checked against genuine zarr-python output. The default stays uncompressed, so the existing unit tests are unchanged.
  • tests/symmetry/conftest.py: adds matlab_sha256(), the SHA-256 in MATLAB's typecast(x(:), 'uint8') byte order. Both readArrays index c,z,y,x identically (MATLAB x(i,j,k,l) is numpy x[i-1,j-1,k-1,l-1]); only the flattening differs, because x(:) is column-major.
  • make_artifacts/format/test_read_omezarr.py: writes example.zarr, metadata.json and checksums.json, with the MATLAB half's 8 regions: whole levels, one chunk, a straddle, and the x-edge chunk. It first checks readArray against the fixture's ground truth.
  • read_artifacts/format/test_read_omezarr.py: checks every region's digest against the other port's checksums, and checks the store really is Blosc+Zstd. A mismatch names the regions.
  • test-symmetry.yml:
  • test_omezarr_reader.py (runs in the normal matrix): the Blosc+Zstd fixture is real zarr v2 Blosc+Zstd, reads back as the ground truth, and rejects an unknown codec.

Two things found along the way

  • The digest convention was the risky assumption, so I checked it without MATLAB. I replayed readArray.m's operations literally in numpy, using column-major semantics: typecast, reshape(linear, flip(chunkShape)), permute(…, nd:-1:1), place the chunk, typecast(got(:)). The digest matched matlab_sha256(readArray(…)) on all three arrays tried. A C-order digest differs, so the comparison can tell the two orders apart.
  • Random data partly bypasses Zstd. The level-0 volume is seeded uniform uint16 noise, which is incompressible, so Blosc stores those chunks verbatim (the MEMCPYED flag) and Zstd never runs for them. That's 10 of 18 chunks. The other 8 (downsampled levels and zero-padded edge chunks) are genuinely Zstd-compressed, so both of Blosc's decode paths get exercised. MATLAB's fixture uses the same kind of data (randi(rs, 65535, …)), so it has the same mix. The unit test asserts that genuine Zstd frames exist, so the Zstd path can't silently drop out.

Verification

  • pytest: 728 passed (+3), 15 skipped (unchanged), 24 xfailed. black and ruff check are clean. Bridge checks (strict): 191 passed.
  • Strict and CI-shaped, by hand:
    • Stage 3: 7 passed, including OME-Zarr.
    • Stage 2 (with Python's artifacts standing in for MATLAB's): 17 passed, 17 deselected, 0 skipped.
    • Corrupting one checksum fails with SHA-256 mismatch against matlabArtifacts for: straddle (mean level 1).
  • Real MATLAB, in CI only: the cross-language run is the first time each language reads the other's actual OME-Zarr store. That is MATLAB's real fixture in Stage 2, and MATLAB reading the zarr-python store in Stage 4.

Generated by Claude Code

NDR-matlab's readOmeZarr symmetry test had no Python half: Python never
read the OME-Zarr store MATLAB writes, and MATLAB's pythonArtifacts case
was deselected in Stage 4 as a declared gap.

- tests/_omezarr_fixture.py: codec="blosc-zstd" writes every array
  through zarr-python (zarr v2, numcodecs Blosc, Zstd level 5, byte
  shuffle), the codec the lab's data uses and makeExampleFixture.m
  writes. The default stays uncompressed, so the unit tests are unchanged.
- tests/symmetry/conftest.py: matlab_sha256(), the SHA-256 of an array's
  bytes in MATLAB's typecast(x(:)) order (Fortran order, little-endian).
  Both readArrays index c,z,y,x identically; only the flattening differs.
- make_artifacts/format/test_read_omezarr.py: writes example.zarr,
  metadata.json and checksums.json (the MATLAB half's 8 regions) after
  checking readArray against the fixture's ground truth.
- read_artifacts/format/test_read_omezarr.py: checks every region's
  digest against the other port's, and that the store is Blosc+Zstd.
- test-symmetry.yml: drop the readOmeZarr deselection from Stage 4, and
  install .[dev,formats] so zarr/numcodecs are present (a missing one
  fails under strict mode).
- test_omezarr_reader.py: the blosc-zstd fixture is real zarr v2
  Blosc+Zstd (some frames genuinely Zstd-compressed, not just memcpyed:
  the level-0 noise is incompressible), reads back as ground truth, and
  rejects an unknown codec.

The digest convention was checked against a literal numpy replay of
MATLAB readArray.m (reshape(flip(chunkShape)), permute, got(:)): equal
on all three arrays tried, and a C-order digest differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QjkvVXsdCVw6NNe5SnGNyX
@stevevanhooser
stevevanhooser merged commit 321d31e into main Oct 21, 2026
7 checks passed
@stevevanhooser
stevevanhooser deleted the claude/clever-bardeen-hf5jgx branch October 21, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port the OME-Zarr symmetry test (readOmeZarr) to Python

2 participants