Repository navigation
symmetry: port the OME-Zarr symmetry test (readOmeZarr) to Python - #37
Merged
Merged
Conversation
NDR-matlab's readOmeZarr symmetry test had no Python half: Python never read the OME-Zarr store MATLAB writes, and MATLAB's pythonArtifacts case was deselected in Stage 4 as a declared gap. - tests/_omezarr_fixture.py: codec="blosc-zstd" writes every array through zarr-python (zarr v2, numcodecs Blosc, Zstd level 5, byte shuffle), the codec the lab's data uses and makeExampleFixture.m writes. The default stays uncompressed, so the unit tests are unchanged. - tests/symmetry/conftest.py: matlab_sha256(), the SHA-256 of an array's bytes in MATLAB's typecast(x(:)) order (Fortran order, little-endian). Both readArrays index c,z,y,x identically; only the flattening differs. - make_artifacts/format/test_read_omezarr.py: writes example.zarr, metadata.json and checksums.json (the MATLAB half's 8 regions) after checking readArray against the fixture's ground truth. - read_artifacts/format/test_read_omezarr.py: checks every region's digest against the other port's, and that the store is Blosc+Zstd. - test-symmetry.yml: drop the readOmeZarr deselection from Stage 4, and install .[dev,formats] so zarr/numcodecs are present (a missing one fails under strict mode). - test_omezarr_reader.py: the blosc-zstd fixture is real zarr v2 Blosc+Zstd (some frames genuinely Zstd-compressed, not just memcpyed: the level-0 noise is incompressible), reads back as ground truth, and rejects an unknown codec. The digest convention was checked against a literal numpy replay of MATLAB readArray.m (reshape(flip(chunkShape)), permute, got(:)): equal on all three arrays tried, and a C-order digest differs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QjkvVXsdCVw6NNe5SnGNyX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #36. NDR-matlab's
readOmeZarrsymmetry test now has a Python half. The MATLAB half already exists, so there is no NDR-matlab counterpart to merge.Changes
tests/_omezarr_fixture.py: a newcodec="blosc-zstd"option writes every array through zarr-python: zarr v2,numcodecs.Blosc(cname="zstd", clevel=5, shuffle=SHUFFLE). That is the codec the lab's data uses, and the onemakeExampleFixture.mwrites. Writing via zarr-python means MATLAB's hand-rolled Blosc/Zstd reader is checked against genuine zarr-python output. The default stays uncompressed, so the existing unit tests are unchanged.tests/symmetry/conftest.py: addsmatlab_sha256(), the SHA-256 in MATLAB'stypecast(x(:), 'uint8')byte order. BothreadArrays indexc,z,y,xidentically (MATLABx(i,j,k,l)is numpyx[i-1,j-1,k-1,l-1]); only the flattening differs, becausex(:)is column-major.make_artifacts/format/test_read_omezarr.py: writesexample.zarr,metadata.jsonandchecksums.json, with the MATLAB half's 8 regions: whole levels, one chunk, a straddle, and the x-edge chunk. It first checksreadArrayagainst the fixture's ground truth.read_artifacts/format/test_read_omezarr.py: checks every region's digest against the other port's checksums, and checks the store really is Blosc+Zstd. A mismatch names the regions.test-symmetry.yml:noPythonHalf = ["readOmeZarr"]deselection that symmetry: a comparison that could not run fails CI instead of passing #35 declared in Stage 4, so MATLAB now reads Python's store;.[dev,formats]sozarrandnumcodecsare present. Under strict mode a missing one fails rather than skips.test_omezarr_reader.py(runs in the normal matrix): the Blosc+Zstd fixture is real zarr v2 Blosc+Zstd, reads back as the ground truth, and rejects an unknown codec.Two things found along the way
readArray.m's operations literally in numpy, using column-major semantics:typecast,reshape(linear, flip(chunkShape)),permute(…, nd:-1:1), place the chunk,typecast(got(:)). The digest matchedmatlab_sha256(readArray(…))on all three arrays tried. A C-order digest differs, so the comparison can tell the two orders apart.uint16noise, which is incompressible, so Blosc stores those chunks verbatim (the MEMCPYED flag) and Zstd never runs for them. That's 10 of 18 chunks. The other 8 (downsampled levels and zero-padded edge chunks) are genuinely Zstd-compressed, so both of Blosc's decode paths get exercised. MATLAB's fixture uses the same kind of data (randi(rs, 65535, …)), so it has the same mix. The unit test asserts that genuine Zstd frames exist, so the Zstd path can't silently drop out.Verification
pytest: 728 passed (+3), 15 skipped (unchanged), 24 xfailed.blackandruff checkare clean. Bridge checks (strict): 191 passed.SHA-256 mismatch against matlabArtifacts for: straddle (mean level 1).Generated by Claude Code