Repository navigation
Port the OME-Zarr symmetry test (readOmeZarr) to Python #36
Description
NDR-matlab has a cross-language symmetry test for ndr.format.omezarr.readArray (tools/tests/+ndr/+symmetry/{+makeArtifacts,+readArtifacts}/+format/readOmeZarr.m). NDR-python has no half of it, so in test-symmetry.yml:
- Stage 2: Python never reads the OME-Zarr artifacts MATLAB writes. Nothing on the Python side checks that the two readers decode the same store to the same bytes.
- Stage 4: MATLAB's
pythonArtifactscase can't run, so symmetry: a comparison that could not run fails CI instead of passing #35 deselects it by name as a declared known gap (noPythonHalf = ["readOmeZarr"]).
Zarr is lossless, so if the two ports decode the same chunks to different values, one of them is wrong. That is exactly what this test exists to catch, and today it only runs MATLAB-against-MATLAB.
What the MATLAB half does
-
makeArtifacts: builds
example.zarrwithndr.test.format.omezarr.makeExampleFixture(dir, 'WithChunks', true):- a dual pyramid: a shared level-0 array plus
meanandmaxchains, zarr v2; - a seeded
uint16[1 8 8 10]volume in[1 4 4 4]chunks, so there are edge chunks; - Blosc+Zstd compression.
It then writes
metadata.json(fixtureName,level0Shape,level0ChunkShape,dtype,seed) andchecksums.json. The checksums file holds 8 regions (whole arrays at levels 1 and 2 of both pyramids, inside one chunk, straddling chunks, reaching the x edge, inside the x edge). Each region has the SHA-256 oftypecast(got(:), 'uint8'). - a dual pyramid: a shared level-0 array plus
-
readArtifacts: re-reads the other port's fixture with its own reader, and checks each region's SHA-256.
What the Python half needs
- The same SHA, from the Python side. Both
readArrays return the stored axis order (c,z,y,x) with a 1-based, inclusiveRegion = [start; stop]. MATLAB places chunks sogot(i,j,k,l) == zarr[i-1,j-1,k-1,l-1]. Butgot(:)is column-major, so the Python digest issha256(np.asarray(arr, "<u2").ravel(order="F").tobytes()). - A Blosc+Zstd fixture written by zarr-python.
tests/_omezarr_fixture.pyalready mirrorsmakeExampleFixture's layout and seed, but writes uncompressed chunks by hand. That suits the unit tests, but MATLAB's own docstring warns that an uncompressed fixture "would prove nothing about the Blosc+Zstd path the lab actually writes". The make side should write the arrays through zarr-python (zarr v2 format,numcodecs.Blosc(cname="zstd", clevel=5, shuffle=SHUFFLE)), the same library that writes the lab's real data. Then MATLAB's hand-rolled reader gets checked against genuine zarr-python output. zarr 3.x's v2 output already matches what MATLAB'sreadZArrayMetaaccepts:"filters": null,"dimension_separator": ".", C order,blosc/zstd. tests/symmetry/make_artifacts/format/test_read_omezarr.pywrites the same three things (example.zarr,metadata.json,checksums.jsonwith the same 8 regions) topythonArtifacts/format/readOmeZarr/testReadOmeZarrArtifacts/.tests/symmetry/read_artifacts/format/test_read_omezarr.py, parametrized oversource_type, checks every region's SHA-256 against the other port's checksums.test-symmetry.yml:- remove the
noPythonHalfdeselection from Stage 4, so MATLAB reads Python's fixture; - install
.[dev,formats]instead of.[dev], sincezarrandnumcodecsare informats. Otherwise strict mode (Symmetry CI passes green when the other language's artifacts are missing (a fourth "skip is a silent pass") #34) turns theimportorskip("zarr")into a failure.
- remove the
Done when
The symmetry job runs readOmeZarr both ways across the languages, with nothing deselected for it:
- Stage 2: Python reads MATLAB's fixture and matches every checksum.
- Stage 4: MATLAB reads Python's fixture and matches every checksum, alongside its own.
Activity