Skip to content

Construct Volume from loaders; load volume meta at loader creation - #499

Open
frasercl wants to merge 33 commits into
refactor/remove-threadable-loaderfrom
refactor/sync-meta-load
Open

frasercl wants to merge 33 commits into
refactor/remove-threadable-loaderfrom
refactor/sync-meta-load

Conversation

@frasercl

@frasercl frasercl commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Review time: medium-large (30-35min)

Problem

This PR takes another step towards making Volume the primary client-facing interface for initiating volume data loads, rather than VolumeLoader, by altering the Volume constructor to require a VolumeLoader.

To enable this, all VolumeLoader methods that previously initiated a load of metadata rather than actual data are now synchronous. This has the effect of moving metadata loading to loader creation. Most loaders therefore now follow the OmeZarrLoader pattern of having a private constructor and an async static method that "opens" the loader. Once a loader is open, it must have all the data it needs to construct metadata objects immediately.

Solution

The loader interface change described above was pretty trivial on most VolumeLoader implementers, but two required more intensive changes:

  • OMEZarrLoader is responsible for computing the extent of a data request and picking a scale level. The metadata that this PR requires it to load in advance is more complex, and the code that loaded the relevant metadata was relatively entangled with the code that converted it into the required types.
  • WorkerLoader is a handle to a VolumeLoader running on a WebWorker, so everything it does involves message passing and is therefore async. This PR enables WorkerLoader to hold onto metadata on the main thread, so it can return results from metadata methods synchronously.

@frasercl
frasercl added this pull request to stack #500 October 14, 2026 01:52
Comment thread public/index.ts Outdated
Comment on lines +1110 to +1116
const channels: number[] = [];
myState.channelGui.forEach(({ enabled }, idx) => {
if (enabled) {
channels.push(idx);
}
});
loader.loadVolumeData(volume, { ...volume.loadSpec, channels }, onChannelDataArrived);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ultimate aim of this stack of PRs is to make this code, which initiates data loading, a call on volume with a simpler signature, rather than a call to loader. This PR doesn't get all the way there and in fact makes this procedure temporarily a bit more complicated; the next PR in this stack is set to clean it up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is helpful context!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Crop preservation, volume-state initialization, and demo-loading regressions remain unresolved.

5 open findings
What changed in this PR

Moves metadata loading to loader creation, allowing Volume to construct synchronously from an initialized loader.

Changes:

  • Replaces asynchronous metadata methods with synchronous getters.
  • Transfers worker-loader metadata to the main thread.
  • Updates volume construction, demo usage, and tests.
File Description
src/​workers/​VolumeLoadWorker.ts Returns metadata when creating loaders.
src/​workers/​VolumeLoaderContext.ts Provides synchronous metadata access.
src/​workers/​types.ts Updates worker messages and metadata types.
src/​VolumeDrawable.ts Reads names through image info.
src/​Volume.ts Constructs from loaders and adds loader replacement.
src/​test/​zarr_utils.test.ts Updates scale-helper input.
src/​test/​volume.test.ts Adapts construction tests and mocks.
src/​test/​TripleSliceVolume.test.ts Uses loader-backed test volumes.
src/​loaders/​zarr_utils/​utils.ts Generalizes scale-helper parameter type.
src/​loaders/​VolumeLoader.ts Revises the metadata interface.
src/​loaders/​TiffLoader.ts Loads metadata during asynchronous creation.
src/​loaders/​RawArrayLoader.ts Makes metadata getters synchronous.
src/​loaders/​OpenCellLoader.ts Makes metadata getters synchronous.
src/​loaders/​OmeZarrLoader.ts Separates metadata extraction and conversion.
src/​loaders/​JsonImageInfoLoader.ts Preloads JSON metadata and exports converters.
src/​loaders/​index.ts Uses asynchronous loader factories.
src/​ImageInfo.ts Adds metadata accessors.
public/​types.ts Makes demo loaders optional.
public/​index.ts Adapts demo construction and loading.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/Volume.ts Resolved
Comment thread src/Volume.ts Resolved
Comment thread src/loaders/OmeZarrLoader.ts Resolved
Comment thread public/index.ts Resolved
Comment thread public/index.ts Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Metadata initialization, load-spec propagation, and shared translation mutation need correction before approval.

1 open finding
5 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Pass loadSpec to volume construction

public/​index.ts:1108

The supplied loadSpec is no longer passed to volume construction. A non-default time or scale-level constraint is used by getDims above but discarded for the actual volume and data load, unlike the previous createVolume(loadSpec, ...) call. Pass it as the constructor's second argument.

Medium severity Prevent mutation of cached transform translation

src/​loaders/​JsonImageInfoLoader.ts:74

jsonToImageInfo now runs against WorkerLoader's cached main-thread metadata, but tr aliases json.transform.translation and is rescaled in place. Creating another volume from the same loader and time point therefore rescales the translation shared by an already-created volume. Previously, worker responses isolated the returned arrays. Copy the translation before rescaling so conversion neither changes cached metadata nor shares this array between volumes.

🧠 Review effort: Balanced

Comment thread src/Volume.ts Resolved
@frasercl
frasercl force-pushed the refactor/sync-meta-load branch from d81c5d0 to f723181 Compare October 15, 2026 18:25
@frasercl
frasercl marked this pull request as ready for review October 16, 2026 22:05
@frasercl
frasercl requested a review from a team as a code owner October 16, 2026 22:06
@ShrimpCryptid
ShrimpCryptid self-requested a review October 19, 2026 17:36
Comment thread src/loaders/VolumeLoaderUtils.ts Resolved
Comment thread src/loaders/JsonImageInfoLoader.ts Resolved
Comment thread src/test/TripleSliceVolume.test.ts Resolved
Comment thread public/index.ts Resolved
Comment thread src/ImageInfo.ts Outdated
z: imageInfo.physicalPixelSize.z + imageInfo.spatialUnit,
};
metadata["Multiresolution levels"] = rawImageInfo.multiscaleLevelDims;
// TODO decide???? combined or not?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we ever decide on whatever this was about?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically: is it okay to combine the channel counts of all separate sources into one channel count?

I think so. I'm unsatisfied with how metadata works for volumes with multiple sources, but people aren't asking for it to be fixed.

Comment thread src/loaders/TiffLoader.ts

constructor(url: string[]) {
private constructor(
private url: string[],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be stored as a remapped URL?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean a URL object? These strings are already passed through remapUri in the actual (async) constructor.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I meant to ask if the string URL should be remapped here for safety. I guess if a user is creating a TiffLoader from scratch, they already should know that their URL works.

This does point me towards a larger question of how you want to direct users towards the Loader.new() API by default. Should there be method comments? Will the upcoming documentation updates handle this?

Comment thread src/workers/VolumeLoadWorker.ts Resolved
Comment thread src/loaders/OmeZarrLoader.ts Resolved
meta = { type: VolumeFileFormat.JSON, info: (loader as JsonImageInfoLoader).jsonInfo };
} else {
meta = {
type: options?.fileType as Exclude<VolumeFileFormat, VolumeFileFormat.ZARR | VolumeFileFormat.JSON>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type: options?.fileType as Exclude<VolumeFileFormat, VolumeFileFormat.ZARR | VolumeFileFormat.JSON>,
type: fileType,

Should type correctly without needing the cast

@frasercl
frasercl force-pushed the refactor/sync-meta-load branch from 0ca74c9 to e090368 Compare November 11, 2026 01:24
@toloudis

Copy link
Copy Markdown
Contributor

In the PR description of Problem, I'm missing the WHY. Why should we even do this? what does it enable or make better?

@toloudis toloudis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a hard PR to read/review. It seems to have some things that are simply renamings, and other places where code was lifted and moved into new functions but with tiny little changes. So it's hard to tell, as a reviewer, if things are bad without really understanding all the details of fetching and loading these bits of data.

It looks like a lot more places in the code suddenly depend on data structures that should be internal to the core implementations of specific format loaders. At least, the json loader, which is so very close to being completely legacy, should not be creeping in to so many places.

Comment thread public/index.ts
async function loadVolume(name: string, loadSpec: LoadSpec, loader: VolumeLoader): Promise<void> {
const fullDims = await loader.loadDims(loadSpec);
function loadVolume(name: string, loadSpec: LoadSpec, loader: VolumeLoader): void {
const fullDims = loader.getDims(loadSpec);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this now doing the await internally?


/* eslint-disable @typescript-eslint/naming-convention */
type JsonImageInfo = {
export type JsonImageInfo = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

who else is going to use this outside of this module?


const response = await fetch(remapUri(this.urls[time]));
const imageInfo = (await response.json()) as JsonImageInfo;
static async new(urls: string | string[], cache?: VolumeCache): Promise<JsonImageInfoLoader> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any concerns about reusing the keyword new? definitely made me pause and think, it might be confusing to see it where it's being called

Comment thread src/workers/types.ts
import type { ZarrLoaderFetchOptions, ZarrLoaderMetadata } from "../loaders/OmeZarrLoader.js";
import type { JsonImageInfo } from "../loaders/JsonImageInfoLoader.js";

export type VolumeLoaderMetadata =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could use a comment explaining why it's here and what the type will be used for. I think it might be a code smell that this module suddenly has to depend on JsonImageInfoLoader

return { imageInfo: jsonToImageInfo(this.meta.info[loadSpec.time]), loadSpec };
} else {
return { imageInfo: this.meta.imageInfo, loadSpec };
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getDims and getImageInfo look like they should be abstracted so that implementation details of the specific formats don't have to leak into this file.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants