💨 ODD PI 26.4 Objective 1: TEMPO optimized data delivery for AIR4US

[abarciauskas-bgse]

Motivation + Description

AIR4US is launching its beta portal in September. Stakeholders of the portal will benefit from efficient access, visualization and analysis for the dataset. TEMPO gridded datasets will be evaluated for optimized integration, likely using virtualization. Implementation will take into consideration the intended use of the dataset in the AIR4US portal as well as the extensibility of any proposed solution to the wider community.

Acceptance Criteria / Sub-tasks

• Coordination with AIR4US stakeholders
• Dataset and use case evaluation
• Design optimized representation and integration (e.g. virtual icechunk store)
• Prototype
• Implement + document

Resources

• Air Quality Portal Instance Requirements
• https://github.com/Air-Quality-Portal/aq-arch/issues/92

LOE

0.32 FTE for the PI

Dependencies

• AIR4US project leadership for direction on use case
• AIR4US front-end engineering for integration support
• Data services for data publication support

[maxrjones]

Here's my plan for this objective, for @abarciauskas-bgse and @Jeanne-le-Roux to flag any concerns:

Start Date	End Date	Hours	Scope	Deliverable
Wed, Jul 22, 2026	Wed, Jul 22, 2026	5	Initial dataset virtualization scoping	List of risks and mitigation strategies; adjusted delivery plan if needed
Mon, Aug 3, 2026	Wed, Aug 5, 2026	18	Build prototype	Architecture document and static Icechunk store
Thu, Aug 6, 2026	Thu, Aug 6, 2026	2	Prototype review by @abarciauskas-bgse or @sharkinsspatial	Sign-off on plan for continuous updates
Fri, Aug 7, 2026	Mon, Aug 10, 2026	12	Build and test pipeline for continuous updates	Continuous Icechunk Store, Pipeline, IaC
Tue, Aug 11, 2026	Mon, Aug 17, 2026	12	Monitor pipeline and fix any issues	Document noting any issues and how they were resolved
Mon, Aug 17, 2026	Fri, Aug 21, 2026	20	Reserved time for delays	—
Fri, Aug 21, 2026	Fri, Aug 21, 2026	—	Final delivery	Virtual store + pipeline, final report and closed objective

A couple questions about the initial objective:

Coordination with AIR4US stakeholders

Can we lead on @Jeanne-le-Roux for this? I'm hoping that I can be invited to a weekly project standup to stay engaged with the overall project, without needing to independently coordinate with their stakeholders.

@abarciauskas-bgse I also want to confirm that any API changes/features in titiler-multidim are out of scope and either unnecessary or tracked by other objectives?

Other open questions:

1. Who adds the virtual store to the AIR4US STAC catalog?
2. How are COG pipelines monitored? Are they part of the ODSI grafana dashboard? Should continuously updated TEMPO pipelines be?

[Jeanne-le-Roux]

Hi @maxrjones thanks for putting this together. Your final delivery date is not too far off from what I put in the high-level implementation schedule I started working on yesterday, so I think this is good (I originally put the 14th with a big question mark :)

Yes, I will be facilitating the stakeholder interaction so that should not take much (if any) of your time. The main area where I could see pulling you in for stakeholder interaction is if there are technical questions from the TEMPO SMEs or if anything comes up during testing.

As for

1. Who adds the virtual store to the AIR4US STAC catalog?

I'd coordinate with @siddharth0248

2. How are COG pipelines monitored? Are they part of the ODSI grafana dashboard? Should continuously updated TEMPO pipelines be?

@siddharth0248 might know but I've also started talking with @amarouane-ABDELHAK on the observability system

[maxrjones]

@abarciauskas-bgse I reduced the FTE for this objective to only include working hours on this objective (1 FTE - time OOO - time on community objectives - time on internal responsibilities). That makes for .32 FTE, which I think is reasonable for accomplishing this work. That's ~twice the time in the work plan, which should be available for support on the other AIR4US and/or ODD objectives.

[abarciauskas-bgse]

@maxrjones thanks for putting the timeline together

@abarciauskas-bgse I also want to confirm that any API changes/features in titiler-multidim are out of scope and either unnecessary or tracked by other objectives?

I am fine with making titiler-multidim support a separate objective, but I am hoping that we can support at least a tiles endpoint this PI. I recognize that titiler-multidim support may be more work and require API changes. However, I'm not sure the virtual dataset can be featured in the beta AIR4US portal without the tiles endpoint, since the AIR4US use cases at this point (again, as far as I am aware) are via the website (i.e. browser-based visual exploration), not a JupyterHub.

However, adding that objective really depends on your capacity to support it. I think during our AQ meeting last week I mentioned testing and preparing titiler-multidim integration in parallel with the virtualization work, but it seemed like there was some hesitancy about that approach which is why we proposed writing up a more specific delivery timeline. Given the current estimate for this work is 0.32 FTE, do you think if we deemed the tiles endpoint high priority you might be able to support it or should we involve some other team members?

[maxrjones]

@maxrjones thanks for putting the timeline together

@abarciauskas-bgse I also want to confirm that any API changes/features in titiler-multidim are out of scope and either unnecessary or tracked by other objectives?

I am fine with making titiler-multidim support a separate objective, but I am hoping that we can support at least a tiles endpoint this PI. I recognize that titiler-multidim support may be more work and require API changes. However, I'm not sure the virtual dataset can be featured in the beta AIR4US portal without the tiles endpoint, since the AIR4US use cases at this point (again, as far as I am aware) are via the website (i.e. browser-based visual exploration), not a JupyterHub.

However, adding that objective really depends on your capacity to support it. I think during our AQ meeting last week I mentioned testing and preparing titiler-multidim integration in parallel with the virtualization work, but it seemed like there was some hesitancy about that approach which is why we proposed writing up a more specific delivery timeline. Given the current estimate for this work is 0.32 FTE, do you think if we deemed the tiles endpoint high priority you might be able to support it or should we involve some other team members?

thanks Aimee, this is very helpful feedback. I narrowed my scope from the meeting based on the original AC in this ticket, but will work on broadening it again to include at least a tiles endpoint.

[abarciauskas-bgse]

I narrowed my scope from the meeting based on the original AC in this ticket

@maxrjones ah that makes sense! sorry the change in scope from the original AC is causing more work for you. I modified the AC above to include the titiler-multidim tiles endpoint.

I do think we should scope our obligation to producing a working tiles endpoint, and that the front end team should own integrating it into a front-end UI.

[Jeanne-le-Roux]

Is it possible to add this as a parent ticket? I don't have permission to do so. https://github.com/Air-Quality-Portal/aq-arch/issues/126

I've pulled this objective ticket into our AQ sprint board so that we can track it from a high level.

[sharkinsspatial]

@Jeanne-le-Roux I don't have access to the . https://github.com/Air-Quality-Porta repo but if you can grant me view privileges I can add it. 👍

[Jeanne-le-Roux]

@sharkinsspatial invite sent!

[sharkinsspatial]

@maxrjones For reference. A discussion on the Openscapes Slack with @danielfromearth about some of the complications with the TEMPO data https://openscapes.slack.com/archives/C0B4S7NPY4Q/p1783629014557679

---

@danny Kaufman 👋 Quick update. It looks like we'll be taking on some of the TEMPO virtualization work in the next few weeks so @max Jones and @AimeeB will probably be reaching out to you soon for some guidance/input. On a related note I've been working on a PR for aligning "sparse" virtual datasets. for concatenation github.com/zarr-developers/VirtualiZarr/pull/1032 which I think might address the issues you were initially hitting here github.com/zarr-developers/VirtualiZarr/issues/487. If you have a free moment it might be worthwhile testing your concatenation use cases against this PR.

---

Hey @sean Harkins, finally got around to testing the TEMPO concatenation cases against #1032. Thanks again for pointing me to it!

Unfortunately, looking like it doesn't quite address the specific blocker, though I can see how it would for whole-chunk cases. The raggedness across TEMPO Level-2 granules is sub-chunk: for the product arrays, each granule is stored as a single chunk spanning the whole mirror_step dimension (chunks == shape), and mirror_step varies between granules — for some granules in a scan it's 132 and others it's 131.

So concatenation still trips an error from check_same_chunk_shapes ("Requires ZEP003"):

 ~/VirtualiZarr/virtualizarr/manifests/utils.py:239, in check_same_chunk_shapes(chunks_list)
    237 for other_chunks in other_chunks_list:
    238     if other_chunks != first_chunks:
--> 239         raise ValueError(
    240             f"Cannot concatenate arrays with inconsistent chunk shapes: {other_chunks} vs {first_chunks} ."
    241             "Requires ZEP003 (Variable-length Chunks)."
    242         )

ValueError: Cannot concatenate arrays with inconsistent chunk shapes: (131, 2048) vs (132, 2048) .Requires ZEP003 (Variable-length Chunks).

What do you think? I suppose it looks like TEMPO Level-2 might still be reliant on variable-length / fill-value chunk support (#12 / #22 / #38). The only other option that's been raised before is mixing virtual + native (padded) chunks in Icechunk, but because many granules are the short length, that would probably still end up materializing a lot of chunks.

Happy to talk more about it whenever we have a chance! and perhaps at ESIP?

---

@danny Kaufman Thanks for investigating this. Looking through the issues I was unclear if this was indeed a sub-chunk issue or not so thanks for confirming. Unfortunately, any xarray level alignment or padding we do can only operate at the chunk level as this is our smallest increment of operation in Virtualizarr.

IIUC looking at github.com/zarr-developers/VirtualiZarr/issues/22 I think our only possible solution here is the approach Deepak suggested of using a custom codec which can do sub-chunk indexing 🤕 or as you suggested, materializing the the problematic chunks to full values.

Unfortunately I'll be away camping during ESIP but my colleague @AimeeB will be there and I believe my colleague @max Jones who will be working on TEMPO virtualization will be attending virtually (that was too many virtuals 😆 )