Repository navigation
Release 2026.10.0: MPI diagnosis, floor fix, and Globus onboarding - #195
Merged
Merged
Conversation
A conservative remap on the UCAR endpoint comes back as WorkerLost, and the message says it is memory and to subset first. It is not memory, and subsetting cannot fix it, so that advice costs an afternoon. What is actually happening, measured on Casper: the identical remap runs in 0.09 seconds under check_remote_yac and kills the worker through run_analysis. Same node, same YAC 3.18, same method, same mesh. The only difference is that the probe runs it in a worker-side subprocess, so an MPI_Init abort takes the child instead of the worker. Open MPI initialises as a singleton and is fine; MPICH aborts, and an abort inside a Globus worker surfaces here with nothing mentioning MPI. Three changes. The WorkerLost handler now asks what the call was doing. A YAC-backed method -- conservative, nnn, dnn, average, or an explicit backend="yac" -- gets the MPI explanation, how to confirm it (check_remote_yac, whose conservative_ok separates "YAC is broken" from "YAC cannot run in this process"), and what to do instead. The fallback it offers is qualified: inverse_distance_weighted and bilinear do run, and neither conserves the field integral, which is the only reason to have asked for conservative. Every other lost worker still gets the memory advice, which is right for it. check_remote_yac now exercises the conservative path. It only ever ran nearest_neighbor, which uses uxarray's own engine and never calls into YAC -- so it reported "YAC OK" on a worker where every conservative remap died. Importing YAC and computing weights with it are different capabilities and the probe only tested the first. The new block is separate from the existing one so a site that can import YAC but not compute weights is reported as exactly that, and it catches BaseException because a YAC failure can raise SystemExit. conservative_ok is deliberately not part of the exit status: a site that only wants the non-YAC methods is not broken. ucar_endpoint.sh gains check-mpi, which runs the bare import and the launched import side by side and says which case you are in. Chrysalis has had check-yac under srun since it hit this; UCAR had nothing. Not automated: the repair. Putting workers inside an MPI context means GlobusMPIEngine or a launcher prefix on the engine, not an export in worker_init -- worker_init runs before the worker process exists. A wrong engine block takes the endpoint down for every tool rather than just the remap, so check-mpi prints what to change and stops. The regression tests cover the routing, not only the predicate. _looks_yac_backed can be correct and never fire if _run_on_hpc hands it the wrong thing, which is what the first draft of this did: it read only kwargs, every caller passes positionally, and a real remap still got the memory advice with the unit tests green.
Three new users got stuck in the same place, and it was not the hard part. The Globus explanation existed, at line 64 of remote-hpc.md, under a heading nobody reaches: README's "Pick your path" sends HPC users to that page's *middle*, past the identity step. So the prerequisite was written down and unreachable, which reads the same as missing. docs/globus-basics.md is the ten-minute version: which of the several Globus services you actually need, why the identity has to be the institutional one, and the three things people reasonably expect that are not true. It is 70 lines because a page nobody finishes helps nobody. The identity point is the one that costs the most time. A Globus login is not automatically tied to a cluster account; sign in with a personal Google account when the HPC login is you@institution.edu and you get an identity that authenticates perfectly and sees none of your endpoints -- indistinguishable, from the outside, from a dead endpoint. The fix is linking identities, not making a second account, and nothing said so. README now routes path 2 to connect-a-cluster.md rather than the long form, and names globus-basics.md for paths 2-4. docs/ucar.md gains two things. Endpoints are per-user: a colleague's working Casper endpoint gives you nothing, since it runs as their account and reads their files, and the conda and YAC paths in that page are per-$USER too. And the WorkerLost-on-conservative-remap entry, with how to tell MPI from memory -- check_remote_yac runs the same remap in a subprocess, so conservative_ok separates "YAC is broken" from "YAC cannot run in this process."
The 2026.10.0 entry was written before these landed. Adds the YAC/MPI diagnosis, the check_remote_yac gap that let a broken worker report itself healthy, the McpError/MCPError floor break, and the onboarding pages. Security count 49 -> 55 for the filelock advisories.
`grep jain` found nothing in src/, scripts/ or docs/ -- those all use $USER or $NCAR_USERNAME already. The tests were the exception. tests/test_hpc_safety.py asserted on a PYTHONPATH containing /lcrc/group/e3sm/jain/uxarray-yac-src. The rule under test keys on "yac" appearing in the path and has nothing to do with whose checkout it is, so a real name there reads like the behaviour depends on it. Now <user>. tests/test_remote_remap_live.py printed three copy-pasteable paths under one person's group directory. A colleague running the live tests would copy them verbatim and get "file not found" on a cluster where the file exists -- under their own name. Hoisted to a $MESH variable with <user> in it, so the thing to change is visible and typed once. The /scratch/rjain paths in the transfer tests address a fake service and never touch a filesystem, but a reader cannot tell that from the string. /scratch/demo. Authorship in pyproject.toml and the release workflows is left alone -- that is a name doing its job.
scripts/chrysalis_endpoint.sh check-yac reported "remap_ok": true from a nearest_neighbor remap, which runs on uxarray's own engine and never calls into YAC. It would therefore pass on a worker where every conservative remap dies -- and on Chrysalis that is exactly the state: check-yac green, and `method="conservative"` through run_analysis kills the worker. Same gap check_remote_yac had. It now also runs the conservative remap and reports conservative_ok. Deliberately not part of the exit status: a site that only wants the non-YAC methods is not broken, so this is reported rather than enforced. Also trims the onboarding pages, which had grown to three descriptions of the same thing. globus-basics.md owns the identity and the Compute-is-not-Transfer point; connect-a-cluster.md owns the five-command path; remote-hpc.md is the long form for when one of those did not work. It was still opening with its own Globus-identity step, its own install-the-extra step, and a second copy of the three-layer diagram -- 107 lines saying what the other two now say. Its steps renumber 3-4-5 to 1-2-3, since the first two are gone. The trio is 549 lines against 569 for the two pages it replaces, and the Globus prerequisite is now reachable instead of buried at line 64 of the longest one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepares 2026.10.0 and clears the three things that were blocking it. One
PR because the release carries all of them.
Release
Version moves to
2026.10.0across all five files (pyproject.toml,server.json×2,__init__.py,conda/recipe/meta.yaml), mirroring uxarray2026.10.0. CHANGELOG written by hand —
prepare_release.pystamps an emptysection, and 21 commits of real work shipping undocumented is not a release
note.
Blocker 1 — Upstream Compatibility was red
My own regression from #193.
McpErroron mcp 1.x,MCPErroron 2.x;importing one name fails on the other SDK, and only the weekly floor job
resolves 1.x — so it was invisible until that run went red. Verified against a
real mcp 1.27 install, not just asserted.
Blocker 2 — Dependency audit was red
Six filelock advisories. Bumped to 4.2.4;
osv-scannernow reports noissues found.
Blocker 3 — conservative remap kills the UCAR worker
The error blamed memory and advised subsetting. It is MPI, and subsetting
cannot fix it — so that advice costs an afternoon.
Measured on Casper, and this is what settles it:
check_remote_yac)run_analysis)WorkerLostSame node, same YAC 3.18, same mesh, same method. The only difference is
whether an
MPI_Initabort can take the worker with it. Open MPI initialisesas a singleton; MPICH aborts.
Three changes:
(
conservative,nnn,dnn,average, or explicitbackend="yac") getsthe MPI explanation, how to confirm it, and a qualified fallback —
inverse_distance_weightedandbilinearrun, and neither conserves thefield integral, which is the only reason to have asked for conservative.
Every other lost worker still gets the memory advice.
check_remote_yacwas reporting "YAC OK" on a broken worker. It onlyever ran
nearest_neighbor, which uses uxarray's own engine and never callsinto YAC. Importing YAC and computing weights with it are different
capabilities; the probe only tested the first.
conservative_oknowseparates "YAC is broken" from "YAC cannot run in this process".
ucar_endpoint.sh check-mpiruns the bare and launched imports side byside and says which case you are in. Chrysalis has had an equivalent since
it hit this; UCAR had nothing.
Not automated: the repair. Putting workers in an MPI context means
GlobusMPIEngineor a launcher prefix on the engine — not an export inworker_init, which runs before the worker process exists. I started wiring aYAC_LAUNCHERvariable, realised it would look applied and do nothing, andremoved it. A wrong engine block takes the endpoint down for every tool, not
just the remap.
check-mpiprints what to change and stops.Docs — onboarding
New users kept getting stuck in the same place, and it was not the hard part.
The Globus explanation existed, at line 64 of
remote-hpc.md, under aheading nobody reaches — README's "Pick your path" sends HPC users to that
page's middle, past the identity step. Written down and unreachable reads the
same as missing.
docs/globus-basics.md(74 lines): which of the several Globus servicesyou need, why the identity must be the institutional one, and three things
people reasonably expect that are not true. Short on purpose — a page nobody
finishes helps nobody.
tied to a cluster account. Sign in with a personal Google account when your
HPC login is
you@institution.eduand you get an identity thatauthenticates perfectly and sees none of your endpoints — indistinguishable
from a dead endpoint. The fix is linking identities, not a second account.
docs/ucar.md: endpoints are per-user. A colleague's working Casperendpoint gives you nothing — it runs as their account and reads their files.
The conda and YAC paths there are per-
$USERtoo. Plus theWorkerLost-vs-MPI triage above.
connect-a-cluster.mdrather than the long form.Verification
1555 passed, 1 skipped, 44 deselected, 1 xfailedtest_mcp_protocol.py18 passedsphinx-build: clean, new page renders, zero warnings.mdlinks across docs/ and README resolveosv-scanner: no issues foundA note on the tests
The first version of the MPI fix read only
kwargs— but every caller into_run_on_hpcpasses positionally, so it never fired. Unit tests green, realremap still got the memory advice. There is now a test that drives the actual
call path and catches exactly that.
Not included
ruff checkreports 198 pre-existing errors on clean main.--fixreformats 100+ lines I never touched, so I formatted only my own files and
left the rest. Worth its own PR, not smuggled into a release.