diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e0618f..9f3dca1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,140 @@ built against; see `docs/release.md`. Versions through `0.3.1` were SemVer. ## Unreleased +## 2026.10.0 — 2026-10-23 +### Added +- `subset_advanced` selects faces near a point, by radius + (`subset.bounding_circle`) or by neighbour count + (`subset.nearest_neighbor`). An empty selection is refused with the mesh + extent, the same rule the bounding box already followed; UXarray's own + message says the circle was empty without saying where a circle would + not be. +- `regrid_elements` carries a field between nodes, edges and faces through + the `topological_*` reductions. This is what makes node-centred output + usable: every vector-calculus operation needs face-centred data, and the + refusal a caller got had no repair behind it because the server could not + run the standard one. +- `domain/uxarray_surface.py` records which of UXarray this server reaches + and which it deliberately does not, one verdict per method with a reason. + Tests hold it to the installed library, so a new method upstream fails + until someone decides about it and a renamed one fails the day the rename + lands rather than the next time a user follows our advice into a method + that is gone. Deliberately not a count: "we expose N of M" goes red on + every upstream release for reasons nobody can act on. + + All three came from replaying the notebooks in UXARRAY/uxarray-community + through the server. They are separate tools rather than new `run_analysis` + operations because #89 already measures that front door as too large to + describe itself. + +### Changed +- Floors `uxarray[geo,viz]>=2026.10.0`. The previous pin's `<2026.10` + ceiling excluded the release outright. The resolve is quiet -- 189 + packages, uxarray the only one that moves -- and 2026.10.0's `viz` extra + caps `cartopy<0.26`, which settles UXARRAY/uxarray#1773 upstream rather + than here. +- Floors `toolregistry>=0.19.0` and `toolregistry-server>=0.7.0`. 0.19 + carries the heterogeneous-union fix (Oaklight/ToolRegistry#269) this + project has been waiting on. What we serve for `lat_spec` is still not + upstream's rendering: `tuple` and `list` both become `array`, so the + union arrives with the same branch twice and untyped items. The schema + helper now replaces that rendering rather than filling an absence, and + the core tool specification got smaller as a result -- 36271 to 36143 + bytes. +- The container declares its Python version once. It was written in four + places, two `FROM` lines and two `--python 3.12` flags, and Dependabot + rewrites `FROM` and nothing else -- so its 3.14 bump produced an image + whose base had no 3.12 beside a sync that demanded one. +- `SETUP.sh` reproduces CI. It ran a bare `uv sync` and an unfiltered + `pytest`, from before the suite moved to marker selection, so following + it produced 19 failures that were all missing optional dependencies. It + also gated on the ambient `python3`, which is not the interpreter the + project runs on: a machine with 3.9 on `PATH` could not run the script at + all while `uv sync` on that same machine builds a working 3.12 + environment. + +### Fixed +- A YAC-backed remap that loses its worker is told what killed it. The + message said memory and advised subsetting; on the UCAR endpoint the + cause is MPI, and subsetting cannot fix it. Importing YAC calls + `MPI_Init`, which Open MPI serves as a singleton and MPICH aborts, and + an abort inside a Globus worker arrives with nothing mentioning MPI. + Measured on Casper: the identical conservative remap runs in 0.09s + under `check_remote_yac` and kills the worker through `run_analysis` -- + same node, same YAC, same mesh, the only difference being that the + probe runs it in a subprocess. Any other lost worker still gets the + memory advice, which is right for it. +- `check_remote_yac` exercises the conservative path. It only ever ran + `nearest_neighbor`, which uses uxarray's own engine and never calls + into YAC, so it reported "YAC OK" on a worker where every conservative + remap died. `conservative_ok` now separates "YAC is broken" from "YAC + cannot run in this process", which is the distinction that picks the + repair. +- `scripts/ucar_endpoint.sh check-mpi` runs the bare and launched imports + side by side and names which case the site is in. Chrysalis has had an + equivalent since it hit this; UCAR had nothing. +- The MCP SDK exception is found under either spelling. `McpError` on + 1.x, `MCPError` on 2.x; importing one name fails on the other SDK, and + only the weekly floor job resolves 1.x, so the wrong one went unnoticed + until that run went red. +- Tool annotations reach clients on mcp 1.x, not only 2.x. At the declared + floor the shim attached nothing, so every tool was served with no hints + at all -- the `readOnlyHint` a client reads to decide what it may + auto-approve was absent exactly where it is documented as present. Three + SDK differences, each independently sufficient: the handler dict is keyed + by method string on 2.x and by request type on 1.x, 1.x returns a + `ServerResult` envelope whose `.root` holds the tools, and the attribute + itself is public on one and private on the other. +- Our annotations outrank inferred ones. toolregistry-server 0.7.0 began + attaching annotations of its own and marked `manage_session` and + `resume_workflow` `readOnlyHint=True`; the first resets session state and + clears artifacts, the second runs pending workflow steps. Nine tools left + deliberately unrated came back carrying hints. Where this server has + rated a tool its rating is what goes out, and where it has not, nothing + does -- absence is honest, an unchecked inference is not. +- `get_capabilities` says which of what it lists is executable. It + advertises native UXarray calls, several of which have no operation + behind them -- `grid.get_ball_tree()`, `var.integrate()` -- and nothing + distinguished them, so a model reading the result would plan a call that + does not exist. +- The "not face-centered" refusals name their repair. They were correct and + a dead end: the standard fix is `topological_mean("face")`, which this + server could not run. They now point at `regrid_elements`, and a test + follows that advice to confirm it works rather than landing on a second + error. +- An unknown tool name is reported as JSON-RPC `-32602` rather than a + result carrying `isError`, following toolregistry-server 0.7.0. That is + the correct reading: `isError` is for a tool that ran and failed, and an + unknown name means nothing ran. +- The oversized-SCRIP test fixture no longer needs 8 GiB. It built each + dask array as a single 6.5 GiB chunk and `to_netcdf` materialized one at + a time, so the test peaked at 7.45 GiB before the code under test ran. + That fits GitHub's runners and not a laptop. Chunking along + `grid_corners` holds each write to ~191 MiB; the full suite now peaks at + 1.36 GiB. + +### Documentation +- `docs/globus-basics.md` says what Globus is before anything asks you to + use it. The explanation existed, in the middle of `remote-hpc.md`, + under a heading the README routes HPC users straight past -- written + down and unreachable, which reads the same as missing. It also names + the identity trap that costs the most time: a Globus login is not + automatically tied to a cluster account, and signing in with a personal + account produces an identity that authenticates fine and sees none of + your endpoints, which looks exactly like a dead endpoint. +- `docs/ucar.md` says endpoints are per-user. A colleague's working + Casper endpoint gives you nothing: it runs as their account and reads + their files. The conda and YAC paths on that page are per-`$USER` too. + +### Security +- Clears 55 advisories in the lockfile, all in transitively-resolved + packages: 44 against pillow 12.3.0 (5 Critical), PYSEC-2026-4320 against + certifi 2026.7.22 (9.1 Critical), and four against virtualenv 21.14.7. + certifi is the trust store behind every outbound HTTPS call this server + makes -- Globus Compute submission, transfers, remote dataset reads -- so + a Critical there is not a transitive detail. A further six against + filelock 4.1.1 cleared the same way. + ## 2026.9.1 — 2026-09-25 ### Changed - Floors `uxarray[geo,viz]>=2026.9.1`. Upstream 2026.9.1 moved cartopy, diff --git a/README.md b/README.md index 1e78adb..55e6cb6 100644 --- a/README.md +++ b/README.md @@ -36,11 +36,15 @@ You are most likely one of: 1. **Local user** — laptop only, no HPC. → [Local install](#local-install). 2. **HPC user, endpoint already exists** — someone at your lab gave you a Globus Compute endpoint UUID. → [Local install](#local-install), then - [docs/remote-hpc.md](docs/remote-hpc.md). + [docs/connect-a-cluster.md](docs/connect-a-cluster.md). 3. **HPC user, your own personal endpoint** — you have a Globus identity and shell access to an HPC machine, and want to stand up an endpoint just for yourself. → [Local install](#local-install), then [docs/operating-an-endpoint.md](docs/operating-an-endpoint.md#solo-personal-endpoint-quickstart). + +> **New to Globus?** Paths 2–4 all assume an identity linked to your HPC +> account. [docs/globus-basics.md](docs/globus-basics.md) is ten minutes and +> saves most of the common dead ends. 4. **Group / shared endpoint operator** — you're standing one up for a team, project, or lab. → [Local install](#local-install), then the full [docs/operating-an-endpoint.md](docs/operating-an-endpoint.md) including diff --git a/conda/recipe/meta.yaml b/conda/recipe/meta.yaml index d3b438b..34d6a8c 100644 --- a/conda/recipe/meta.yaml +++ b/conda/recipe/meta.yaml @@ -1,5 +1,5 @@ {% set name = "uxarray-mcp" %} -{% set version = "2026.9.1" %} +{% set version = "2026.10.0" %} {% set python_min = "3.12" %} package: diff --git a/docs/connect-a-cluster.md b/docs/connect-a-cluster.md index 46b2e69..27040ff 100644 --- a/docs/connect-a-cluster.md +++ b/docs/connect-a-cluster.md @@ -3,6 +3,11 @@ Your data is on an HPC machine. You want to ask an AI about it without downloading anything. This page gets you there. +> **Never used Globus?** Ten minutes on +> [globus-basics.md](globus-basics.md) first. It covers the identity, why it +> has to be your institutional one, and why Compute is not Transfer — the +> three things that account for most stuck setups. + It is machine-agnostic: the same five commands work for Chrysalis, Casper, Perlmutter or a university cluster nobody has written a page about. Site pages ([chrysalis](chrysalis.md), [ucar](ucar.md), [improv](improv.md)) @@ -39,15 +44,9 @@ Three things, in three places, each of which can be wrong independently: Diagnose in that order. Most "it doesn't work" is the first or the third, and the error usually names the second. -### Two things people reasonably expect, which are not true - -- **An endpoint is per-user, not per-machine.** It runs *as your HPC - account* and can read exactly what you can read. There is no shared - "Chrysalis endpoint" unless someone deliberately operates one for a group. -- **Compute is not Transfer.** Globus Compute runs code; Globus Transfer - moves files. Separate services, separate consents. A working endpoint - cannot copy a file. You usually do not need Transfer at all — see - [data-transfer.md](data-transfer.md). +Two surprises worth knowing before you start — endpoints are per-user, and +Compute is not Transfer. Both in +[globus-basics.md](globus-basics.md#three-surprises). --- diff --git a/docs/globus-basics.md b/docs/globus-basics.md new file mode 100644 index 0000000..2fae3f3 --- /dev/null +++ b/docs/globus-basics.md @@ -0,0 +1,74 @@ +# Globus, the short version + +Ten minutes, a browser, no cost. Read this once before connecting a cluster. + +## Which bit you need + +Globus is several services. You need Auth (login) and Compute (runs your +code). You almost certainly do not need Transfer. + +**Compute is not Transfer.** Different services, separate consents. A +working Compute endpoint cannot copy a file. If the word "consent" shows up +and you were not trying to move a file, you are in the wrong service. + +The point of this server is that the analysis goes to the data and a number +comes back, so moving files is a separate decision: +[data-transfer.md](data-transfer.md). + +## Get an identity + +Already used Globus for HPC file transfers? You have one. Skip ahead. + +Otherwise: go to and **sign in with your +institution** — Argonne, NCAR/UCAR, NERSC, most universities. Use the same +account that holds your HPC login. + +That last part matters. A Globus identity is not automatically tied to your +cluster account. Sign in with a personal Google account when your HPC login +is `you@institution.edu` and you get an identity that authenticates fine and +then sees none of your endpoints — which looks exactly like a dead endpoint. +Already done it? Link the institutional identity at + rather than starting over. + +## Check it + +```bash +uxarray-mcp doctor +uxarray-mcp endpoints discover +``` + +The first Globus command opens a browser. Finish the login and re-run. + +`discover` asks Globus what your identity can see, which also answers *where +do I get a UUID?* — it is printed once when an endpoint registers and is easy +to lose. Nobody has to send you one. + +| You see | Next | +|---|---| +| An endpoint listed | [connect-a-cluster.md](connect-a-cluster.md) | +| Nothing | [Create one](connect-a-cluster.md#create-20-minutes) | +| `ENDPOINT_NOT_ONLINE` | It is asleep — see below | +| A login loop | Wrong identity; see above | + +## Three surprises + +**An endpoint is per-user, not per-machine.** It runs as *your* cluster +account and reads what you can read. Your colleague's working endpoint tells +you nothing about whether you have one. + +**Registered is not working.** It means a manager process phoned home. It +does not mean a worker exists, has `uxarray`, or can read your file. Those +fail separately, which is why `doctor`, `validate_hpc_setup` and +`probe_path_access` check them in that order. + +**Endpoints sleep.** On a scheduler-backed endpoint the worker lives in a +batch allocation. When it ends, the endpoint stays registered with nothing to +run on. `ENDPOINT_NOT_ONLINE` usually means *ask again in a minute*. A slow +first call after an idle period is normal. + +## One caution + +An endpoint's operator can see the functions you submit and the paths you +name. Submit nothing you would not put in a ticket. +[SECURITY.md](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/SECURITY.md) +has the rest. diff --git a/docs/index.rst b/docs/index.rst index 781ec04..22a87c2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,6 +23,7 @@ HTTP clients) from a single install. :maxdepth: 2 :caption: Running on HPC (optional) + globus-basics connect-a-cluster remote-hpc data-transfer diff --git a/docs/remote-hpc.md b/docs/remote-hpc.md index b91de0e..67e45d5 100644 --- a/docs/remote-hpc.md +++ b/docs/remote-hpc.md @@ -1,111 +1,23 @@ -# Remote HPC Setup — Connecting to an Existing Endpoint - -> **New here? Read [connect-a-cluster.md](connect-a-cluster.md) first.** It -> covers the same ground in five commands and explains the concepts, for any -> machine. This page is the long form: every step spelled out, with the -> reasoning and the failure modes. Come here when the short version is not -> enough, or when something has gone wrong. -> -> If your question is *"where do I get a UUID?"*, the answer is -> `uxarray-mcp endpoints discover`. - -This page is for users who **already have an HPC account** (Argonne, NCAR, -NERSC, a university cluster, etc.) and want their AI agent to analyze data -that lives on that machine. - -> **Prerequisites — check each one before starting:** -> 1. uxarray-mcp installed locally and working with your AI client. See the -> [README](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/README.md#local-install). -> 2. A user account on the HPC machine, with shell access. -> 3. The data you want to analyze is readable by your HPC account. -> 4. Either: someone has given you a **Globus Compute endpoint UUID** for -> that machine, **or** you plan to stand up your own — in which case -> stop and read [operating-an-endpoint.md](operating-an-endpoint.md) first, -> then come back here. -> 5. You have read [SECURITY.md](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/SECURITY.md). Connecting an endpoint -> means the operator can see what you submit. - -The total setup is **5 steps**, expect **15–30 minutes** the first time. +# Remote HPC — the long form ---- - -## The three pieces - -Before the steps, the mental model. Remote analysis has three things that -have to all work, in three different places: - -```text -┌────────────────────┐ ┌──────────────────────────┐ -│ YOUR LAPTOP │ │ HPC MACHINE │ -│ │ │ │ -│ AI client │ │ Globus Compute endpoint │ -│ │ stdio │ │ (runs as your HPC │ -│ uxarray-mcp ──────┼─Globus─┼──▶ user, executes │ -│ + globus- │ Compute│ Python sent from │ -│ compute SDK │ HTTPS │ your laptop) │ -│ + your Globus │ │ │ -│ identity │ │ uxarray + xarray in │ -│ │ │ the worker environment │ -└────────────────────┘ └──────────────────────────┘ -``` - -Most "it doesn't work" failures come from one of these three layers being -misconfigured. The steps below address each one in order. - -An endpoint reporting `registered` only means the lightweight manager process -is alive and reachable — it does **not** prove the worker environment has -`uxarray` installed, can reach the scheduler, or can read your file. Validate -in order: manager reachable → a tiny no-op function runs → the exact file path -is readable → only then run a real UXarray tool. `validate_hpc_setup` and -`probe_path_access` exist for exactly this staged check. - ---- - -## Step 1 — Get a Globus identity (one-time, 5 min) - -If you've ever used Globus File Transfer for HPC, you already have this. -Skip to Step 2. - -If not: +[globus-basics.md](globus-basics.md) explains Globus and the identity. +[connect-a-cluster.md](connect-a-cluster.md) gets you running in five +commands. **Come here when one of those did not work**, or when you want +the reasoning behind a step. -1. Go to . -2. Sign in with your **institutional identity** if your lab is listed - (Argonne, NCAR/UCAR, LBL/NERSC, Oak Ridge, university SSO, etc.). This - ensures your Globus identity is linked to the same account that has HPC - access. -3. If your institution isn't listed, create a free Globus ID. -4. Verify you can reach . You're done. - -> No fee. Globus is free for non-commercial research use. - ---- - -## Step 2 — Install the HPC extras locally (1 min) - -```bash -uv tool install --python 3.12 "uxarray-mcp[hpc]" -# or to upgrade an existing install: -# uv tool upgrade --python 3.12 --extra hpc uxarray-mcp -``` - -The `--python 3.12` flag matters: most HPC endpoints today (chrysalis, -Casper, Improv) run Python 3.12 conda stacks, and Globus Compute's serializer -is fragile across Python minor versions. Mismatch raises `WorkerLost` on -non-trivial payloads. Tracked upstream at -[globus/globus-compute#2139](https://github.com/globus/globus-compute/issues/2139). - -This adds the `globus-compute-sdk` to your local install. Verify: - -```bash -uxarray-mcp doctor -``` +Assumes you already have: a working local install, an HPC account with +shell access, data that account can read, and either an endpoint UUID or +a plan to stand one up ([operating-an-endpoint.md](operating-an-endpoint.md)). +Also assumes you have read +[SECURITY.md](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/SECURITY.md) +— connecting to an endpoint means its operator sees what you submit. -The JSON report should show the Globus Compute SDK as available. (No endpoint -health checks yet — that's Step 4.) +Getting a Globus identity and installing the HPC extra are covered in +those pages. This one picks up from the UUID. --- -## Step 3 — Register the endpoint UUID locally (1 min) +## 1. Register the endpoint UUID locally You need a name to refer to the endpoint by, plus the UUID and the filesystem prefix where the data lives. @@ -167,13 +79,13 @@ the repo). --- -## Step 4 — Authenticate to Globus and probe the endpoint (2 min) +## 2. Authenticate and probe the endpoint ```bash uxarray-mcp doctor --endpoint NAME --timeout-seconds 180 ``` -Replace `NAME` with what you used in Step 3 (`ucar`, `improv`, etc.). +Replace `NAME` with the name you registered above (`ucar`, `improv`, etc.). The first run will open a browser to for OAuth consent. Approve it. The token is cached at `~/.globus_compute/`. @@ -196,7 +108,7 @@ Ask the operator. See troubleshooting below. --- -## Step 5 — Use it from your AI client (2 min) +## 3. Use it from your AI client In Claude or your MCP client, prompt: diff --git a/docs/ucar.md b/docs/ucar.md index d612c2e..3c9439c 100644 --- a/docs/ucar.md +++ b/docs/ucar.md @@ -32,6 +32,17 @@ That gives you every tool except conservative remapping. The rest of this page describes `scripts/ucar_endpoint.sh`, which is the same thing with NCAR's verified module list, an optional YAC build, and a fixed profile name. +## If someone else set this up + +Endpoints are per-user. A colleague's working Casper endpoint does not give +you one — it runs as *their* account and reads *their* files. You need your +own, and `uxarray-mcp endpoints discover` tells you whether you have it. + +Same for the conda env and the YAC build below: they live under a specific +`$USER`, so paths in this page like `/glade/work/$USER/conda-envs/uxarray_dev` +are yours, not shared. Set `NCAR_USERNAME` if your NCAR login differs from +your local `$USER`. + ## Key Points - **You do need a clone of this repo on Casper**, because the endpoint scripts @@ -178,3 +189,24 @@ synthesized from the prefix, so there is nothing to edit in the script. **`doctor` passes but YAC tools fail** — run `yac_smoke_test.py`, which checks importability on the compute worker specifically rather than on the login node. + +**A conservative remap kills the worker (`WorkerLost`)** — almost certainly +MPI, not memory, and the two look identical from the outside. Importing YAC +calls `MPI_Init`: Open MPI starts fine as a singleton, MPICH aborts the +process, and an abort inside a Globus worker arrives as a lost worker with +nothing mentioning MPI. + +Tell them apart with `check_remote_yac(endpoint=...)`. It runs the same +conservative remap in a *subprocess*, where an abort is survivable: + +- `conservative_ok: true` — YAC is fine. The worker needs to start inside an + MPI context (`GlobusMPIEngine`, or a launcher prefix on the engine). An + export in `worker_init` cannot do this; that runs before the worker exists. +- `conservative_ok: false` — the build is the problem, not the launcher. + +`./scripts/ucar_endpoint.sh check-mpi` runs the bare and launched imports side +by side and tells you which case you are in. + +Meanwhile `inverse_distance_weighted` and `bilinear` work — they use uxarray's +own engine. Neither conserves the field integral, which is usually the reason +conservative was asked for, so this is a stopgap and not a fix. diff --git a/pyproject.toml b/pyproject.toml index 4e2d667..492b66c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "uxarray-mcp" -version = "2026.9.1" +version = "2026.10.0" description = "MCP server for analyzing unstructured meshes with UXarray" readme = "README.md" keywords = ["uxarray", "mcp", "unstructured grids", "scientific computing", "globus compute"] diff --git a/scripts/chrysalis_endpoint.sh b/scripts/chrysalis_endpoint.sh index cfc4224..920a678 100755 --- a/scripts/chrysalis_endpoint.sh +++ b/scripts/chrysalis_endpoint.sh @@ -410,7 +410,29 @@ except Exception as exc: out["remap_error"] = f"{type(exc).__name__}: {exc}" out["remap_traceback"] = traceback.format_exc() +# The conservative remap, which is the only thing YAC is here for. +# +# `remap_ok` above is nearest_neighbor, which runs on uxarray's own engine +# and never calls into YAC -- so it reports success on a worker where every +# conservative remap dies. Importing YAC and computing weights with it are +# different capabilities and only the second one is what a caller asking for +# a conservative remap needs. +try: + t0 = time.perf_counter() + conservative = uxda.remap(destination_grid=dst, method="conservative", backend="yac") + out["conservative_ok"] = True + out["conservative_seconds"] = round(time.perf_counter() - t0, 3) + out["conservative_finite"] = int(np.isfinite(np.asarray(conservative)).sum()) +except BaseException as exc: + # BaseException: a YAC/MPI failure can raise SystemExit rather than a + # normal error, and catching only Exception would let this exit 0 with + # no verdict recorded. + out["conservative_ok"] = False + out["conservative_error"] = f"{type(exc).__name__}: {exc}"[:400] + print(json.dumps(out, indent=2)) +# `conservative_ok` is deliberately not part of the exit status: a site that +# only wants the non-YAC methods is not broken. It is reported, not enforced. raise SystemExit( 0 if out.get("yac_core_ok") diff --git a/scripts/ucar_endpoint.sh b/scripts/ucar_endpoint.sh index 67bb84f..3a1e208 100755 --- a/scripts/ucar_endpoint.sh +++ b/scripts/ucar_endpoint.sh @@ -23,6 +23,18 @@ WITH_YAC="${WITH_YAC:-0}" # HEALPix remap ran. worker_init sources activate-yac.sh from this prefix, so # `restart` is what picks the change up. YAC_VERSION="${YAC_VERSION:-3.20.2}" +# Whether this site needs YAC imported under an MPI launcher. +# +# `check-mpi` answers that question; acting on the answer is still manual. +# Open MPI initialises as a singleton and needs nothing. MPICH aborts, and +# the endpoint then needs its *workers* started inside an MPI context -- +# which means `GlobusMPIEngine`, or a `launcher_prefix` on the engine, not an +# export in worker_init. Casper was observed failing this way and has not yet +# been reconfigured, so this is deliberately not automated: a wrong engine +# block takes the endpoint down for every tool, not just the remap. +# +# Tracked rather than guessed at. See the WorkerLost message in +# remote/agent.py for what a caller is told in the meantime. YAC_PREFIX="$HOME/opt/yac-$YAC_VERSION" YAC_ACTIVATE="$YAC_PREFIX/activate-yac.sh" TMUX_SESSION="uxarray-endpoint" @@ -39,6 +51,7 @@ Usage (run on a Casper login node): ucar_endpoint.sh start Load env + start endpoint in tmux ucar_endpoint.sh restart Stop running endpoint, then start fresh ucar_endpoint.sh status Show endpoint list + ucar_endpoint.sh check-mpi Does importing YAC need an MPI launcher? Environment overrides: ENDPOINT_NAME Globus Compute endpoint profile name (default: ucar-uxarray-yac) @@ -109,6 +122,64 @@ EOF echo " wrote $YAC_ACTIVATE (YAC $YAC_VERSION, python $pyver)" } +# Does importing YAC survive outside an MPI launcher? +# +# Importing YAC calls MPI_Init. Open MPI initialises as a singleton and is +# fine; MPICH aborts the process instead. Inside a Globus Compute worker that +# abort takes the worker down, and the caller sees `WorkerLost` with nothing +# mentioning MPI -- which reads as an out-of-memory kill and sends people to +# measure their mesh. Observed on Casper: `method="conservative"` killed the +# worker while the identical remap finished in 0.09s under +# `check_remote_yac`, which runs it in a subprocess. +# +# This answers the question directly, before anyone submits a remap. If the +# bare import dies and the launched one lives, the endpoint's worker_init +# needs a launcher. +_check_mpi() { + _load_modules + _activate_env + local probe='import yac.core, sys; print("import ok:", sys.executable)' + + echo "==> Bare import (what a Globus worker does today)" + if python -c "$probe" 2>&1 | sed 's/^/ /'; then + local bare_ok=1 + else + local bare_ok=0 + echo " FAILED -- this is what takes the worker down." + fi + + echo + echo "==> Under a launcher" + local launcher="" + for cand in "mpiexec -n 1" "srun --ntasks 1"; do + if command -v "${cand%% *}" >/dev/null 2>&1; then launcher="$cand"; break; fi + done + if [[ -z "$launcher" ]]; then + echo " No mpiexec or srun on PATH; cannot compare." + return 1 + fi + echo " using: $launcher" + local launched_ok=1 + $launcher python -c "$probe" 2>&1 | sed 's/^/ /' || launched_ok=0 + + echo + if [[ "$bare_ok" == "1" ]]; then + echo "VERDICT: bare import works. MPI is not what is killing the worker;" + echo " if conservative remaps still fail, look at memory next." + elif [[ "$launched_ok" == "1" ]]; then + echo "VERDICT: bare import dies, launched import works -- this is the MPI" + echo " singleton problem." + echo " The workers must start inside an MPI context. In the endpoint's" + echo " user_config_template.yaml.j2, either switch the engine to" + echo " GlobusMPIEngine, or give the current engine a launcher prefix so" + echo " each worker runs under: $launcher" + echo " Then: $(basename "$0") restart" + else + echo "VERDICT: YAC does not import either way. This is not MPI --" + echo " check the build at $YAC_PREFIX." + fi +} + _check_yac() { if python -c 'import yac; import yac.core; print(" YAC OK:", yac.core.__file__)' 2>/dev/null; then return 0 @@ -134,6 +205,12 @@ _configure() { if [[ "$WITH_YAC" == "1" ]]; then _ensure_yac_activate yac_line=$'\n source '"$YAC_ACTIVATE" + # NOTE: if `check-mpi` says this site needs a launcher, the change goes + # in the engine block below rather than here -- worker_init runs *before* + # the worker process starts, so exporting something here cannot put that + # process inside an MPI context. Left unwired on purpose rather than + # shipped as a setting that looks applied and does nothing. See the + # YAC_LAUNCHER note at the top of this file. elif [[ "$ENDPOINT_NAME" == *yac* ]]; then echo "ERROR: '$ENDPOINT_NAME' is named for YAC but WITH_YAC is 0." >&2 echo " Rebuild it with YAC: WITH_YAC=1 $(basename "$0") configure" >&2 @@ -261,5 +338,6 @@ case "${1:-}" in _do_start) _do_start ;; # internal: invoked by tmux restart) _restart ;; status) _status ;; + check-mpi) _check_mpi ;; *) usage; exit 1 ;; esac diff --git a/server.json b/server.json index 11ac978..790739c 100644 --- a/server.json +++ b/server.json @@ -3,7 +3,7 @@ "name": "io.github.UXARRAY/uxarray-mcp-server", "title": "UXarray MCP", "description": "Inspect, subset, remap and plot unstructured climate meshes, locally or on HPC.", - "version": "2026.9.1", + "version": "2026.10.0", "websiteUrl": "https://uxarray-mcp-server.readthedocs.io", "repository": { "source": "github", @@ -13,7 +13,7 @@ { "registryType": "pypi", "identifier": "uxarray-mcp", - "version": "2026.9.1", + "version": "2026.10.0", "runtimeHint": "uvx", "transport": { "type": "stdio" diff --git a/src/uxarray_mcp/__init__.py b/src/uxarray_mcp/__init__.py index 808ac90..deaeff2 100644 --- a/src/uxarray_mcp/__init__.py +++ b/src/uxarray_mcp/__init__.py @@ -3,4 +3,4 @@ from uxarray_mcp.tools import inspect_mesh __all__ = ["inspect_mesh"] -__version__ = "2026.9.1" +__version__ = "2026.10.0" diff --git a/src/uxarray_mcp/remote/agent.py b/src/uxarray_mcp/remote/agent.py index 62733e7..f21553a 100644 --- a/src/uxarray_mcp/remote/agent.py +++ b/src/uxarray_mcp/remote/agent.py @@ -39,8 +39,37 @@ def action(fn): # type: ignore[no-redef] ) from .config import HPCConfig +#: Remap methods that compute their weights inside YAC. +#: +#: Kept here rather than imported from ``domain.remap_backend`` because this +#: module is on the submit path and the comparison is a membership test, not +#: a resolution. The names match what a caller passes as ``method``. +_YAC_BACKED_METHODS = frozenset({"conservative", "nnn", "dnn", "average"}) + + +def _looks_yac_backed(args: Any = (), kwargs: Any = None) -> bool: + """Whether the call that just died was asking YAC to compute weights. + + Scans the positional arguments as well as the keywords. Every call into + :meth:`_run_on_hpc` passes its arguments positionally, so reading only + ``kwargs`` found nothing and the YAC branch below never fired -- the + first version of this did exactly that and still printed the memory + advice for a conservative remap. + + A plain membership test over the strings is enough and stays correct if + an argument is inserted: these method names are specific enough that a + path or variable name will not collide with them. + """ + values = [a for a in (args or ()) if isinstance(a, str)] + if isinstance(kwargs, dict): + values += [v for v in kwargs.values() if isinstance(v, str)] + lowered = {v.lower() for v in values} + return "yac" in lowered or bool(lowered & _YAC_BACKED_METHODS) + -def _normalize_remote_error(exc: Exception, config: Any) -> Exception: +def _normalize_remote_error( + exc: Exception, config: Any, *, yac_backed: bool = False +) -> Exception: """Collapse a worker traceback into the same concise error a local run raises. Globus Compute re-raises worker failures with the full remote traceback @@ -105,6 +134,34 @@ def _normalize_remote_error(exc: Exception, config: Any) -> Exception: where = ( f" (worker {worker.group(1)} on {host.group(1)})" if host and worker else "" ) + if yac_backed: + # Memory is the usual cause of a lost worker, but not for this + # call. Importing YAC runs ``MPI_Init``; Open MPI initialises + # fine as a singleton, MPICH aborts the process instead, and an + # abort inside a Globus worker arrives here as WorkerLost with + # nothing mentioning MPI. Observed on Casper, where the same + # conservative remap that kills the worker completes in 0.09s + # under ``check_remote_yac`` -- which runs it in a worker-side + # *subprocess*. Same node, same YAC, same method: the difference + # is only whether an abort can take the worker with it. + # + # Sending the caller to look at mesh size here costs them the + # afternoon, because subsetting cannot fix it. + return RuntimeError( + f"The worker on {endpoint} died running a YAC-backed remap" + f"{where}. This is usually MPI, not memory: importing YAC " + "calls MPI_Init, and on an MPICH site that aborts the " + "process rather than initialising as a singleton, which " + "arrives here as a lost worker. Confirm with " + "`check_remote_yac(endpoint=...)`: if its `conservative_ok` " + "is true, YAC itself is fine and the endpoint needs its " + "workers launched under an MPI launcher (`mpiexec -n 1` or " + "`srun --ntasks 1`) in worker_init. Until then, " + "method='inverse_distance_weighted' or 'bilinear' run on " + "uxarray's own engine and do not need YAC -- but note that " + "neither conserves the field integral, which is the reason " + "to ask for conservative in the first place." + ) return RuntimeError( f"The worker on {endpoint} died while running this task{where}. " "The usual cause is memory: the worker was killed, not the task. " @@ -796,7 +853,9 @@ async def _run_on_hpc(self, func, *args, **kwargs) -> Dict[str, Any]: None, future.result, self.config.timeout_seconds ) except Exception as exc: - raise _normalize_remote_error(exc, self.config) from None + raise _normalize_remote_error( + exc, self.config, yac_backed=_looks_yac_backed(args, kwargs) + ) from None # Attach provenance with the correct HPC venue — the remote functions # are self contained and don't call attach_provenance themselves. diff --git a/src/uxarray_mcp/remote/compute_functions.py b/src/uxarray_mcp/remote/compute_functions.py index 9c63376..274b60c 100644 --- a/src/uxarray_mcp/remote/compute_functions.py +++ b/src/uxarray_mcp/remote/compute_functions.py @@ -3302,7 +3302,42 @@ def _surface(mod): out["remap_error"] = f"{type(exc).__name__}: {exc}" out["remap_traceback"] = traceback.format_exc() +# The conservative remap, which is the only reason YAC is here. +# +# Everything above passes with no working YAC weight computation: +# `nearest_neighbor` runs on uxarray's own engine and never calls into the +# library, so the probe reported "YAC OK" on a worker where +# `method="conservative"` killed the Globus worker outright. Importing YAC +# and computing weights with it are different capabilities, and only the +# second one is what a caller asking for a conservative remap needs. +# +# Kept separate from the block above so a site that has YAC importable but +# cannot compute weights is reported as exactly that, rather than as a +# blanket failure. Running inside this subprocess is what makes the check +# safe at all: a YAC/MPI abort takes down the child and is reported, where +# the same call in the worker surfaces as an unexplained WorkerLost. +try: + t0 = time.perf_counter() + conservative = uxda.remap( + destination_grid=dst, method="conservative", backend="yac" + ) + arr = np.asarray(conservative) + out["conservative_ok"] = True + out["conservative_seconds"] = round(time.perf_counter() - t0, 3) + out["conservative_dst_shape"] = list(conservative.shape) + out["conservative_finite"] = int(np.isfinite(arr).sum()) +except BaseException as exc: + # BaseException, not Exception: a YAC/MPI failure can raise SystemExit + # rather than a normal error, and catching only Exception would let the + # probe exit 0 through the bottom of this file with no verdict recorded. + out["conservative_ok"] = False + out["conservative_error"] = f"{type(exc).__name__}: {exc}"[:400] + out["conservative_traceback"] = traceback.format_exc()[-1200:] + print(json.dumps(out)) +# `conservative_ok` is deliberately not part of the exit status. The probe's +# job is to report what works; a site that wants only the non-conservative +# methods is not broken, and `available` already turns on the YAC import. raise SystemExit(0 if out.get("yac_helper_ok") and out.get("remap_ok") else 1) """ diff --git a/tests/test_calver_release.py b/tests/test_calver_release.py index a58b696..2d2850f 100644 --- a/tests/test_calver_release.py +++ b/tests/test_calver_release.py @@ -9,6 +9,7 @@ from __future__ import annotations import importlib.util +import json import sys from pathlib import Path @@ -251,11 +252,39 @@ def sandbox(self, tmp_path, monkeypatch): recipe.write_text('{% set version = "2026.9.0" %}\n') changelog = tmp_path / "CHANGELOG.md" changelog.write_text("# Changelog\n\n## Unreleased\n\n- Something.\n") + # Redirected like every other file `main()` rewrites. Without this + # the sandbox leaked: `server.json` was added to the release script + # after this fixture was written, so running the suite rewrote the + # real manifest in the working tree to whatever version the last + # test happened to use. Harmless on a normal branch, because the + # value it lands on matches what is committed -- but during a + # release it silently reverted the version bump, and + # `test_server_manifest.py` then failed on a file nobody had + # edited. + server_json = tmp_path / "server.json" + server_json.write_text( + json.dumps( + { + "name": "io.github.UXARRAY/uxarray-mcp-server", + "version": "2026.9.0", + "packages": [ + { + "registryType": "pypi", + "identifier": "uxarray-mcp", + "version": "2026.9.0", + } + ], + }, + indent=2, + ) + + "\n" + ) monkeypatch.setattr(prepare_release, "PYPROJECT", pyproject) monkeypatch.setattr(prepare_release, "INIT", init) monkeypatch.setattr(prepare_release, "CONDA_RECIPE", recipe) monkeypatch.setattr(prepare_release, "CHANGELOG", changelog) + monkeypatch.setattr(prepare_release, "SERVER_JSON", server_json) monkeypatch.setattr(prepare_release, "_relock", lambda version: None) monkeypatch.setattr(prepare_release, "_latest_tag", lambda: "v2026.9.0") @@ -263,7 +292,12 @@ def sandbox(self, tmp_path, monkeypatch): output.write_text("") monkeypatch.setenv("GITHUB_OUTPUT", str(output)) monkeypatch.setattr(sys, "argv", ["prepare_release.py"]) - return {"pyproject": pyproject, "changelog": changelog, "output": output} + return { + "pyproject": pyproject, + "changelog": changelog, + "output": output, + "server_json": server_json, + } def _outputs(self, sandbox) -> dict[str, str]: return dict( @@ -359,3 +393,52 @@ def test_the_shipped_pin_matches_exactly_once(self): def test_the_shipped_floor_is_readable(self): assert _parse_version(prepare_release._current_uxarray_floor()) + + +class TestTheSandboxCoversEveryFileTheScriptRewrites: + """A release-script test must not edit the working tree. + + ``server.json`` was added to ``prepare_release.py`` after the sandbox + fixture was written, and the fixture was not extended, so running the + suite rewrote the real manifest. On a normal branch that is invisible + -- the version it lands on is the one already committed. During a + release it is not: the suite reverted the version bump mid-run, and + ``test_server_manifest.py`` failed on a file nobody had touched. + + This fails when the next such constant is added, rather than during + the next release. + """ + + #: Module-level paths that ``main()`` writes to. Anything here must be + #: redirected by the sandbox fixture above. + WRITTEN_PATHS = ("PYPROJECT", "INIT", "CONDA_RECIPE", "CHANGELOG", "SERVER_JSON") + + def test_every_written_path_is_redirected(self, tmp_path, monkeypatch): + """Applies the same fixture, then checks nothing was left behind.""" + cls = TestTheDailyPollDecidesWhenToRelease + cls.sandbox.__wrapped__(cls(), tmp_path, monkeypatch) + for name in self.WRITTEN_PATHS: + value = Path(getattr(prepare_release, name)) + assert tmp_path in value.parents, ( + f"prepare_release.{name} still points at {value}, so a test " + "that calls main() edits the real file" + ) + + def test_no_write_target_is_missing_from_the_list(self): + """Catches a new ``_write_*`` helper nobody added above.""" + import inspect + + source = inspect.getsource(prepare_release) + declared = { + name + for name in dir(prepare_release) + if name.isupper() and isinstance(getattr(prepare_release, name), Path) + } + # ROOT is a directory, not a file the script rewrites. + declared.discard("ROOT") + unlisted = sorted(declared - set(self.WRITTEN_PATHS)) + for name in unlisted: + assert f"{name}.write_text" not in source, ( + f"prepare_release.{name} is written but not in WRITTEN_PATHS, " + "so the sandbox does not redirect it" + ) diff --git a/tests/test_globus_transfer.py b/tests/test_globus_transfer.py index 80a25b4..b2635c6 100644 --- a/tests/test_globus_transfer.py +++ b/tests/test_globus_transfer.py @@ -69,7 +69,7 @@ def _profile(**overrides) -> GlobusTransferProfile: base = { "remote_collection_id": "remote-uuid", "local_collection_id": "local-uuid", - "remote_write_root": "/scratch/rjain", + "remote_write_root": "/scratch/demo", "remote_read_root": "/lcrc/group/e3sm", "collection_roots": (), } @@ -189,7 +189,7 @@ def test_an_unmatched_path_passes_through_unchanged(self): # path rewritten under the wrong root names a real file nobody asked # about. assert ( - to_collection_path("/home/rjain/out.nc", ("/lcrc",)) == "/home/rjain/out.nc" + to_collection_path("/home/demo/out.nc", ("/lcrc",)) == "/home/demo/out.nc" ) def test_a_prefix_sibling_does_not_match_a_root(self): @@ -231,7 +231,7 @@ def test_an_upload_lands_under_the_write_root(self, tmp_path): source.write_text("x") service = TransferService(_profile(), FakeTransferClient()) plan = service.plan_put(source, "runs/mesh.nc") - assert plan.destination_path == "/scratch/rjain/runs/mesh.nc" + assert plan.destination_path == "/scratch/demo/runs/mesh.nc" assert plan.source_path == str(source.resolve()) assert plan.recursive is False @@ -264,14 +264,14 @@ def test_the_write_root_is_readable_too(self, tmp_path): # Somewhere you may put a file is somewhere you may look at one: a # download of what was just uploaded must not fail on the read root. service = TransferService(_profile(), FakeTransferClient()) - plan = service.plan_get("/scratch/rjain/out.nc", tmp_path / "out.nc") - assert plan.source_path == "/scratch/rjain/out.nc" + plan = service.plan_get("/scratch/demo/out.nc", tmp_path / "out.nc") + assert plan.source_path == "/scratch/demo/out.nc" def test_a_path_under_no_readable_root_names_them_all(self, tmp_path): service = TransferService(_profile(), FakeTransferClient()) with pytest.raises(PathOutsideRoot) as excinfo: - service.plan_get("/home/rjain/out.nc", tmp_path / "out.nc") - assert "/scratch/rjain" in str(excinfo.value) + service.plan_get("/home/demo/out.nc", tmp_path / "out.nc") + assert "/scratch/demo" in str(excinfo.value) assert "/lcrc/group/e3sm" in str(excinfo.value) def test_a_read_root_does_not_widen_writes(self, tmp_path): @@ -284,7 +284,7 @@ def test_a_read_root_does_not_widen_writes(self, tmp_path): def test_without_a_read_root_reads_fall_back_to_the_write_root(self, tmp_path): service = TransferService(_profile(remote_read_root=None), FakeTransferClient()) assert ( - service.plan_get("out.nc", tmp_path).source_path == "/scratch/rjain/out.nc" + service.plan_get("out.nc", tmp_path).source_path == "/scratch/demo/out.nc" ) with pytest.raises(PathOutsideRoot): service.plan_get("/lcrc/group/e3sm/out.nc", tmp_path) @@ -305,7 +305,7 @@ def test_collection_translation_applies_to_the_submitted_path(self, tmp_path): _profile(collection_roots=("/scratch",)), FakeTransferClient() ) plan = service.plan_put(source, "runs/mesh.nc") - assert plan.destination_path == "/rjain/runs/mesh.nc" + assert plan.destination_path == "/demo/runs/mesh.nc" class TestAnEndpointWithoutTheBlockMovesNoFiles: @@ -341,7 +341,7 @@ def test_a_submission_id_is_fetched_and_carried(self, tmp_path): assert payload["submission_id"] == "submission-1" assert payload["source_endpoint"] == "local-uuid" assert payload["destination_endpoint"] == "remote-uuid" - assert payload["DATA"][0]["destination_path"] == "/scratch/rjain/runs/mesh.nc" + assert payload["DATA"][0]["destination_path"] == "/scratch/demo/runs/mesh.nc" assert result["task_id"] == "task-1" def test_checksums_are_verified_and_mail_is_not_sent(self, tmp_path): diff --git a/tests/test_hpc_safety.py b/tests/test_hpc_safety.py index e83d511..5250565 100644 --- a/tests/test_hpc_safety.py +++ b/tests/test_hpc_safety.py @@ -108,10 +108,14 @@ def test_globus_offline_maps_to_offline(self): def test_yac_pythonpath_is_expected_runtime_path(self): """Endpoint-side YAC source/runtime paths are not a worker leak.""" + # The last entry is a group checkout, which on a real cluster sits + # under someone's username. Written with a placeholder because the + # check keys on "yac" appearing in the path, not on whose it is -- + # a real name here reads like the rule depends on it. pythonpath = ( "/home/testuser/src/yac/build/python:" "/home/testuser/local/yac-3.17/lib/python3.12/site-packages:" - "/lcrc/group/e3sm/jain/uxarray-yac-src" + "/lcrc/group/e3sm//uxarray-yac-src" ) assert health._is_expected_yac_pythonpath(pythonpath) is True diff --git a/tests/test_mcp_protocol.py b/tests/test_mcp_protocol.py index 8ef7ce2..7baca65 100644 --- a/tests/test_mcp_protocol.py +++ b/tests/test_mcp_protocol.py @@ -298,10 +298,16 @@ async def test_calling_an_unknown_tool_is_handled(self): name means nothing ran -- so this asserts the new contract rather than pinning the old one. """ - from mcp.shared.exceptions import MCPError + # `McpError` on mcp 1.x, renamed `MCPError` in 2.x. Importing one + # name fails on the other SDK, and the floor job is the only place + # that resolves 1.x -- so asking for the wrong one is invisible + # until the weekly run goes red, which is how this arrived. + import mcp.shared.exceptions as _exc + + mcp_error = getattr(_exc, "MCPError", None) or _exc.McpError async with connect() as (session, _init): - with pytest.raises(MCPError) as err: + with pytest.raises(mcp_error) as err: await session.call_tool("no_such_tool_exists", {}) assert "no_such_tool_exists" in str(err.value) diff --git a/tests/test_remote_agent.py b/tests/test_remote_agent.py index 0bd45ea..0566fde 100644 --- a/tests/test_remote_agent.py +++ b/tests/test_remote_agent.py @@ -1,5 +1,6 @@ """Tests for remote execution via Academy agent and Globus Compute.""" +import types from unittest.mock import MagicMock, patch import pytest @@ -448,3 +449,150 @@ def test_remote_inspect_variable_invalid_name(self, synthetic_mesh_with_data): with pytest.raises(ValueError, match="Variable 'salinity' not found"): remote_inspect_variable(grid_file, data_file, "salinity") + + +class TestALostWorkerIsDiagnosedByWhatItWasRunning: + """``WorkerLost`` has two very different causes, and one message. + + Memory is the usual one, and the standing advice -- check the mesh size, + subset first -- is right for it. It is wrong for a YAC-backed remap, and + expensively so: subsetting cannot fix an MPI abort, so the caller spends + the afternoon measuring a mesh that was never the problem. + + Observed on Casper. ``method="conservative"`` killed the worker, while + the identical remap ran in 0.09s under ``check_remote_yac`` -- which + executes it in a worker-side *subprocess*. Same node, same YAC, same + method: the only difference is whether an ``MPI_Init`` abort can take + the worker down with it. + """ + + WORKER_LOST = ( + "Task failure due to loss of worker 0 on host crhtc52\n" + "***** One common cause of WorkerLost exceptions is Python version " + "mismatch *****" + ) + + @staticmethod + def _normalize(**kw): + from uxarray_mcp.remote.agent import _normalize_remote_error + + cfg = types.SimpleNamespace(endpoint_id="x", default_endpoint="ucar") + return str( + _normalize_remote_error( + RuntimeError(TestALostWorkerIsDiagnosedByWhatItWasRunning.WORKER_LOST), + cfg, + **kw, + ) + ) + + def test_an_ordinary_lost_worker_still_blames_memory(self): + """The common case must not regress into MPI advice.""" + msg = self._normalize() + assert "memory" in msg.lower() + assert "MPI" not in msg + + def test_a_yac_remap_is_told_about_mpi_instead(self): + msg = self._normalize(yac_backed=True) + assert "MPI" in msg + assert "check_remote_yac" in msg + + def test_the_mpi_message_does_not_send_anyone_to_subset(self): + """Subsetting is the memory remedy and cannot fix an abort.""" + msg = self._normalize(yac_backed=True) + assert "subset" not in msg.lower() + + def test_the_fallback_it_offers_is_qualified(self): + """idw/bilinear do run -- but they are not conservative. + + Offering them without saying so trades a loud failure for a quiet + wrong answer, which is the trade this server exists to refuse. + """ + msg = self._normalize(yac_backed=True) + assert "inverse_distance_weighted" in msg + assert "conserve" in msg.lower() + + +class TestTheYacRouteIsRecognisedFromPositionalArguments: + """Every call into ``_run_on_hpc`` passes its arguments positionally. + + The first version of this read only ``kwargs`` and so never fired: a + real conservative remap on Casper still printed the memory advice. The + detection has to look at the positional arguments too. + """ + + @staticmethod + def _looks(*args, **kwargs): + from uxarray_mcp.remote.agent import _looks_yac_backed + + return _looks_yac_backed(args, kwargs or None) + + def test_a_positional_method_is_seen(self): + assert self._looks("/g.nc", "/d.nc", "rrfac", "conservative") + + def test_a_positional_backend_is_seen(self): + assert self._looks("/g.nc", "/d.nc", "rrfac", "yac") + + def test_keywords_still_work(self): + assert self._looks(method="conservative") + assert self._looks(backend="yac") + + @pytest.mark.parametrize("method", ["nnn", "dnn", "average"]) + def test_the_other_yac_methods_count(self, method): + """All four route through YAC, not just the conservative one.""" + assert self._looks("/g.nc", method) + + def test_an_ordinary_remap_is_not_flagged(self): + assert not self._looks("/g.nc", "/d.nc", "rrfac", "nearest_neighbor") + assert not self._looks("/g.nc", "/d.nc", "rrfac", "bilinear") + + def test_a_path_that_merely_contains_the_word_is_not_flagged(self): + """`yac` appears in real cluster paths; only a whole argument counts.""" + assert not self._looks("/glade/u/home/x/uxarray-yac-src/grid.nc") + + +class TestTheDiagnosisSurvivesTheCallPath: + """The routing, not just the predicate. + + ``_looks_yac_backed`` can be perfectly correct and still never fire if + ``_run_on_hpc`` hands it the wrong thing. That is exactly what happened: + the first version passed only ``kwargs``, every caller passes its + arguments positionally, and a real conservative remap on Casper still + got the memory advice. Testing the predicate alone did not catch it. + """ + + async def test_a_conservative_remap_through_run_on_hpc_gets_mpi_advice( + self, monkeypatch + ): + from uxarray_mcp.remote import agent as agent_mod + + lost = RuntimeError("Task failure due to loss of worker 0 on host crhtc52") + + class _Future: + def result(self, *_a, **_k): + raise lost + + class _Executor: + def submit(self, *_a, **_k): + return _Future() + + cfg = types.SimpleNamespace( + endpoint_id="abc", default_endpoint="ucar", timeout_seconds=5 + ) + ag = UXarrayComputeAgent.__new__(UXarrayComputeAgent) + ag.config = cfg + monkeypatch.setattr( + agent_mod, "_record_task_handle_when_assigned", lambda *a, **k: None + ) + monkeypatch.setattr( + ag, "_get_executor", lambda *a, **k: _Executor(), raising=False + ) + + with pytest.raises(RuntimeError) as err: + # Positional, exactly as every real caller does it. + await ag._run_on_hpc( + lambda *a: None, "/g.nc", "/d.nc", "rrfac", "conservative" + ) + assert "MPI" in str(err.value), ( + "a positionally-passed conservative remap lost its diagnosis " + "somewhere between _run_on_hpc and the error handler" + ) diff --git a/tests/test_remote_remap_live.py b/tests/test_remote_remap_live.py index cd679e7..566859e 100644 --- a/tests/test_remote_remap_live.py +++ b/tests/test_remote_remap_live.py @@ -9,9 +9,10 @@ these are read by the worker. Verified against Chrysalis on 2026-10-13:: UXMCP_LIVE_ENDPOINT=chrysalis \\ - UXMCP_LIVE_GRID=/lcrc/group/e3sm/jain/uxarray-yac-src/test/meshfiles/mpas/QU/480/grid.nc \\ - UXMCP_LIVE_DATA=/lcrc/group/e3sm/jain/uxarray-yac-src/test/meshfiles/mpas/QU/480/data.nc \\ - UXMCP_LIVE_TARGET_GRID=/lcrc/group/e3sm/jain/uxarray-yac-src/test/meshfiles/mpas/dyamond-30km/gradient_grid_subset.nc \\ + MESH=/lcrc/group/e3sm//uxarray-yac-src/test/meshfiles \\ + UXMCP_LIVE_GRID=$MESH/mpas/QU/480/grid.nc \\ + UXMCP_LIVE_DATA=$MESH/mpas/QU/480/data.nc \\ + UXMCP_LIVE_TARGET_GRID=$MESH/mpas/dyamond-30km/gradient_grid_subset.nc \\ UXMCP_LIVE_VARIABLE=bottomDepth \\ uv run --no-sync pytest tests/test_remote_remap_live.py -m live -v diff --git a/tests/test_transfer_tools.py b/tests/test_transfer_tools.py index d494054..3dbff18 100644 --- a/tests/test_transfer_tools.py +++ b/tests/test_transfer_tools.py @@ -23,7 +23,7 @@ def _transfer_profile(**overrides) -> GlobusTransferProfile: base = { "remote_collection_id": "remote-uuid", "local_collection_id": "local-uuid", - "remote_write_root": "/scratch/rjain", + "remote_write_root": "/scratch/demo", "remote_read_root": "/lcrc/group/e3sm", } base.update(overrides) @@ -162,12 +162,12 @@ def test_a_path_outside_the_root_is_reported_not_raised(self, monkeypatch): def test_an_endpoint_without_a_block_says_what_to_add(self, monkeypatch): monkeypatch.setattr(transfer_tools, "_config", lambda: (_config(None), None)) - result = transfer_tools.transfer_ls("/scratch/rjain") + result = transfer_tools.transfer_ls("/scratch/demo") assert result["submitted"] is False assert "globus_transfer" in result["message"] def test_every_result_carries_provenance_and_an_operation_id(self, wired): - result = transfer_tools.transfer_ls("/scratch/rjain") + result = transfer_tools.transfer_ls("/scratch/demo") assert result["_provenance"]["operation_id"] assert result["_provenance"]["tool"] == "transfer_ls" @@ -239,7 +239,7 @@ def test_the_probe_lists_the_write_root(self, monkeypatch): class Client: def operation_ls(self, collection_id, **kwargs): - assert kwargs["path"] == "/scratch/rjain" + assert kwargs["path"] == "/scratch/demo" return {"DATA": [{"name": "runs", "type": "dir"}]} monkeypatch.setattr( diff --git a/uv.lock b/uv.lock index b585fb5..36102d0 100644 --- a/uv.lock +++ b/uv.lock @@ -1193,11 +1193,11 @@ wheels = [ [[package]] name = "filelock" -version = "4.1.1" +version = "4.2.4" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/35/c8/1d457d9150ff948f2ce6ada7715e0eeebbe5d3b58a45271a1e222474bcd3/filelock-4.1.1.tar.gz", hash = "sha256:7ba0927482c5a814b0a7f391d029ccdb8010f576f0a74c0dcde1811e8bc4c1b6", size = 563430, upload-time = "2026-10-11T16:11:54.373Z" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/28/3893324bcadf8f89b1616637468d766af3f625008c0735bf704fdfa86617/filelock-4.2.4.tar.gz", hash = "sha256:c6d8663535e1650ee430507b4ffbac7b464dd64b7647ea0112a9f406499d163d", size = 652444, upload-time = "2026-10-19T01:57:51.001Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/d7/8b/f837f52905395ba4510fe61f753c24833fb0a9c76e21267bb9f828b664a9/filelock-4.1.1-py3-none-any.whl", hash = "sha256:3f4a557945a7b0f95efeb1f432267affe5d45ac8ddde2aed1b97ebb62382c089", size = 132460, upload-time = "2026-10-11T16:11:52.753Z" }, + { url = "https://files.pythonhosted.org/packages/e0/1b/5e74289a1878e666bb4b65fe5805a04cafcac2aa5f519ef1ead67ef91c56/filelock-4.2.4-py3-none-any.whl", hash = "sha256:301241438c57936ef93f4243a94f2559022268bfcf9b375a9a72eb0f8db8358e", size = 162475, upload-time = "2026-10-19T01:57:49.419Z" }, ] [[package]] @@ -5137,7 +5137,7 @@ viz = [ [[package]] name = "uxarray-mcp" -version = "2026.9.1" +version = "2026.10.0" source = { editable = "." } dependencies = [ { name = "holoviews" },