From cfa6b8a95a6fdd9ef9174b7b75e496d0d99b0f02 Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Mon, 26 Oct 2026 14:02:01 -0500 Subject: [PATCH 1/6] Tell a YAC remap that died what actually killed it A conservative remap on the UCAR endpoint comes back as WorkerLost, and the message says it is memory and to subset first. It is not memory, and subsetting cannot fix it, so that advice costs an afternoon. What is actually happening, measured on Casper: the identical remap runs in 0.09 seconds under check_remote_yac and kills the worker through run_analysis. Same node, same YAC 3.18, same method, same mesh. The only difference is that the probe runs it in a worker-side subprocess, so an MPI_Init abort takes the child instead of the worker. Open MPI initialises as a singleton and is fine; MPICH aborts, and an abort inside a Globus worker surfaces here with nothing mentioning MPI. Three changes. The WorkerLost handler now asks what the call was doing. A YAC-backed method -- conservative, nnn, dnn, average, or an explicit backend="yac" -- gets the MPI explanation, how to confirm it (check_remote_yac, whose conservative_ok separates "YAC is broken" from "YAC cannot run in this process"), and what to do instead. The fallback it offers is qualified: inverse_distance_weighted and bilinear do run, and neither conserves the field integral, which is the only reason to have asked for conservative. Every other lost worker still gets the memory advice, which is right for it. check_remote_yac now exercises the conservative path. It only ever ran nearest_neighbor, which uses uxarray's own engine and never calls into YAC -- so it reported "YAC OK" on a worker where every conservative remap died. Importing YAC and computing weights with it are different capabilities and the probe only tested the first. The new block is separate from the existing one so a site that can import YAC but not compute weights is reported as exactly that, and it catches BaseException because a YAC failure can raise SystemExit. conservative_ok is deliberately not part of the exit status: a site that only wants the non-YAC methods is not broken. ucar_endpoint.sh gains check-mpi, which runs the bare import and the launched import side by side and says which case you are in. Chrysalis has had check-yac under srun since it hit this; UCAR had nothing. Not automated: the repair. Putting workers inside an MPI context means GlobusMPIEngine or a launcher prefix on the engine, not an export in worker_init -- worker_init runs before the worker process exists. A wrong engine block takes the endpoint down for every tool rather than just the remap, so check-mpi prints what to change and stops. The regression tests cover the routing, not only the predicate. _looks_yac_backed can be correct and never fire if _run_on_hpc hands it the wrong thing, which is what the first draft of this did: it read only kwargs, every caller passes positionally, and a real remap still got the memory advice with the unit tests green. --- CHANGELOG.md | 97 +++++++++++++ conda/recipe/meta.yaml | 2 +- pyproject.toml | 2 +- scripts/ucar_endpoint.sh | 78 +++++++++++ server.json | 4 +- src/uxarray_mcp/__init__.py | 2 +- src/uxarray_mcp/remote/agent.py | 64 ++++++++- src/uxarray_mcp/remote/compute_functions.py | 35 +++++ tests/test_calver_release.py | 85 ++++++++++- tests/test_mcp_protocol.py | 10 +- tests/test_remote_agent.py | 148 ++++++++++++++++++++ uv.lock | 8 +- 12 files changed, 521 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e0618f..838ed07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,103 @@ built against; see `docs/release.md`. Versions through `0.3.1` were SemVer. ## Unreleased +## 2026.10.0 — 2026-10-23 +### Added +- `subset_advanced` selects faces near a point, by radius + (`subset.bounding_circle`) or by neighbour count + (`subset.nearest_neighbor`). An empty selection is refused with the mesh + extent, the same rule the bounding box already followed; UXarray's own + message says the circle was empty without saying where a circle would + not be. +- `regrid_elements` carries a field between nodes, edges and faces through + the `topological_*` reductions. This is what makes node-centred output + usable: every vector-calculus operation needs face-centred data, and the + refusal a caller got had no repair behind it because the server could not + run the standard one. +- `domain/uxarray_surface.py` records which of UXarray this server reaches + and which it deliberately does not, one verdict per method with a reason. + Tests hold it to the installed library, so a new method upstream fails + until someone decides about it and a renamed one fails the day the rename + lands rather than the next time a user follows our advice into a method + that is gone. Deliberately not a count: "we expose N of M" goes red on + every upstream release for reasons nobody can act on. + + All three came from replaying the notebooks in UXARRAY/uxarray-community + through the server. They are separate tools rather than new `run_analysis` + operations because #89 already measures that front door as too large to + describe itself. + +### Changed +- Floors `uxarray[geo,viz]>=2026.10.0`. The previous pin's `<2026.10` + ceiling excluded the release outright. The resolve is quiet -- 189 + packages, uxarray the only one that moves -- and 2026.10.0's `viz` extra + caps `cartopy<0.26`, which settles UXARRAY/uxarray#1773 upstream rather + than here. +- Floors `toolregistry>=0.19.0` and `toolregistry-server>=0.7.0`. 0.19 + carries the heterogeneous-union fix (Oaklight/ToolRegistry#269) this + project has been waiting on. What we serve for `lat_spec` is still not + upstream's rendering: `tuple` and `list` both become `array`, so the + union arrives with the same branch twice and untyped items. The schema + helper now replaces that rendering rather than filling an absence, and + the core tool specification got smaller as a result -- 36271 to 36143 + bytes. +- The container declares its Python version once. It was written in four + places, two `FROM` lines and two `--python 3.12` flags, and Dependabot + rewrites `FROM` and nothing else -- so its 3.14 bump produced an image + whose base had no 3.12 beside a sync that demanded one. +- `SETUP.sh` reproduces CI. It ran a bare `uv sync` and an unfiltered + `pytest`, from before the suite moved to marker selection, so following + it produced 19 failures that were all missing optional dependencies. It + also gated on the ambient `python3`, which is not the interpreter the + project runs on: a machine with 3.9 on `PATH` could not run the script at + all while `uv sync` on that same machine builds a working 3.12 + environment. + +### Fixed +- Tool annotations reach clients on mcp 1.x, not only 2.x. At the declared + floor the shim attached nothing, so every tool was served with no hints + at all -- the `readOnlyHint` a client reads to decide what it may + auto-approve was absent exactly where it is documented as present. Three + SDK differences, each independently sufficient: the handler dict is keyed + by method string on 2.x and by request type on 1.x, 1.x returns a + `ServerResult` envelope whose `.root` holds the tools, and the attribute + itself is public on one and private on the other. +- Our annotations outrank inferred ones. toolregistry-server 0.7.0 began + attaching annotations of its own and marked `manage_session` and + `resume_workflow` `readOnlyHint=True`; the first resets session state and + clears artifacts, the second runs pending workflow steps. Nine tools left + deliberately unrated came back carrying hints. Where this server has + rated a tool its rating is what goes out, and where it has not, nothing + does -- absence is honest, an unchecked inference is not. +- `get_capabilities` says which of what it lists is executable. It + advertises native UXarray calls, several of which have no operation + behind them -- `grid.get_ball_tree()`, `var.integrate()` -- and nothing + distinguished them, so a model reading the result would plan a call that + does not exist. +- The "not face-centered" refusals name their repair. They were correct and + a dead end: the standard fix is `topological_mean("face")`, which this + server could not run. They now point at `regrid_elements`, and a test + follows that advice to confirm it works rather than landing on a second + error. +- An unknown tool name is reported as JSON-RPC `-32602` rather than a + result carrying `isError`, following toolregistry-server 0.7.0. That is + the correct reading: `isError` is for a tool that ran and failed, and an + unknown name means nothing ran. +- The oversized-SCRIP test fixture no longer needs 8 GiB. It built each + dask array as a single 6.5 GiB chunk and `to_netcdf` materialized one at + a time, so the test peaked at 7.45 GiB before the code under test ran. + That fits GitHub's runners and not a laptop. Chunking along + `grid_corners` holds each write to ~191 MiB; the full suite now peaks at + 1.36 GiB. + +### Security +- Clears 49 advisories in the lockfile, all in transitively-resolved + packages: 44 against pillow 12.3.0 (5 Critical), PYSEC-2026-4320 against + certifi 2026.7.22 (9.1 Critical), and four against virtualenv 21.14.7. + certifi is the trust store behind every outbound HTTPS call this server + makes -- Globus Compute submission, transfers, remote dataset reads -- so + a Critical there is not a transitive detail. + ## 2026.9.1 — 2026-09-25 ### Changed - Floors `uxarray[geo,viz]>=2026.9.1`. Upstream 2026.9.1 moved cartopy, diff --git a/conda/recipe/meta.yaml b/conda/recipe/meta.yaml index d3b438b..34d6a8c 100644 --- a/conda/recipe/meta.yaml +++ b/conda/recipe/meta.yaml @@ -1,5 +1,5 @@ {% set name = "uxarray-mcp" %} -{% set version = "2026.9.1" %} +{% set version = "2026.10.0" %} {% set python_min = "3.12" %} package: diff --git a/pyproject.toml b/pyproject.toml index 4e2d667..492b66c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "uxarray-mcp" -version = "2026.9.1" +version = "2026.10.0" description = "MCP server for analyzing unstructured meshes with UXarray" readme = "README.md" keywords = ["uxarray", "mcp", "unstructured grids", "scientific computing", "globus compute"] diff --git a/scripts/ucar_endpoint.sh b/scripts/ucar_endpoint.sh index 67bb84f..3a1e208 100755 --- a/scripts/ucar_endpoint.sh +++ b/scripts/ucar_endpoint.sh @@ -23,6 +23,18 @@ WITH_YAC="${WITH_YAC:-0}" # HEALPix remap ran. worker_init sources activate-yac.sh from this prefix, so # `restart` is what picks the change up. YAC_VERSION="${YAC_VERSION:-3.20.2}" +# Whether this site needs YAC imported under an MPI launcher. +# +# `check-mpi` answers that question; acting on the answer is still manual. +# Open MPI initialises as a singleton and needs nothing. MPICH aborts, and +# the endpoint then needs its *workers* started inside an MPI context -- +# which means `GlobusMPIEngine`, or a `launcher_prefix` on the engine, not an +# export in worker_init. Casper was observed failing this way and has not yet +# been reconfigured, so this is deliberately not automated: a wrong engine +# block takes the endpoint down for every tool, not just the remap. +# +# Tracked rather than guessed at. See the WorkerLost message in +# remote/agent.py for what a caller is told in the meantime. YAC_PREFIX="$HOME/opt/yac-$YAC_VERSION" YAC_ACTIVATE="$YAC_PREFIX/activate-yac.sh" TMUX_SESSION="uxarray-endpoint" @@ -39,6 +51,7 @@ Usage (run on a Casper login node): ucar_endpoint.sh start Load env + start endpoint in tmux ucar_endpoint.sh restart Stop running endpoint, then start fresh ucar_endpoint.sh status Show endpoint list + ucar_endpoint.sh check-mpi Does importing YAC need an MPI launcher? Environment overrides: ENDPOINT_NAME Globus Compute endpoint profile name (default: ucar-uxarray-yac) @@ -109,6 +122,64 @@ EOF echo " wrote $YAC_ACTIVATE (YAC $YAC_VERSION, python $pyver)" } +# Does importing YAC survive outside an MPI launcher? +# +# Importing YAC calls MPI_Init. Open MPI initialises as a singleton and is +# fine; MPICH aborts the process instead. Inside a Globus Compute worker that +# abort takes the worker down, and the caller sees `WorkerLost` with nothing +# mentioning MPI -- which reads as an out-of-memory kill and sends people to +# measure their mesh. Observed on Casper: `method="conservative"` killed the +# worker while the identical remap finished in 0.09s under +# `check_remote_yac`, which runs it in a subprocess. +# +# This answers the question directly, before anyone submits a remap. If the +# bare import dies and the launched one lives, the endpoint's worker_init +# needs a launcher. +_check_mpi() { + _load_modules + _activate_env + local probe='import yac.core, sys; print("import ok:", sys.executable)' + + echo "==> Bare import (what a Globus worker does today)" + if python -c "$probe" 2>&1 | sed 's/^/ /'; then + local bare_ok=1 + else + local bare_ok=0 + echo " FAILED -- this is what takes the worker down." + fi + + echo + echo "==> Under a launcher" + local launcher="" + for cand in "mpiexec -n 1" "srun --ntasks 1"; do + if command -v "${cand%% *}" >/dev/null 2>&1; then launcher="$cand"; break; fi + done + if [[ -z "$launcher" ]]; then + echo " No mpiexec or srun on PATH; cannot compare." + return 1 + fi + echo " using: $launcher" + local launched_ok=1 + $launcher python -c "$probe" 2>&1 | sed 's/^/ /' || launched_ok=0 + + echo + if [[ "$bare_ok" == "1" ]]; then + echo "VERDICT: bare import works. MPI is not what is killing the worker;" + echo " if conservative remaps still fail, look at memory next." + elif [[ "$launched_ok" == "1" ]]; then + echo "VERDICT: bare import dies, launched import works -- this is the MPI" + echo " singleton problem." + echo " The workers must start inside an MPI context. In the endpoint's" + echo " user_config_template.yaml.j2, either switch the engine to" + echo " GlobusMPIEngine, or give the current engine a launcher prefix so" + echo " each worker runs under: $launcher" + echo " Then: $(basename "$0") restart" + else + echo "VERDICT: YAC does not import either way. This is not MPI --" + echo " check the build at $YAC_PREFIX." + fi +} + _check_yac() { if python -c 'import yac; import yac.core; print(" YAC OK:", yac.core.__file__)' 2>/dev/null; then return 0 @@ -134,6 +205,12 @@ _configure() { if [[ "$WITH_YAC" == "1" ]]; then _ensure_yac_activate yac_line=$'\n source '"$YAC_ACTIVATE" + # NOTE: if `check-mpi` says this site needs a launcher, the change goes + # in the engine block below rather than here -- worker_init runs *before* + # the worker process starts, so exporting something here cannot put that + # process inside an MPI context. Left unwired on purpose rather than + # shipped as a setting that looks applied and does nothing. See the + # YAC_LAUNCHER note at the top of this file. elif [[ "$ENDPOINT_NAME" == *yac* ]]; then echo "ERROR: '$ENDPOINT_NAME' is named for YAC but WITH_YAC is 0." >&2 echo " Rebuild it with YAC: WITH_YAC=1 $(basename "$0") configure" >&2 @@ -261,5 +338,6 @@ case "${1:-}" in _do_start) _do_start ;; # internal: invoked by tmux restart) _restart ;; status) _status ;; + check-mpi) _check_mpi ;; *) usage; exit 1 ;; esac diff --git a/server.json b/server.json index 11ac978..790739c 100644 --- a/server.json +++ b/server.json @@ -3,7 +3,7 @@ "name": "io.github.UXARRAY/uxarray-mcp-server", "title": "UXarray MCP", "description": "Inspect, subset, remap and plot unstructured climate meshes, locally or on HPC.", - "version": "2026.9.1", + "version": "2026.10.0", "websiteUrl": "https://uxarray-mcp-server.readthedocs.io", "repository": { "source": "github", @@ -13,7 +13,7 @@ { "registryType": "pypi", "identifier": "uxarray-mcp", - "version": "2026.9.1", + "version": "2026.10.0", "runtimeHint": "uvx", "transport": { "type": "stdio" diff --git a/src/uxarray_mcp/__init__.py b/src/uxarray_mcp/__init__.py index 808ac90..deaeff2 100644 --- a/src/uxarray_mcp/__init__.py +++ b/src/uxarray_mcp/__init__.py @@ -3,4 +3,4 @@ from uxarray_mcp.tools import inspect_mesh __all__ = ["inspect_mesh"] -__version__ = "2026.9.1" +__version__ = "2026.10.0" diff --git a/src/uxarray_mcp/remote/agent.py b/src/uxarray_mcp/remote/agent.py index 62733e7..672e168 100644 --- a/src/uxarray_mcp/remote/agent.py +++ b/src/uxarray_mcp/remote/agent.py @@ -40,7 +40,37 @@ def action(fn): # type: ignore[no-redef] from .config import HPCConfig -def _normalize_remote_error(exc: Exception, config: Any) -> Exception: +#: Remap methods that compute their weights inside YAC. +#: +#: Kept here rather than imported from ``domain.remap_backend`` because this +#: module is on the submit path and the comparison is a membership test, not +#: a resolution. The names match what a caller passes as ``method``. +_YAC_BACKED_METHODS = frozenset({"conservative", "nnn", "dnn", "average"}) + + +def _looks_yac_backed(args: Any = (), kwargs: Any = None) -> bool: + """Whether the call that just died was asking YAC to compute weights. + + Scans the positional arguments as well as the keywords. Every call into + :meth:`_run_on_hpc` passes its arguments positionally, so reading only + ``kwargs`` found nothing and the YAC branch below never fired -- the + first version of this did exactly that and still printed the memory + advice for a conservative remap. + + A plain membership test over the strings is enough and stays correct if + an argument is inserted: these method names are specific enough that a + path or variable name will not collide with them. + """ + values = [a for a in (args or ()) if isinstance(a, str)] + if isinstance(kwargs, dict): + values += [v for v in kwargs.values() if isinstance(v, str)] + lowered = {v.lower() for v in values} + return "yac" in lowered or bool(lowered & _YAC_BACKED_METHODS) + + +def _normalize_remote_error( + exc: Exception, config: Any, *, yac_backed: bool = False +) -> Exception: """Collapse a worker traceback into the same concise error a local run raises. Globus Compute re-raises worker failures with the full remote traceback @@ -105,6 +135,34 @@ def _normalize_remote_error(exc: Exception, config: Any) -> Exception: where = ( f" (worker {worker.group(1)} on {host.group(1)})" if host and worker else "" ) + if yac_backed: + # Memory is the usual cause of a lost worker, but not for this + # call. Importing YAC runs ``MPI_Init``; Open MPI initialises + # fine as a singleton, MPICH aborts the process instead, and an + # abort inside a Globus worker arrives here as WorkerLost with + # nothing mentioning MPI. Observed on Casper, where the same + # conservative remap that kills the worker completes in 0.09s + # under ``check_remote_yac`` -- which runs it in a worker-side + # *subprocess*. Same node, same YAC, same method: the difference + # is only whether an abort can take the worker with it. + # + # Sending the caller to look at mesh size here costs them the + # afternoon, because subsetting cannot fix it. + return RuntimeError( + f"The worker on {endpoint} died running a YAC-backed remap" + f"{where}. This is usually MPI, not memory: importing YAC " + "calls MPI_Init, and on an MPICH site that aborts the " + "process rather than initialising as a singleton, which " + "arrives here as a lost worker. Confirm with " + "`check_remote_yac(endpoint=...)`: if its `conservative_ok` " + "is true, YAC itself is fine and the endpoint needs its " + "workers launched under an MPI launcher (`mpiexec -n 1` or " + "`srun --ntasks 1`) in worker_init. Until then, " + "method='inverse_distance_weighted' or 'bilinear' run on " + "uxarray's own engine and do not need YAC -- but note that " + "neither conserves the field integral, which is the reason " + "to ask for conservative in the first place." + ) return RuntimeError( f"The worker on {endpoint} died while running this task{where}. " "The usual cause is memory: the worker was killed, not the task. " @@ -796,7 +854,9 @@ async def _run_on_hpc(self, func, *args, **kwargs) -> Dict[str, Any]: None, future.result, self.config.timeout_seconds ) except Exception as exc: - raise _normalize_remote_error(exc, self.config) from None + raise _normalize_remote_error( + exc, self.config, yac_backed=_looks_yac_backed(args, kwargs) + ) from None # Attach provenance with the correct HPC venue — the remote functions # are self contained and don't call attach_provenance themselves. diff --git a/src/uxarray_mcp/remote/compute_functions.py b/src/uxarray_mcp/remote/compute_functions.py index 9c63376..274b60c 100644 --- a/src/uxarray_mcp/remote/compute_functions.py +++ b/src/uxarray_mcp/remote/compute_functions.py @@ -3302,7 +3302,42 @@ def _surface(mod): out["remap_error"] = f"{type(exc).__name__}: {exc}" out["remap_traceback"] = traceback.format_exc() +# The conservative remap, which is the only reason YAC is here. +# +# Everything above passes with no working YAC weight computation: +# `nearest_neighbor` runs on uxarray's own engine and never calls into the +# library, so the probe reported "YAC OK" on a worker where +# `method="conservative"` killed the Globus worker outright. Importing YAC +# and computing weights with it are different capabilities, and only the +# second one is what a caller asking for a conservative remap needs. +# +# Kept separate from the block above so a site that has YAC importable but +# cannot compute weights is reported as exactly that, rather than as a +# blanket failure. Running inside this subprocess is what makes the check +# safe at all: a YAC/MPI abort takes down the child and is reported, where +# the same call in the worker surfaces as an unexplained WorkerLost. +try: + t0 = time.perf_counter() + conservative = uxda.remap( + destination_grid=dst, method="conservative", backend="yac" + ) + arr = np.asarray(conservative) + out["conservative_ok"] = True + out["conservative_seconds"] = round(time.perf_counter() - t0, 3) + out["conservative_dst_shape"] = list(conservative.shape) + out["conservative_finite"] = int(np.isfinite(arr).sum()) +except BaseException as exc: + # BaseException, not Exception: a YAC/MPI failure can raise SystemExit + # rather than a normal error, and catching only Exception would let the + # probe exit 0 through the bottom of this file with no verdict recorded. + out["conservative_ok"] = False + out["conservative_error"] = f"{type(exc).__name__}: {exc}"[:400] + out["conservative_traceback"] = traceback.format_exc()[-1200:] + print(json.dumps(out)) +# `conservative_ok` is deliberately not part of the exit status. The probe's +# job is to report what works; a site that wants only the non-conservative +# methods is not broken, and `available` already turns on the YAC import. raise SystemExit(0 if out.get("yac_helper_ok") and out.get("remap_ok") else 1) """ diff --git a/tests/test_calver_release.py b/tests/test_calver_release.py index a58b696..2d2850f 100644 --- a/tests/test_calver_release.py +++ b/tests/test_calver_release.py @@ -9,6 +9,7 @@ from __future__ import annotations import importlib.util +import json import sys from pathlib import Path @@ -251,11 +252,39 @@ def sandbox(self, tmp_path, monkeypatch): recipe.write_text('{% set version = "2026.9.0" %}\n') changelog = tmp_path / "CHANGELOG.md" changelog.write_text("# Changelog\n\n## Unreleased\n\n- Something.\n") + # Redirected like every other file `main()` rewrites. Without this + # the sandbox leaked: `server.json` was added to the release script + # after this fixture was written, so running the suite rewrote the + # real manifest in the working tree to whatever version the last + # test happened to use. Harmless on a normal branch, because the + # value it lands on matches what is committed -- but during a + # release it silently reverted the version bump, and + # `test_server_manifest.py` then failed on a file nobody had + # edited. + server_json = tmp_path / "server.json" + server_json.write_text( + json.dumps( + { + "name": "io.github.UXARRAY/uxarray-mcp-server", + "version": "2026.9.0", + "packages": [ + { + "registryType": "pypi", + "identifier": "uxarray-mcp", + "version": "2026.9.0", + } + ], + }, + indent=2, + ) + + "\n" + ) monkeypatch.setattr(prepare_release, "PYPROJECT", pyproject) monkeypatch.setattr(prepare_release, "INIT", init) monkeypatch.setattr(prepare_release, "CONDA_RECIPE", recipe) monkeypatch.setattr(prepare_release, "CHANGELOG", changelog) + monkeypatch.setattr(prepare_release, "SERVER_JSON", server_json) monkeypatch.setattr(prepare_release, "_relock", lambda version: None) monkeypatch.setattr(prepare_release, "_latest_tag", lambda: "v2026.9.0") @@ -263,7 +292,12 @@ def sandbox(self, tmp_path, monkeypatch): output.write_text("") monkeypatch.setenv("GITHUB_OUTPUT", str(output)) monkeypatch.setattr(sys, "argv", ["prepare_release.py"]) - return {"pyproject": pyproject, "changelog": changelog, "output": output} + return { + "pyproject": pyproject, + "changelog": changelog, + "output": output, + "server_json": server_json, + } def _outputs(self, sandbox) -> dict[str, str]: return dict( @@ -359,3 +393,52 @@ def test_the_shipped_pin_matches_exactly_once(self): def test_the_shipped_floor_is_readable(self): assert _parse_version(prepare_release._current_uxarray_floor()) + + +class TestTheSandboxCoversEveryFileTheScriptRewrites: + """A release-script test must not edit the working tree. + + ``server.json`` was added to ``prepare_release.py`` after the sandbox + fixture was written, and the fixture was not extended, so running the + suite rewrote the real manifest. On a normal branch that is invisible + -- the version it lands on is the one already committed. During a + release it is not: the suite reverted the version bump mid-run, and + ``test_server_manifest.py`` failed on a file nobody had touched. + + This fails when the next such constant is added, rather than during + the next release. + """ + + #: Module-level paths that ``main()`` writes to. Anything here must be + #: redirected by the sandbox fixture above. + WRITTEN_PATHS = ("PYPROJECT", "INIT", "CONDA_RECIPE", "CHANGELOG", "SERVER_JSON") + + def test_every_written_path_is_redirected(self, tmp_path, monkeypatch): + """Applies the same fixture, then checks nothing was left behind.""" + cls = TestTheDailyPollDecidesWhenToRelease + cls.sandbox.__wrapped__(cls(), tmp_path, monkeypatch) + for name in self.WRITTEN_PATHS: + value = Path(getattr(prepare_release, name)) + assert tmp_path in value.parents, ( + f"prepare_release.{name} still points at {value}, so a test " + "that calls main() edits the real file" + ) + + def test_no_write_target_is_missing_from_the_list(self): + """Catches a new ``_write_*`` helper nobody added above.""" + import inspect + + source = inspect.getsource(prepare_release) + declared = { + name + for name in dir(prepare_release) + if name.isupper() and isinstance(getattr(prepare_release, name), Path) + } + # ROOT is a directory, not a file the script rewrites. + declared.discard("ROOT") + unlisted = sorted(declared - set(self.WRITTEN_PATHS)) + for name in unlisted: + assert f"{name}.write_text" not in source, ( + f"prepare_release.{name} is written but not in WRITTEN_PATHS, " + "so the sandbox does not redirect it" + ) diff --git a/tests/test_mcp_protocol.py b/tests/test_mcp_protocol.py index 8ef7ce2..7baca65 100644 --- a/tests/test_mcp_protocol.py +++ b/tests/test_mcp_protocol.py @@ -298,10 +298,16 @@ async def test_calling_an_unknown_tool_is_handled(self): name means nothing ran -- so this asserts the new contract rather than pinning the old one. """ - from mcp.shared.exceptions import MCPError + # `McpError` on mcp 1.x, renamed `MCPError` in 2.x. Importing one + # name fails on the other SDK, and the floor job is the only place + # that resolves 1.x -- so asking for the wrong one is invisible + # until the weekly run goes red, which is how this arrived. + import mcp.shared.exceptions as _exc + + mcp_error = getattr(_exc, "MCPError", None) or _exc.McpError async with connect() as (session, _init): - with pytest.raises(MCPError) as err: + with pytest.raises(mcp_error) as err: await session.call_tool("no_such_tool_exists", {}) assert "no_such_tool_exists" in str(err.value) diff --git a/tests/test_remote_agent.py b/tests/test_remote_agent.py index 0bd45ea..0566fde 100644 --- a/tests/test_remote_agent.py +++ b/tests/test_remote_agent.py @@ -1,5 +1,6 @@ """Tests for remote execution via Academy agent and Globus Compute.""" +import types from unittest.mock import MagicMock, patch import pytest @@ -448,3 +449,150 @@ def test_remote_inspect_variable_invalid_name(self, synthetic_mesh_with_data): with pytest.raises(ValueError, match="Variable 'salinity' not found"): remote_inspect_variable(grid_file, data_file, "salinity") + + +class TestALostWorkerIsDiagnosedByWhatItWasRunning: + """``WorkerLost`` has two very different causes, and one message. + + Memory is the usual one, and the standing advice -- check the mesh size, + subset first -- is right for it. It is wrong for a YAC-backed remap, and + expensively so: subsetting cannot fix an MPI abort, so the caller spends + the afternoon measuring a mesh that was never the problem. + + Observed on Casper. ``method="conservative"`` killed the worker, while + the identical remap ran in 0.09s under ``check_remote_yac`` -- which + executes it in a worker-side *subprocess*. Same node, same YAC, same + method: the only difference is whether an ``MPI_Init`` abort can take + the worker down with it. + """ + + WORKER_LOST = ( + "Task failure due to loss of worker 0 on host crhtc52\n" + "***** One common cause of WorkerLost exceptions is Python version " + "mismatch *****" + ) + + @staticmethod + def _normalize(**kw): + from uxarray_mcp.remote.agent import _normalize_remote_error + + cfg = types.SimpleNamespace(endpoint_id="x", default_endpoint="ucar") + return str( + _normalize_remote_error( + RuntimeError(TestALostWorkerIsDiagnosedByWhatItWasRunning.WORKER_LOST), + cfg, + **kw, + ) + ) + + def test_an_ordinary_lost_worker_still_blames_memory(self): + """The common case must not regress into MPI advice.""" + msg = self._normalize() + assert "memory" in msg.lower() + assert "MPI" not in msg + + def test_a_yac_remap_is_told_about_mpi_instead(self): + msg = self._normalize(yac_backed=True) + assert "MPI" in msg + assert "check_remote_yac" in msg + + def test_the_mpi_message_does_not_send_anyone_to_subset(self): + """Subsetting is the memory remedy and cannot fix an abort.""" + msg = self._normalize(yac_backed=True) + assert "subset" not in msg.lower() + + def test_the_fallback_it_offers_is_qualified(self): + """idw/bilinear do run -- but they are not conservative. + + Offering them without saying so trades a loud failure for a quiet + wrong answer, which is the trade this server exists to refuse. + """ + msg = self._normalize(yac_backed=True) + assert "inverse_distance_weighted" in msg + assert "conserve" in msg.lower() + + +class TestTheYacRouteIsRecognisedFromPositionalArguments: + """Every call into ``_run_on_hpc`` passes its arguments positionally. + + The first version of this read only ``kwargs`` and so never fired: a + real conservative remap on Casper still printed the memory advice. The + detection has to look at the positional arguments too. + """ + + @staticmethod + def _looks(*args, **kwargs): + from uxarray_mcp.remote.agent import _looks_yac_backed + + return _looks_yac_backed(args, kwargs or None) + + def test_a_positional_method_is_seen(self): + assert self._looks("/g.nc", "/d.nc", "rrfac", "conservative") + + def test_a_positional_backend_is_seen(self): + assert self._looks("/g.nc", "/d.nc", "rrfac", "yac") + + def test_keywords_still_work(self): + assert self._looks(method="conservative") + assert self._looks(backend="yac") + + @pytest.mark.parametrize("method", ["nnn", "dnn", "average"]) + def test_the_other_yac_methods_count(self, method): + """All four route through YAC, not just the conservative one.""" + assert self._looks("/g.nc", method) + + def test_an_ordinary_remap_is_not_flagged(self): + assert not self._looks("/g.nc", "/d.nc", "rrfac", "nearest_neighbor") + assert not self._looks("/g.nc", "/d.nc", "rrfac", "bilinear") + + def test_a_path_that_merely_contains_the_word_is_not_flagged(self): + """`yac` appears in real cluster paths; only a whole argument counts.""" + assert not self._looks("/glade/u/home/x/uxarray-yac-src/grid.nc") + + +class TestTheDiagnosisSurvivesTheCallPath: + """The routing, not just the predicate. + + ``_looks_yac_backed`` can be perfectly correct and still never fire if + ``_run_on_hpc`` hands it the wrong thing. That is exactly what happened: + the first version passed only ``kwargs``, every caller passes its + arguments positionally, and a real conservative remap on Casper still + got the memory advice. Testing the predicate alone did not catch it. + """ + + async def test_a_conservative_remap_through_run_on_hpc_gets_mpi_advice( + self, monkeypatch + ): + from uxarray_mcp.remote import agent as agent_mod + + lost = RuntimeError("Task failure due to loss of worker 0 on host crhtc52") + + class _Future: + def result(self, *_a, **_k): + raise lost + + class _Executor: + def submit(self, *_a, **_k): + return _Future() + + cfg = types.SimpleNamespace( + endpoint_id="abc", default_endpoint="ucar", timeout_seconds=5 + ) + ag = UXarrayComputeAgent.__new__(UXarrayComputeAgent) + ag.config = cfg + monkeypatch.setattr( + agent_mod, "_record_task_handle_when_assigned", lambda *a, **k: None + ) + monkeypatch.setattr( + ag, "_get_executor", lambda *a, **k: _Executor(), raising=False + ) + + with pytest.raises(RuntimeError) as err: + # Positional, exactly as every real caller does it. + await ag._run_on_hpc( + lambda *a: None, "/g.nc", "/d.nc", "rrfac", "conservative" + ) + assert "MPI" in str(err.value), ( + "a positionally-passed conservative remap lost its diagnosis " + "somewhere between _run_on_hpc and the error handler" + ) diff --git a/uv.lock b/uv.lock index b585fb5..36102d0 100644 --- a/uv.lock +++ b/uv.lock @@ -1193,11 +1193,11 @@ wheels = [ [[package]] name = "filelock" -version = "4.1.1" +version = "4.2.4" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/35/c8/1d457d9150ff948f2ce6ada7715e0eeebbe5d3b58a45271a1e222474bcd3/filelock-4.1.1.tar.gz", hash = "sha256:7ba0927482c5a814b0a7f391d029ccdb8010f576f0a74c0dcde1811e8bc4c1b6", size = 563430, upload-time = "2026-10-11T16:11:54.373Z" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/28/3893324bcadf8f89b1616637468d766af3f625008c0735bf704fdfa86617/filelock-4.2.4.tar.gz", hash = "sha256:c6d8663535e1650ee430507b4ffbac7b464dd64b7647ea0112a9f406499d163d", size = 652444, upload-time = "2026-10-19T01:57:51.001Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/d7/8b/f837f52905395ba4510fe61f753c24833fb0a9c76e21267bb9f828b664a9/filelock-4.1.1-py3-none-any.whl", hash = "sha256:3f4a557945a7b0f95efeb1f432267affe5d45ac8ddde2aed1b97ebb62382c089", size = 132460, upload-time = "2026-10-11T16:11:52.753Z" }, + { url = "https://files.pythonhosted.org/packages/e0/1b/5e74289a1878e666bb4b65fe5805a04cafcac2aa5f519ef1ead67ef91c56/filelock-4.2.4-py3-none-any.whl", hash = "sha256:301241438c57936ef93f4243a94f2559022268bfcf9b375a9a72eb0f8db8358e", size = 162475, upload-time = "2026-10-19T01:57:49.419Z" }, ] [[package]] @@ -5137,7 +5137,7 @@ viz = [ [[package]] name = "uxarray-mcp" -version = "2026.9.1" +version = "2026.10.0" source = { editable = "." } dependencies = [ { name = "holoviews" }, From 8b782bb970b3fbc0c74b72acd1ef7bb893c365e3 Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Mon, 26 Oct 2026 19:02:16 -0500 Subject: [PATCH 2/6] Say what Globus is before asking anyone to use it Three new users got stuck in the same place, and it was not the hard part. The Globus explanation existed, at line 64 of remote-hpc.md, under a heading nobody reaches: README's "Pick your path" sends HPC users to that page's *middle*, past the identity step. So the prerequisite was written down and unreachable, which reads the same as missing. docs/globus-basics.md is the ten-minute version: which of the several Globus services you actually need, why the identity has to be the institutional one, and the three things people reasonably expect that are not true. It is 70 lines because a page nobody finishes helps nobody. The identity point is the one that costs the most time. A Globus login is not automatically tied to a cluster account; sign in with a personal Google account when the HPC login is you@institution.edu and you get an identity that authenticates perfectly and sees none of your endpoints -- indistinguishable, from the outside, from a dead endpoint. The fix is linking identities, not making a second account, and nothing said so. README now routes path 2 to connect-a-cluster.md rather than the long form, and names globus-basics.md for paths 2-4. docs/ucar.md gains two things. Endpoints are per-user: a colleague's working Casper endpoint gives you nothing, since it runs as their account and reads their files, and the conda and YAC paths in that page are per-$USER too. And the WorkerLost-on-conservative-remap entry, with how to tell MPI from memory -- check_remote_yac runs the same remap in a subprocess, so conservative_ok separates "YAC is broken" from "YAC cannot run in this process." --- README.md | 6 +++- docs/connect-a-cluster.md | 5 +++ docs/globus-basics.md | 74 +++++++++++++++++++++++++++++++++++++++ docs/index.rst | 1 + docs/ucar.md | 32 +++++++++++++++++ 5 files changed, 117 insertions(+), 1 deletion(-) create mode 100644 docs/globus-basics.md diff --git a/README.md b/README.md index 1e78adb..55e6cb6 100644 --- a/README.md +++ b/README.md @@ -36,11 +36,15 @@ You are most likely one of: 1. **Local user** — laptop only, no HPC. → [Local install](#local-install). 2. **HPC user, endpoint already exists** — someone at your lab gave you a Globus Compute endpoint UUID. → [Local install](#local-install), then - [docs/remote-hpc.md](docs/remote-hpc.md). + [docs/connect-a-cluster.md](docs/connect-a-cluster.md). 3. **HPC user, your own personal endpoint** — you have a Globus identity and shell access to an HPC machine, and want to stand up an endpoint just for yourself. → [Local install](#local-install), then [docs/operating-an-endpoint.md](docs/operating-an-endpoint.md#solo-personal-endpoint-quickstart). + +> **New to Globus?** Paths 2–4 all assume an identity linked to your HPC +> account. [docs/globus-basics.md](docs/globus-basics.md) is ten minutes and +> saves most of the common dead ends. 4. **Group / shared endpoint operator** — you're standing one up for a team, project, or lab. → [Local install](#local-install), then the full [docs/operating-an-endpoint.md](docs/operating-an-endpoint.md) including diff --git a/docs/connect-a-cluster.md b/docs/connect-a-cluster.md index 46b2e69..e597146 100644 --- a/docs/connect-a-cluster.md +++ b/docs/connect-a-cluster.md @@ -3,6 +3,11 @@ Your data is on an HPC machine. You want to ask an AI about it without downloading anything. This page gets you there. +> **Never used Globus?** Ten minutes on +> [globus-basics.md](globus-basics.md) first. It covers the identity, why it +> has to be your institutional one, and why Compute is not Transfer — the +> three things that account for most stuck setups. + It is machine-agnostic: the same five commands work for Chrysalis, Casper, Perlmutter or a university cluster nobody has written a page about. Site pages ([chrysalis](chrysalis.md), [ucar](ucar.md), [improv](improv.md)) diff --git a/docs/globus-basics.md b/docs/globus-basics.md new file mode 100644 index 0000000..2fae3f3 --- /dev/null +++ b/docs/globus-basics.md @@ -0,0 +1,74 @@ +# Globus, the short version + +Ten minutes, a browser, no cost. Read this once before connecting a cluster. + +## Which bit you need + +Globus is several services. You need Auth (login) and Compute (runs your +code). You almost certainly do not need Transfer. + +**Compute is not Transfer.** Different services, separate consents. A +working Compute endpoint cannot copy a file. If the word "consent" shows up +and you were not trying to move a file, you are in the wrong service. + +The point of this server is that the analysis goes to the data and a number +comes back, so moving files is a separate decision: +[data-transfer.md](data-transfer.md). + +## Get an identity + +Already used Globus for HPC file transfers? You have one. Skip ahead. + +Otherwise: go to and **sign in with your +institution** — Argonne, NCAR/UCAR, NERSC, most universities. Use the same +account that holds your HPC login. + +That last part matters. A Globus identity is not automatically tied to your +cluster account. Sign in with a personal Google account when your HPC login +is `you@institution.edu` and you get an identity that authenticates fine and +then sees none of your endpoints — which looks exactly like a dead endpoint. +Already done it? Link the institutional identity at + rather than starting over. + +## Check it + +```bash +uxarray-mcp doctor +uxarray-mcp endpoints discover +``` + +The first Globus command opens a browser. Finish the login and re-run. + +`discover` asks Globus what your identity can see, which also answers *where +do I get a UUID?* — it is printed once when an endpoint registers and is easy +to lose. Nobody has to send you one. + +| You see | Next | +|---|---| +| An endpoint listed | [connect-a-cluster.md](connect-a-cluster.md) | +| Nothing | [Create one](connect-a-cluster.md#create-20-minutes) | +| `ENDPOINT_NOT_ONLINE` | It is asleep — see below | +| A login loop | Wrong identity; see above | + +## Three surprises + +**An endpoint is per-user, not per-machine.** It runs as *your* cluster +account and reads what you can read. Your colleague's working endpoint tells +you nothing about whether you have one. + +**Registered is not working.** It means a manager process phoned home. It +does not mean a worker exists, has `uxarray`, or can read your file. Those +fail separately, which is why `doctor`, `validate_hpc_setup` and +`probe_path_access` check them in that order. + +**Endpoints sleep.** On a scheduler-backed endpoint the worker lives in a +batch allocation. When it ends, the endpoint stays registered with nothing to +run on. `ENDPOINT_NOT_ONLINE` usually means *ask again in a minute*. A slow +first call after an idle period is normal. + +## One caution + +An endpoint's operator can see the functions you submit and the paths you +name. Submit nothing you would not put in a ticket. +[SECURITY.md](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/SECURITY.md) +has the rest. diff --git a/docs/index.rst b/docs/index.rst index 781ec04..22a87c2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,6 +23,7 @@ HTTP clients) from a single install. :maxdepth: 2 :caption: Running on HPC (optional) + globus-basics connect-a-cluster remote-hpc data-transfer diff --git a/docs/ucar.md b/docs/ucar.md index d612c2e..3c9439c 100644 --- a/docs/ucar.md +++ b/docs/ucar.md @@ -32,6 +32,17 @@ That gives you every tool except conservative remapping. The rest of this page describes `scripts/ucar_endpoint.sh`, which is the same thing with NCAR's verified module list, an optional YAC build, and a fixed profile name. +## If someone else set this up + +Endpoints are per-user. A colleague's working Casper endpoint does not give +you one — it runs as *their* account and reads *their* files. You need your +own, and `uxarray-mcp endpoints discover` tells you whether you have it. + +Same for the conda env and the YAC build below: they live under a specific +`$USER`, so paths in this page like `/glade/work/$USER/conda-envs/uxarray_dev` +are yours, not shared. Set `NCAR_USERNAME` if your NCAR login differs from +your local `$USER`. + ## Key Points - **You do need a clone of this repo on Casper**, because the endpoint scripts @@ -178,3 +189,24 @@ synthesized from the prefix, so there is nothing to edit in the script. **`doctor` passes but YAC tools fail** — run `yac_smoke_test.py`, which checks importability on the compute worker specifically rather than on the login node. + +**A conservative remap kills the worker (`WorkerLost`)** — almost certainly +MPI, not memory, and the two look identical from the outside. Importing YAC +calls `MPI_Init`: Open MPI starts fine as a singleton, MPICH aborts the +process, and an abort inside a Globus worker arrives as a lost worker with +nothing mentioning MPI. + +Tell them apart with `check_remote_yac(endpoint=...)`. It runs the same +conservative remap in a *subprocess*, where an abort is survivable: + +- `conservative_ok: true` — YAC is fine. The worker needs to start inside an + MPI context (`GlobusMPIEngine`, or a launcher prefix on the engine). An + export in `worker_init` cannot do this; that runs before the worker exists. +- `conservative_ok: false` — the build is the problem, not the launcher. + +`./scripts/ucar_endpoint.sh check-mpi` runs the bare and launched imports side +by side and tells you which case you are in. + +Meanwhile `inverse_distance_weighted` and `bilinear` work — they use uxarray's +own engine. Neither conserves the field integral, which is usually the reason +conservative was asked for, so this is a stopgap and not a fix. From 429abf16790d4e33b22d13a485be7ebc0a978062 Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Mon, 26 Oct 2026 19:02:59 -0500 Subject: [PATCH 3/6] Record the MPI, floor and docs work in the changelog The 2026.10.0 entry was written before these landed. Adds the YAC/MPI diagnosis, the check_remote_yac gap that let a broken worker report itself healthy, the McpError/MCPError floor break, and the onboarding pages. Security count 49 -> 55 for the filelock advisories. --- CHANGELOG.md | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 838ed07..9f3dca1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -59,6 +59,29 @@ built against; see `docs/release.md`. Versions through `0.3.1` were SemVer. environment. ### Fixed +- A YAC-backed remap that loses its worker is told what killed it. The + message said memory and advised subsetting; on the UCAR endpoint the + cause is MPI, and subsetting cannot fix it. Importing YAC calls + `MPI_Init`, which Open MPI serves as a singleton and MPICH aborts, and + an abort inside a Globus worker arrives with nothing mentioning MPI. + Measured on Casper: the identical conservative remap runs in 0.09s + under `check_remote_yac` and kills the worker through `run_analysis` -- + same node, same YAC, same mesh, the only difference being that the + probe runs it in a subprocess. Any other lost worker still gets the + memory advice, which is right for it. +- `check_remote_yac` exercises the conservative path. It only ever ran + `nearest_neighbor`, which uses uxarray's own engine and never calls + into YAC, so it reported "YAC OK" on a worker where every conservative + remap died. `conservative_ok` now separates "YAC is broken" from "YAC + cannot run in this process", which is the distinction that picks the + repair. +- `scripts/ucar_endpoint.sh check-mpi` runs the bare and launched imports + side by side and names which case the site is in. Chrysalis has had an + equivalent since it hit this; UCAR had nothing. +- The MCP SDK exception is found under either spelling. `McpError` on + 1.x, `MCPError` on 2.x; importing one name fails on the other SDK, and + only the weekly floor job resolves 1.x, so the wrong one went unnoticed + until that run went red. - Tool annotations reach clients on mcp 1.x, not only 2.x. At the declared floor the shim attached nothing, so every tool was served with no hints at all -- the `readOnlyHint` a client reads to decide what it may @@ -95,13 +118,27 @@ built against; see `docs/release.md`. Versions through `0.3.1` were SemVer. `grid_corners` holds each write to ~191 MiB; the full suite now peaks at 1.36 GiB. +### Documentation +- `docs/globus-basics.md` says what Globus is before anything asks you to + use it. The explanation existed, in the middle of `remote-hpc.md`, + under a heading the README routes HPC users straight past -- written + down and unreachable, which reads the same as missing. It also names + the identity trap that costs the most time: a Globus login is not + automatically tied to a cluster account, and signing in with a personal + account produces an identity that authenticates fine and sees none of + your endpoints, which looks exactly like a dead endpoint. +- `docs/ucar.md` says endpoints are per-user. A colleague's working + Casper endpoint gives you nothing: it runs as their account and reads + their files. The conda and YAC paths on that page are per-`$USER` too. + ### Security -- Clears 49 advisories in the lockfile, all in transitively-resolved +- Clears 55 advisories in the lockfile, all in transitively-resolved packages: 44 against pillow 12.3.0 (5 Critical), PYSEC-2026-4320 against certifi 2026.7.22 (9.1 Critical), and four against virtualenv 21.14.7. certifi is the trust store behind every outbound HTTPS call this server makes -- Globus Compute submission, transfers, remote dataset reads -- so - a Critical there is not a transitive detail. + a Critical there is not a transitive detail. A further six against + filelock 4.1.1 cleared the same way. ## 2026.9.1 — 2026-09-25 ### Changed From c003a00ca6c8ca2ce14dbd4e2e7e3951c774be17 Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Mon, 26 Oct 2026 19:07:29 -0500 Subject: [PATCH 4/6] Drop the stray blank line ruff's import sort objected to --- src/uxarray_mcp/remote/agent.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/uxarray_mcp/remote/agent.py b/src/uxarray_mcp/remote/agent.py index 672e168..f21553a 100644 --- a/src/uxarray_mcp/remote/agent.py +++ b/src/uxarray_mcp/remote/agent.py @@ -39,7 +39,6 @@ def action(fn): # type: ignore[no-redef] ) from .config import HPCConfig - #: Remap methods that compute their weights inside YAC. #: #: Kept here rather than imported from ``domain.remap_backend`` because this From fb665bac5c49e05c2db9c2fb42c08042095b043b Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Mon, 26 Oct 2026 20:05:07 -0500 Subject: [PATCH 5/6] Take one person's username out of the test fixtures `grep jain` found nothing in src/, scripts/ or docs/ -- those all use $USER or $NCAR_USERNAME already. The tests were the exception. tests/test_hpc_safety.py asserted on a PYTHONPATH containing /lcrc/group/e3sm/jain/uxarray-yac-src. The rule under test keys on "yac" appearing in the path and has nothing to do with whose checkout it is, so a real name there reads like the behaviour depends on it. Now . tests/test_remote_remap_live.py printed three copy-pasteable paths under one person's group directory. A colleague running the live tests would copy them verbatim and get "file not found" on a cluster where the file exists -- under their own name. Hoisted to a $MESH variable with in it, so the thing to change is visible and typed once. The /scratch/rjain paths in the transfer tests address a fake service and never touch a filesystem, but a reader cannot tell that from the string. /scratch/demo. Authorship in pyproject.toml and the release workflows is left alone -- that is a name doing its job. --- tests/test_globus_transfer.py | 20 ++++++++++---------- tests/test_hpc_safety.py | 6 +++++- tests/test_remote_remap_live.py | 7 ++++--- tests/test_transfer_tools.py | 8 ++++---- 4 files changed, 23 insertions(+), 18 deletions(-) diff --git a/tests/test_globus_transfer.py b/tests/test_globus_transfer.py index 80a25b4..b2635c6 100644 --- a/tests/test_globus_transfer.py +++ b/tests/test_globus_transfer.py @@ -69,7 +69,7 @@ def _profile(**overrides) -> GlobusTransferProfile: base = { "remote_collection_id": "remote-uuid", "local_collection_id": "local-uuid", - "remote_write_root": "/scratch/rjain", + "remote_write_root": "/scratch/demo", "remote_read_root": "/lcrc/group/e3sm", "collection_roots": (), } @@ -189,7 +189,7 @@ def test_an_unmatched_path_passes_through_unchanged(self): # path rewritten under the wrong root names a real file nobody asked # about. assert ( - to_collection_path("/home/rjain/out.nc", ("/lcrc",)) == "/home/rjain/out.nc" + to_collection_path("/home/demo/out.nc", ("/lcrc",)) == "/home/demo/out.nc" ) def test_a_prefix_sibling_does_not_match_a_root(self): @@ -231,7 +231,7 @@ def test_an_upload_lands_under_the_write_root(self, tmp_path): source.write_text("x") service = TransferService(_profile(), FakeTransferClient()) plan = service.plan_put(source, "runs/mesh.nc") - assert plan.destination_path == "/scratch/rjain/runs/mesh.nc" + assert plan.destination_path == "/scratch/demo/runs/mesh.nc" assert plan.source_path == str(source.resolve()) assert plan.recursive is False @@ -264,14 +264,14 @@ def test_the_write_root_is_readable_too(self, tmp_path): # Somewhere you may put a file is somewhere you may look at one: a # download of what was just uploaded must not fail on the read root. service = TransferService(_profile(), FakeTransferClient()) - plan = service.plan_get("/scratch/rjain/out.nc", tmp_path / "out.nc") - assert plan.source_path == "/scratch/rjain/out.nc" + plan = service.plan_get("/scratch/demo/out.nc", tmp_path / "out.nc") + assert plan.source_path == "/scratch/demo/out.nc" def test_a_path_under_no_readable_root_names_them_all(self, tmp_path): service = TransferService(_profile(), FakeTransferClient()) with pytest.raises(PathOutsideRoot) as excinfo: - service.plan_get("/home/rjain/out.nc", tmp_path / "out.nc") - assert "/scratch/rjain" in str(excinfo.value) + service.plan_get("/home/demo/out.nc", tmp_path / "out.nc") + assert "/scratch/demo" in str(excinfo.value) assert "/lcrc/group/e3sm" in str(excinfo.value) def test_a_read_root_does_not_widen_writes(self, tmp_path): @@ -284,7 +284,7 @@ def test_a_read_root_does_not_widen_writes(self, tmp_path): def test_without_a_read_root_reads_fall_back_to_the_write_root(self, tmp_path): service = TransferService(_profile(remote_read_root=None), FakeTransferClient()) assert ( - service.plan_get("out.nc", tmp_path).source_path == "/scratch/rjain/out.nc" + service.plan_get("out.nc", tmp_path).source_path == "/scratch/demo/out.nc" ) with pytest.raises(PathOutsideRoot): service.plan_get("/lcrc/group/e3sm/out.nc", tmp_path) @@ -305,7 +305,7 @@ def test_collection_translation_applies_to_the_submitted_path(self, tmp_path): _profile(collection_roots=("/scratch",)), FakeTransferClient() ) plan = service.plan_put(source, "runs/mesh.nc") - assert plan.destination_path == "/rjain/runs/mesh.nc" + assert plan.destination_path == "/demo/runs/mesh.nc" class TestAnEndpointWithoutTheBlockMovesNoFiles: @@ -341,7 +341,7 @@ def test_a_submission_id_is_fetched_and_carried(self, tmp_path): assert payload["submission_id"] == "submission-1" assert payload["source_endpoint"] == "local-uuid" assert payload["destination_endpoint"] == "remote-uuid" - assert payload["DATA"][0]["destination_path"] == "/scratch/rjain/runs/mesh.nc" + assert payload["DATA"][0]["destination_path"] == "/scratch/demo/runs/mesh.nc" assert result["task_id"] == "task-1" def test_checksums_are_verified_and_mail_is_not_sent(self, tmp_path): diff --git a/tests/test_hpc_safety.py b/tests/test_hpc_safety.py index e83d511..5250565 100644 --- a/tests/test_hpc_safety.py +++ b/tests/test_hpc_safety.py @@ -108,10 +108,14 @@ def test_globus_offline_maps_to_offline(self): def test_yac_pythonpath_is_expected_runtime_path(self): """Endpoint-side YAC source/runtime paths are not a worker leak.""" + # The last entry is a group checkout, which on a real cluster sits + # under someone's username. Written with a placeholder because the + # check keys on "yac" appearing in the path, not on whose it is -- + # a real name here reads like the rule depends on it. pythonpath = ( "/home/testuser/src/yac/build/python:" "/home/testuser/local/yac-3.17/lib/python3.12/site-packages:" - "/lcrc/group/e3sm/jain/uxarray-yac-src" + "/lcrc/group/e3sm//uxarray-yac-src" ) assert health._is_expected_yac_pythonpath(pythonpath) is True diff --git a/tests/test_remote_remap_live.py b/tests/test_remote_remap_live.py index cd679e7..566859e 100644 --- a/tests/test_remote_remap_live.py +++ b/tests/test_remote_remap_live.py @@ -9,9 +9,10 @@ these are read by the worker. Verified against Chrysalis on 2026-10-13:: UXMCP_LIVE_ENDPOINT=chrysalis \\ - UXMCP_LIVE_GRID=/lcrc/group/e3sm/jain/uxarray-yac-src/test/meshfiles/mpas/QU/480/grid.nc \\ - UXMCP_LIVE_DATA=/lcrc/group/e3sm/jain/uxarray-yac-src/test/meshfiles/mpas/QU/480/data.nc \\ - UXMCP_LIVE_TARGET_GRID=/lcrc/group/e3sm/jain/uxarray-yac-src/test/meshfiles/mpas/dyamond-30km/gradient_grid_subset.nc \\ + MESH=/lcrc/group/e3sm//uxarray-yac-src/test/meshfiles \\ + UXMCP_LIVE_GRID=$MESH/mpas/QU/480/grid.nc \\ + UXMCP_LIVE_DATA=$MESH/mpas/QU/480/data.nc \\ + UXMCP_LIVE_TARGET_GRID=$MESH/mpas/dyamond-30km/gradient_grid_subset.nc \\ UXMCP_LIVE_VARIABLE=bottomDepth \\ uv run --no-sync pytest tests/test_remote_remap_live.py -m live -v diff --git a/tests/test_transfer_tools.py b/tests/test_transfer_tools.py index d494054..3dbff18 100644 --- a/tests/test_transfer_tools.py +++ b/tests/test_transfer_tools.py @@ -23,7 +23,7 @@ def _transfer_profile(**overrides) -> GlobusTransferProfile: base = { "remote_collection_id": "remote-uuid", "local_collection_id": "local-uuid", - "remote_write_root": "/scratch/rjain", + "remote_write_root": "/scratch/demo", "remote_read_root": "/lcrc/group/e3sm", } base.update(overrides) @@ -162,12 +162,12 @@ def test_a_path_outside_the_root_is_reported_not_raised(self, monkeypatch): def test_an_endpoint_without_a_block_says_what_to_add(self, monkeypatch): monkeypatch.setattr(transfer_tools, "_config", lambda: (_config(None), None)) - result = transfer_tools.transfer_ls("/scratch/rjain") + result = transfer_tools.transfer_ls("/scratch/demo") assert result["submitted"] is False assert "globus_transfer" in result["message"] def test_every_result_carries_provenance_and_an_operation_id(self, wired): - result = transfer_tools.transfer_ls("/scratch/rjain") + result = transfer_tools.transfer_ls("/scratch/demo") assert result["_provenance"]["operation_id"] assert result["_provenance"]["tool"] == "transfer_ls" @@ -239,7 +239,7 @@ def test_the_probe_lists_the_write_root(self, monkeypatch): class Client: def operation_ls(self, collection_id, **kwargs): - assert kwargs["path"] == "/scratch/rjain" + assert kwargs["path"] == "/scratch/demo" return {"DATA": [{"name": "runs", "type": "dir"}]} monkeypatch.setattr( From 08c6311b444240dd1fc132a80f1166cbe4a7e958 Mon Sep 17 00:00:00 2001 From: Rajeev Jain Date: Mon, 26 Oct 2026 20:41:01 -0500 Subject: [PATCH 6/6] Teach check-yac the difference between YAC and a remap scripts/chrysalis_endpoint.sh check-yac reported "remap_ok": true from a nearest_neighbor remap, which runs on uxarray's own engine and never calls into YAC. It would therefore pass on a worker where every conservative remap dies -- and on Chrysalis that is exactly the state: check-yac green, and `method="conservative"` through run_analysis kills the worker. Same gap check_remote_yac had. It now also runs the conservative remap and reports conservative_ok. Deliberately not part of the exit status: a site that only wants the non-YAC methods is not broken, so this is reported rather than enforced. Also trims the onboarding pages, which had grown to three descriptions of the same thing. globus-basics.md owns the identity and the Compute-is-not-Transfer point; connect-a-cluster.md owns the five-command path; remote-hpc.md is the long form for when one of those did not work. It was still opening with its own Globus-identity step, its own install-the-extra step, and a second copy of the three-layer diagram -- 107 lines saying what the other two now say. Its steps renumber 3-4-5 to 1-2-3, since the first two are gone. The trio is 549 lines against 569 for the two pages it replaces, and the Globus prerequisite is now reachable instead of buried at line 64 of the longest one. --- docs/connect-a-cluster.md | 12 +--- docs/remote-hpc.md | 122 +++++----------------------------- scripts/chrysalis_endpoint.sh | 22 ++++++ 3 files changed, 42 insertions(+), 114 deletions(-) diff --git a/docs/connect-a-cluster.md b/docs/connect-a-cluster.md index e597146..27040ff 100644 --- a/docs/connect-a-cluster.md +++ b/docs/connect-a-cluster.md @@ -44,15 +44,9 @@ Three things, in three places, each of which can be wrong independently: Diagnose in that order. Most "it doesn't work" is the first or the third, and the error usually names the second. -### Two things people reasonably expect, which are not true - -- **An endpoint is per-user, not per-machine.** It runs *as your HPC - account* and can read exactly what you can read. There is no shared - "Chrysalis endpoint" unless someone deliberately operates one for a group. -- **Compute is not Transfer.** Globus Compute runs code; Globus Transfer - moves files. Separate services, separate consents. A working endpoint - cannot copy a file. You usually do not need Transfer at all — see - [data-transfer.md](data-transfer.md). +Two surprises worth knowing before you start — endpoints are per-user, and +Compute is not Transfer. Both in +[globus-basics.md](globus-basics.md#three-surprises). --- diff --git a/docs/remote-hpc.md b/docs/remote-hpc.md index b91de0e..67e45d5 100644 --- a/docs/remote-hpc.md +++ b/docs/remote-hpc.md @@ -1,111 +1,23 @@ -# Remote HPC Setup — Connecting to an Existing Endpoint - -> **New here? Read [connect-a-cluster.md](connect-a-cluster.md) first.** It -> covers the same ground in five commands and explains the concepts, for any -> machine. This page is the long form: every step spelled out, with the -> reasoning and the failure modes. Come here when the short version is not -> enough, or when something has gone wrong. -> -> If your question is *"where do I get a UUID?"*, the answer is -> `uxarray-mcp endpoints discover`. - -This page is for users who **already have an HPC account** (Argonne, NCAR, -NERSC, a university cluster, etc.) and want their AI agent to analyze data -that lives on that machine. - -> **Prerequisites — check each one before starting:** -> 1. uxarray-mcp installed locally and working with your AI client. See the -> [README](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/README.md#local-install). -> 2. A user account on the HPC machine, with shell access. -> 3. The data you want to analyze is readable by your HPC account. -> 4. Either: someone has given you a **Globus Compute endpoint UUID** for -> that machine, **or** you plan to stand up your own — in which case -> stop and read [operating-an-endpoint.md](operating-an-endpoint.md) first, -> then come back here. -> 5. You have read [SECURITY.md](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/SECURITY.md). Connecting an endpoint -> means the operator can see what you submit. - -The total setup is **5 steps**, expect **15–30 minutes** the first time. +# Remote HPC — the long form ---- - -## The three pieces - -Before the steps, the mental model. Remote analysis has three things that -have to all work, in three different places: - -```text -┌────────────────────┐ ┌──────────────────────────┐ -│ YOUR LAPTOP │ │ HPC MACHINE │ -│ │ │ │ -│ AI client │ │ Globus Compute endpoint │ -│ │ stdio │ │ (runs as your HPC │ -│ uxarray-mcp ──────┼─Globus─┼──▶ user, executes │ -│ + globus- │ Compute│ Python sent from │ -│ compute SDK │ HTTPS │ your laptop) │ -│ + your Globus │ │ │ -│ identity │ │ uxarray + xarray in │ -│ │ │ the worker environment │ -└────────────────────┘ └──────────────────────────┘ -``` - -Most "it doesn't work" failures come from one of these three layers being -misconfigured. The steps below address each one in order. - -An endpoint reporting `registered` only means the lightweight manager process -is alive and reachable — it does **not** prove the worker environment has -`uxarray` installed, can reach the scheduler, or can read your file. Validate -in order: manager reachable → a tiny no-op function runs → the exact file path -is readable → only then run a real UXarray tool. `validate_hpc_setup` and -`probe_path_access` exist for exactly this staged check. - ---- - -## Step 1 — Get a Globus identity (one-time, 5 min) - -If you've ever used Globus File Transfer for HPC, you already have this. -Skip to Step 2. - -If not: +[globus-basics.md](globus-basics.md) explains Globus and the identity. +[connect-a-cluster.md](connect-a-cluster.md) gets you running in five +commands. **Come here when one of those did not work**, or when you want +the reasoning behind a step. -1. Go to . -2. Sign in with your **institutional identity** if your lab is listed - (Argonne, NCAR/UCAR, LBL/NERSC, Oak Ridge, university SSO, etc.). This - ensures your Globus identity is linked to the same account that has HPC - access. -3. If your institution isn't listed, create a free Globus ID. -4. Verify you can reach . You're done. - -> No fee. Globus is free for non-commercial research use. - ---- - -## Step 2 — Install the HPC extras locally (1 min) - -```bash -uv tool install --python 3.12 "uxarray-mcp[hpc]" -# or to upgrade an existing install: -# uv tool upgrade --python 3.12 --extra hpc uxarray-mcp -``` - -The `--python 3.12` flag matters: most HPC endpoints today (chrysalis, -Casper, Improv) run Python 3.12 conda stacks, and Globus Compute's serializer -is fragile across Python minor versions. Mismatch raises `WorkerLost` on -non-trivial payloads. Tracked upstream at -[globus/globus-compute#2139](https://github.com/globus/globus-compute/issues/2139). - -This adds the `globus-compute-sdk` to your local install. Verify: - -```bash -uxarray-mcp doctor -``` +Assumes you already have: a working local install, an HPC account with +shell access, data that account can read, and either an endpoint UUID or +a plan to stand one up ([operating-an-endpoint.md](operating-an-endpoint.md)). +Also assumes you have read +[SECURITY.md](https://github.com/UXARRAY/uxarray-mcp-server/blob/main/SECURITY.md) +— connecting to an endpoint means its operator sees what you submit. -The JSON report should show the Globus Compute SDK as available. (No endpoint -health checks yet — that's Step 4.) +Getting a Globus identity and installing the HPC extra are covered in +those pages. This one picks up from the UUID. --- -## Step 3 — Register the endpoint UUID locally (1 min) +## 1. Register the endpoint UUID locally You need a name to refer to the endpoint by, plus the UUID and the filesystem prefix where the data lives. @@ -167,13 +79,13 @@ the repo). --- -## Step 4 — Authenticate to Globus and probe the endpoint (2 min) +## 2. Authenticate and probe the endpoint ```bash uxarray-mcp doctor --endpoint NAME --timeout-seconds 180 ``` -Replace `NAME` with what you used in Step 3 (`ucar`, `improv`, etc.). +Replace `NAME` with the name you registered above (`ucar`, `improv`, etc.). The first run will open a browser to for OAuth consent. Approve it. The token is cached at `~/.globus_compute/`. @@ -196,7 +108,7 @@ Ask the operator. See troubleshooting below. --- -## Step 5 — Use it from your AI client (2 min) +## 3. Use it from your AI client In Claude or your MCP client, prompt: diff --git a/scripts/chrysalis_endpoint.sh b/scripts/chrysalis_endpoint.sh index cfc4224..920a678 100755 --- a/scripts/chrysalis_endpoint.sh +++ b/scripts/chrysalis_endpoint.sh @@ -410,7 +410,29 @@ except Exception as exc: out["remap_error"] = f"{type(exc).__name__}: {exc}" out["remap_traceback"] = traceback.format_exc() +# The conservative remap, which is the only thing YAC is here for. +# +# `remap_ok` above is nearest_neighbor, which runs on uxarray's own engine +# and never calls into YAC -- so it reports success on a worker where every +# conservative remap dies. Importing YAC and computing weights with it are +# different capabilities and only the second one is what a caller asking for +# a conservative remap needs. +try: + t0 = time.perf_counter() + conservative = uxda.remap(destination_grid=dst, method="conservative", backend="yac") + out["conservative_ok"] = True + out["conservative_seconds"] = round(time.perf_counter() - t0, 3) + out["conservative_finite"] = int(np.isfinite(np.asarray(conservative)).sum()) +except BaseException as exc: + # BaseException: a YAC/MPI failure can raise SystemExit rather than a + # normal error, and catching only Exception would let this exit 0 with + # no verdict recorded. + out["conservative_ok"] = False + out["conservative_error"] = f"{type(exc).__name__}: {exc}"[:400] + print(json.dumps(out, indent=2)) +# `conservative_ok` is deliberately not part of the exit status: a site that +# only wants the non-YAC methods is not broken. It is reported, not enforced. raise SystemExit( 0 if out.get("yac_core_ok")