Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,17 @@ more than one license.
| `dev` | `pytest>=8.3` | `9.1.1` | `MIT` | [pytest](https://github.com/pytest-dev/pytest) | `uv / PyPI` |
| `dev` | `ruff>=0.15.12` | `0.15.20` | `MIT` | [Ruff](https://github.com/astral-sh/ruff) | `uv / PyPI` |

## Optional distributed runtime inventory

XPOIS distributed execution requires the runtime selected by the caller.
These runtimes are installed separately from cuPhoton and are not included
in its wheel or dependency lock. See the
[XPOIS guide](docs/components/xpois.md) for installation and launch details.

| Runtime | License and upstream notices | Use and installation |
| --- | --- | --- |
| DragonHPC (`dragonhpc`; import `dragon`) | [MIT](https://github.com/DragonHPC/dragon/blob/0.14.2/LICENSE) | Required for Dragon workers, placement and communication. Install separately in cuPhoton's Python environment; no Dragon source or binaries are bundled. |

## Native system dependency inventory

| Package | Version or version range | License identifier | Upstream | Use in cuPhoton | Distribution |
Expand Down
104 changes: 104 additions & 0 deletions docs/components/xpois.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,107 @@ represent neighboring-pixel covariance, reference-target covariance,
resampling covariance, or fitted-kernel uncertainty. The standardized residual
is a descriptive diagnostic, not a whitened residual or calibrated
significance image. Prefer held-out pixels when assessing fit quality.

## Dragon image-pair batches

`fit-batch-dragon` distributes complete reference/target image-pair fits across
GPU workers. The coordinator reads a manifest, assigns one Dragon
ProcessGroup worker to each selected GPU, and balances work by input size.
Arrays and output files stay on shared storage; workers send compact result
records through Dragon queues. Each image-pair fit runs on one GPU.

### Runtime dependency

The Dragon executor requires [DragonHPC](https://dragonhpc.github.io/dragon/doc/_build/html/index.html)
(Python distribution `dragonhpc`, import `dragon`) in the same Python
environment as cuPhoton on every participating node. Installing cuPhoton,
including its `gpu` extra, does not install DragonHPC. DragonHPC is installed
separately and is not included in cuPhoton's dependency lock or wheel.

For example, install the released DragonHPC 0.14.2 package into a CUDA 13
cuPhoton environment:

```bash
uv sync --locked --python 3.12 --extra gpu
uv pip install --python .venv/bin/python "dragonhpc==0.14.2"
```

The [DragonHPC 0.14.2 wheels](https://pypi.org/project/dragonhpc/0.14.2/#files)
support CPython 3.11 through 3.13 on Linux x86-64 and AArch64 with glibc 2.28
or newer. Use one of those Python versions for this installation; a CPython
3.14 wheel is not published for this DragonHPC release.

Use the installed `.venv/bin/dragon` launcher with the examples below.
`uv sync` removes packages outside the project lock, so repeat the DragonHPC
installation after resynchronizing the environment. Use the same cuPhoton
environment and DragonHPC version on every node. Each run records the
DragonHPC version it discovers. See the
[runtime notices](../../THIRD_PARTY_NOTICES.md#optional-distributed-runtime-inventory)
for licensing and installation details.

### Manifest and launch

Inputs and output directories must be accessible at the same paths on every
node. The command accepts a strict JSON or YAML manifest:

```yaml
schema: cuphoton.xpois.image-pairs/v1
pairs:
- id: detector-0001
reference: /shared/input/reference-0001.fits
target: /shared/input/target-0001.fits
reference_hdu: 1
target_hdu: 1
variance: /shared/input/variance-0001.fits
variance_hdu: 1
```

Loading the manifest records each unique input's resolved path, byte size,
and nanosecond modification time. These values are checked after coordinator
preflight and before and after each item. A change that preserves both size
and modification time is not detected; use immutable input data or external
checksums when content identity matters.

Launch through Dragon within an existing scheduler allocation:

```bash
.venv/bin/dragon examples/xpois/dragon_batch.py \
--manifest /shared/manifests/image-pairs.yaml \
--output-dir /shared/results/xpois-dragon \
--name image-pairs-gpu4 \
--max-workers 4 \
--worker-timeout-sec 3600 \
--backend cupy
```

The wrapper invokes `cuphoton xpois fit-batch-dragon`. This command requires
an explicit GPU backend: `cupy`, `numba-cuda`, or `cutile`. It rejects `auto`
instead of falling back to CPU when a GPU package is unavailable.

The coordinator enumerates actual `Node.gpus` IDs and selects workers
round-robin across hosts. Each worker checks its host and singleton
`CUDA_VISIBLE_DEVICES` assignment before importing the numerical backend.
Loopback hostname aliases are accepted only for a single-node allocation.

### Results and limits

Every attempt uses a new, immutable run directory. Each item has an atomic
terminal record under `records/`; an ordinary item error does not prevent
the remaining items in its shard from running. The final `summary.json`
checks for missing, duplicate, unexpected, malformed, or assignment-inconsistent
item and worker results, as well as nonzero worker exits. A worker that could
not write a terminal record still fails the run, but its declared errors are
reported under `shard_result_audit.write_failed_shards` rather than as an
evidence mismatch. Worker wall time
defaults to one hour. The coordinator attempts bounded stop and close cleanup
after failed starts or joins.

`coordinator_wall_sec` includes setup, worker cleanup, and terminal-result
checks. Writing the final summary falls outside that interval. Per-item
`timings_sec` separates input reads, preprocessing, solving, artifact writes,
review work, and other postprocessing; `wall_sec` retains the enclosing
workflow and item-runner measurements.

The executor does not retry failed items, recover dead workers, or split one
image-pair solve across GPUs. Inspect the terminal status before consuming a
run's outputs, and start a new run after a failed attempt.
16 changes: 16 additions & 0 deletions examples/xpois/dragon_batch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

"""Run XPOIS image-pair batches with Dragon workers."""

from __future__ import annotations

import sys

from cuphoton.core.cli import run_component

if __name__ == "__main__":
raise SystemExit(
run_component("xpois", ["fit-batch-dragon", *sys.argv[1:]])
)
Loading
Loading