From 70399fc3f6a7b29ce8806cb51b9654aba36e13e4 Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Wed, 23 Sep 2026 15:20:05 -0700 Subject: [PATCH 1/5] Document Dragon transport and queue placement Explain native HSTA TCP launches and consumer-local command queues. Separate control latency from pipeline throughput and describe the static shard executor already available in cuPhoton. Signed-off-by: Trent Nelson --- docs/README.md | 2 + docs/components/xpois.md | 4 ++ docs/dragon.md | 146 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 152 insertions(+) create mode 100644 docs/dragon.md diff --git a/docs/README.md b/docs/README.md index 7a65283b..5698bcfa 100644 --- a/docs/README.md +++ b/docs/README.md @@ -38,6 +38,8 @@ intend to adapt. ## Adapting and validating - [Architecture](architecture.md): package boundaries and execution model. +- [Dragon transport and coordination](dragon.md): transport selection, queue + placement, and timing comparisons. - [Adapting the workflows](adapting-workflows.md): a practical extension process. - [Data and artifact contracts](data-artifacts.md): shapes, files, metadata, diff --git a/docs/components/xpois.md b/docs/components/xpois.md index 8f132cd0..a3aba7c8 100644 --- a/docs/components/xpois.md +++ b/docs/components/xpois.md @@ -716,6 +716,10 @@ selection remains a launcher option; `--executor dragon` cannot change the transport of an already running Dragon session. The worker wall-time limit covers the entire invocation, including warmups and all measured rounds. +See [Dragon transport and coordination](../dragon.md) for native HSTA TCP +launch options, queue placement in custom coordinators, and measurement +boundaries. + ### Results and limits Both routes print a compact result containing the executor, run ID, run diff --git a/docs/dragon.md b/docs/dragon.md new file mode 100644 index 00000000..fba2814b --- /dev/null +++ b/docs/dragon.md @@ -0,0 +1,146 @@ +# Dragon transport and coordination + +Dragon places and manages cuPhoton workers; the selected numerical backend +runs each image pair inside its worker. Transport selection belongs to the +Dragon launcher. It does not change cuPhoton's numerical backend or fit options. + +The [XPOIS batch executor](components/xpois.md#launch-with-dragon) assigns a +complete shard to each worker at launch. Workers process their assigned items +and send one terminal result to the coordinator. This avoids a command exchange +for every image pair. + +## Select the application and overlay transports + +Dragon has separate application and infrastructure overlay transports. +`-t tcp -o tcp` selects its Python TCP transport for both. With a Dragon 0.14.2 +installation that includes native HSTA, select native HSTA for application +traffic and retain Python TCP for the overlay: + +```bash +DRAGON_HSTA_FORCE_BACKEND=tcp \ + .venv/bin/dragon -m -N 2 -w slurm -t hsta -o tcp \ + examples/xpois/dragon_batch.py \ + --backend cupy \ + --manifest /shared/manifests/fixed-32.yaml \ + --output-dir /shared/results/xpois-hsta \ + --name fixed-32-hsta16 \ + --max-workers 16 \ + --worker-timeout-sec 3600 +``` + +As in the XPOIS launch example, these paths must be accessible to the allocated +nodes. The manifest needs enough image pairs for the selected workers. Adapt +the node and worker counts to the allocation. + +`DRAGON_HSTA_FORCE_BACKEND=tcp` selects HSTA's native TCP implementation. This +configuration uses TCP rather than UCX or RDMA. Inspect the launcher and +transport logs to confirm HSTA started on each node and exited successfully; +record the Dragon version, arguments and environment with the run. INFO logs +can confirm native HSTA startup without providing network byte counters. + +HSTA TCP is a useful candidate for workloads with frequent small messages. +Qualify it on the target installation with representative numerical work, +output validation and cleanup checks. The Python TCP launch in the XPOIS guide +remains an explicit alternative. Transport availability and the fastest queue +layout depend on the Dragon build and workload. + +## Place queues for a custom command loop + +A custom coordinator that repeatedly releases work can have a different +communication pattern from cuPhoton's static shard executor. Suppose each worker +sends READY and then waits on its own command queue, while the coordinator waits +for all READY messages before sending commands. + +In Dragon 0.14.2's Python TCP transport, remote receives, sends and polls share +an executor. A receive waiting on an empty remote queue can occupy a thread +until a message arrives or its timeout expires. If command queues are all on +the coordinator's node, enough idle remote workers can consume the threads +needed to deliver the remaining READY messages. Both sides then wait. + +For this pattern with Python TCP, place each command queue on its consumer's +host. Keep the result queue on its consumer, the coordinator: + +```python +from dragon.infrastructure.policy import Policy +from dragon.native.machine import Node +from dragon.native.queue import Queue + +# worker_node_id comes from the allocation's discovered Dragon node IDs. +worker_host = Node(worker_node_id).hostname +worker_policy = Policy( + placement=Policy.Placement.HOST_NAME, + host_name=worker_host, +) +command_queue = Queue(maxsize=1, policy=worker_policy) +``` + +Use the hostname reported by Dragon's node discovery for both the queue and +worker placement. This makes the worker's idle receive local and the +coordinator's command send remote. It removes the central population of idle +receive waits while retaining command traffic and result collection. + +`DRAGON_TRANSPORT_TCP_MAX_THREADS` can increase the Python TCP executor ceiling. +It allowed the centralized layout to make progress in testing, but preserved +its idle remote receives and higher control latency. Native HSTA has a different +progress implementation; the Python TCP thread setting still applies to a +Python TCP overlay. Consumer-local queues are not universally faster under HSTA. + +The [existing XPOIS executor](../src/cuphoton/xpois/dragon.py) has one result +queue and passes each shard directly to its worker. It does not need this +command-queue change. For new long-lived services, send useful chunks of work +between coordination points and keep intermediate arrays inside their GPU +worker. If central fan-out remains expensive, a node coordinator can dispatch +to local GPU workers and aggregate their completions while preserving all item +identities and failures. + +## Evidence and timing boundaries + +A Dragon 0.14.2 control experiment used two nodes, with the coordinator on one +and 64 CPU worker processes on the other. Each worker returned a 1024-byte +payload per round; no GPU work or timed receipt-file writes occurred. Each +successful invocation had 12 rounds. Two invocations per configuration reversed +the queue-layout order. The table gives the median of rounds 2–12 for each +invocation; the first round was retained separately. + +| Application transport | Command queue host | Python TCP ceiling | Later-round medians | +|---|---|---:|---:| +| Python TCP | Coordinator | 1024 | 47.235 / 45.307 ms | +| Python TCP | Consumer | 1024 | 33.604 / 28.123 ms | +| Native HSTA TCP | Coordinator | 32 for overlay | 2.371 / 2.545 ms | +| Native HSTA TCP | Consumer | 32 for overlay | 3.027 / 3.093 ms | + +A separate invocation with coordinator queues and a 32-thread Python TCP +ceiling stalled after 32 of 64 READY messages. Consumer queues completed all +12 rounds at the same ceiling. HSTA's first rounds took 5.6–6.3 ms, compared +with the 2.4–3.1 ms later medians above. These are whole control-round timings, +not network-only measurements or predicted application speedups. + +A separate persistent-worker imaging harness compared MPI and both queue +layouts under Python TCP and HSTA TCP on eight GB200 GPUs. It processed +16 image-pair occurrences per round from two base pairs, for three rounds per +configuration. All 240 measured scientific outputs agreed. Batch medians +ranged from 7.515 to 7.719 seconds: the large control-only improvement did not +produce a comparable whole-pipeline gain. These results do not qualify a +512-GPU speedup or establish the best layout at larger node counts. + +For a new comparison, hold the corpus, item count, numerical configuration, +worker CPU budget and runtime versions constant. Rotate run order and retain +the first command round as well as subsequent rounds. Record separately: + +- Launcher-to-exit wall time, including startup, warmup and shutdown. +- Time from worker entry to readiness. +- Whole-batch time from release through collection. +- Worker execution time, with the treatment of output and receipt writes. + +Subtracting the longest worker duration from the whole-batch duration combines +start skew, synchronization, receipt writes, serialization and collection. It +cannot isolate network time. Validate complete output identities and scientific +results, worker exits and runtime cleanup alongside timings. The control probe +also observed an external Dragon launcher exit of zero after an application +failure; inspect the application's terminal result as well as the launcher. + +Free-threaded Python does not free an executor thread blocked in a native +receive: that receive already releases the GIL. A separate serialization or +dispatch profile is needed to establish whether Python execution is a remaining +bottleneck. The experiments above used conventional CPython 3.12; they do not +qualify a free-threaded Dragon runtime. From ec6016deb2b2d341d2adb167e99be7c6674e038b Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Wed, 23 Sep 2026 15:21:52 -0700 Subject: [PATCH 2/5] Clarify the readiness timing boundary Signed-off-by: Trent Nelson --- docs/dragon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dragon.md b/docs/dragon.md index fba2814b..199486d7 100644 --- a/docs/dragon.md +++ b/docs/dragon.md @@ -128,7 +128,7 @@ worker CPU budget and runtime versions constant. Rotate run order and retain the first command round as well as subsequent rounds. Record separately: - Launcher-to-exit wall time, including startup, warmup and shutdown. -- Time from worker entry to readiness. +- Time from harness entry until all workers are ready. - Whole-batch time from release through collection. - Worker execution time, with the treatment of output and receipt writes. From bf787b0848472bfecca584ee006a3ff1ec75a33f Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Thu, 24 Sep 2026 15:53:46 -0700 Subject: [PATCH 3/5] Refresh Dragon execution and coordination guidance Distinguish single-pass execution from the persistent executor stack. Document current timing and lifecycle boundaries and the 256-GPU follow-up without extending its qualification to newer product code. Signed-off-by: Trent Nelson --- docs/README.md | 4 +- docs/components/xpois.md | 4 +- docs/dragon.md | 123 +++++++++++++++++++++++++++++++++------ 3 files changed, 109 insertions(+), 22 deletions(-) diff --git a/docs/README.md b/docs/README.md index 5698bcfa..b17bc4c7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -38,8 +38,8 @@ intend to adapt. ## Adapting and validating - [Architecture](architecture.md): package boundaries and execution model. -- [Dragon transport and coordination](dragon.md): transport selection, queue - placement, and timing comparisons. +- [Dragon transport and coordination](dragon.md): execution models, + consumer-local queues, transport selection, and timing boundaries. - [Adapting the workflows](adapting-workflows.md): a practical extension process. - [Data and artifact contracts](data-artifacts.md): shapes, files, metadata, diff --git a/docs/components/xpois.md b/docs/components/xpois.md index a3aba7c8..0f06060e 100644 --- a/docs/components/xpois.md +++ b/docs/components/xpois.md @@ -717,8 +717,8 @@ transport of an already running Dragon session. The worker wall-time limit covers the entire invocation, including warmups and all measured rounds. See [Dragon transport and coordination](../dragon.md) for native HSTA TCP -launch options, queue placement in custom coordinators, and measurement -boundaries. +launch options, consumer-local queues in the pending persistent executors, +and measurement boundaries. ### Results and limits diff --git a/docs/dragon.md b/docs/dragon.md index 199486d7..80df2266 100644 --- a/docs/dragon.md +++ b/docs/dragon.md @@ -4,10 +4,32 @@ Dragon places and manages cuPhoton workers; the selected numerical backend runs each image pair inside its worker. Transport selection belongs to the Dragon launcher. It does not change cuPhoton's numerical backend or fit options. -The [XPOIS batch executor](components/xpois.md#launch-with-dragon) assigns a -complete shard to each worker at launch. Workers process their assigned items -and send one terminal result to the coordinator. This avoids a command exchange -for every image pair. +## Execution models + +The [XPOIS batch executor](components/xpois.md#launch-with-dragon) on `main` +assigns a complete shard to each worker at launch. Workers process their +assigned items and send one terminal result to the coordinator. The following +extensions are under review; their commands require the corresponding PRs: + +| Path | Coordination and retained state | +|---|---| +| Single-pass XPOIS | One assigned shard and one terminal result per worker; no per-round command queue. | +| [Repeated XPOIS rounds (#50)](https://github.com/NVIDIA/cuPhoton/pull/50) | `--warmup-rounds` and `--measure-rounds` reuse workers and CUDA contexts. Each worker has a command queue on its own host and repeats its assigned shard. | +| [Shared executors (#54)](https://github.com/NVIDIA/cuPhoton/pull/54) and [component commands (#56)](https://github.com/NVIDIA/cuPhoton/pull/56) | xFit, XScan and `xscan run-pipeline` use the shared Dragon/MPI lifecycle. Dragon uses consumer-local command queues for both a single pass and repeated rounds; component workers retain their input/model context. | + +Persistence does not remove all per-item work. XPOIS still reads, transfers and +writes ordinary scientific artifacts each round. Standalone xFit retains its +loaded host input; XScan retains metadata, its model and loader across tasks. +Distributed XScan defaults to `--num-workers 0`. Explicitly requested loader +processes start before READY and persist across tasks and rounds. The combined +pipeline keeps intermediate arrays inside each GPU worker while it +processes an image pair. Each GPU handles complete items; these paths do not +split one image across GPUs. + +Dragon workers load immutable, hash-checked launch descriptors from the shared +filesystem. Launch arguments carry descriptor references rather than the full +manifest, with a 96 KiB size guard. This bounds process-launch payloads; queue +placement addresses a separate coordination cost. ## Select the application and overlay transports @@ -44,12 +66,13 @@ output validation and cleanup checks. The Python TCP launch in the XPOIS guide remains an explicit alternative. Transport availability and the fastest queue layout depend on the Dragon build and workload. -## Place queues for a custom command loop +## Place command queues with their consumers -A custom coordinator that repeatedly releases work can have a different -communication pattern from cuPhoton's static shard executor. Suppose each worker -sends READY and then waits on its own command queue, while the coordinator waits -for all READY messages before sending commands. +The repeated XPOIS and shared Dragon executors use a READY/command/result +protocol. Each worker sends READY after initialization, then waits on its own +command queue. The coordinator waits for all READY messages before releasing +a round. Custom coordinators with the same protocol need to consider queue +placement. In Dragon 0.14.2's Python TCP transport, remote receives, sends and polls share an executor. A receive waiting on an empty remote queue can occupy a thread @@ -85,13 +108,17 @@ its idle remote receives and higher control latency. Native HSTA has a different progress implementation; the Python TCP thread setting still applies to a Python TCP overlay. Consumer-local queues are not universally faster under HSTA. -The [existing XPOIS executor](../src/cuphoton/xpois/dragon.py) has one result -queue and passes each shard directly to its worker. It does not need this -command-queue change. For new long-lived services, send useful chunks of work -between coordination points and keep intermediate arrays inside their GPU -worker. If central fan-out remains expensive, a node coordinator can dispatch -to local GPU workers and aggregate their completions while preserving all item -identities and failures. +The pending repeated-round and shared executors already use consumer-local +command queues. The [single-pass XPOIS path](../src/cuphoton/xpois/dragon.py) +uses only a result queue and needs no command-queue relocation. Neither path +automatically changes the transport or raises the Python TCP thread ceiling; +those remain launcher settings. + +The shared Dragon executor publishes terminal success only after worker +shutdown and lifecycle audits. Per-round success does not establish a +successful complete run. Silent worker exits and shutdown failures fail the +run. After a reported round failure, healthy peers can finish and retain their +artifacts. Check the root terminal summary before consuming a benchmark result. ## Evidence and timing boundaries @@ -121,7 +148,46 @@ layouts under Python TCP and HSTA TCP on eight GB200 GPUs. It processed configuration. All 240 measured scientific outputs agreed. Batch medians ranged from 7.515 to 7.719 seconds: the large control-only improvement did not produce a comparable whole-pipeline gain. These results do not qualify a -512-GPU speedup or establish the best layout at larger node counts. +512-GPU speedup or establish the best layout at larger node counts. This was +a separate harness experiment, with one invocation per configuration. + +Later two-node/eight-GPU product checks covered XPOIS, standalone xFit and +XScan, and the combined pipeline under Dragon and MPI. They established +numerical parity, persistent identities and cleanup for the tested revisions. +They predate the latest lifecycle, loader and input-ownership fixes in +#50/#54/#56; those updated revisions still need distributed GPU requalification. + +### 256-GPU follow-up + +A later instrumented harness compared MPI and Dragon on 256 GPUs across +64 nodes. Fixed batches contained 512 image-pair occurrences (two per GPU); +weak batches contained 4,096 (16 per GPU). All Dragon treatments used the +1024-thread Python TCP ceiling for the applicable application or overlay +transport. The table shows the range of +batch duration minus longest worker duration across all three rounds of each +accepted Dragon invocation, including the first command round: + +| Dragon treatment | Fixed batch remainder | Weak batch remainder | +|---|---:|---:| +| Python TCP, coordinator queues | 209.590–357.407 ms | 361.694–462.434 ms | +| Python TCP, consumer queues | 24.325–32.729 ms | 39.762–45.234 ms | +| Native HSTA TCP, coordinator queues | 8.126–8.629 ms | 7.863–9.451 ms | + +The fixed-batch consumer results combine two accepted launches; each other +table cell comes from one. The complete MPI/Dragon campaign retained 10 accepted +launches, with 30 measured rounds and 58,368 validated image-pair outputs, +alongside two startup failures and four unrun launches. Consumer-local queues +and HSTA reduced the observed coordination remainder, which also includes +dispatch, serialization and collection. It is not a network-only timer. + +The coordinator-queue control also lacked the historical multi-second spike. +Allocation, scheduler segmentation and logging differed from the earlier +campaign, so the spike's cause remains unresolved. There was no 512-GPU +retest. These measurements exercised the retained harness, not the latest +product PR revisions; they do not establish large-scale performance or launch +reliability for those revisions. + +### Compare timing boundaries For a new comparison, hold the corpus, item count, numerical configuration, worker CPU budget and runtime versions constant. Rotate run order and retain @@ -129,8 +195,23 @@ the first command round as well as subsequent rounds. Record separately: - Launcher-to-exit wall time, including startup, warmup and shutdown. - Time from harness entry until all workers are ready. -- Whole-batch time from release through collection. +- Whole-batch time from release through collection of worker completions. - Worker execution time, with the treatment of output and receipt writes. +- Coordinator artifact audits, scientific finalization and shutdown. + +In persistent-round reports, `batch_wall_sec` stops at completion collection; +coordinator artifact audits and scientific finalization follow it. The shared +executor records component merging separately as `finalization_sec`. Keep +readiness and shutdown outside this batch interval, and measure external +launcher-to-exit time independently. MPI finalization and process exit are +outside the executor's reported coordinator time. + +Requested warmup rounds exercise real dispatch, work and collection. Their +outputs and timings remain in the report. Only measured rounds enter timing +statistics; failed warmup rounds, measured rounds or cleanup invalidate the +aggregate. A numerical warmup performed before READY in a separate harness +does not necessarily warm the first command round. Match these policies before +comparing results. Subtracting the longest worker duration from the whole-batch duration combines start skew, synchronization, receipt writes, serialization and collection. It @@ -139,6 +220,12 @@ results, worker exits and runtime cleanup alongside timings. The control probe also observed an external Dragon launcher exit of zero after an application failure; inspect the application's terminal result as well as the launcher. +[The pipeline/stage benchmark (#55)](https://github.com/NVIDIA/cuPhoton/pull/55) +compares a resident device pipeline with fresh processes and intermediate +files on one GPU. It measures workflow reuse and process/file costs, including +additional intermediate outputs; it does not compare Dragon transports or +reproduce the stock component CLI chain. + Free-threaded Python does not free an executor thread blocked in a native receive: that receive already releases the GIL. A separate serialization or dispatch profile is needed to establish whether Python execution is a remaining From 507877a69dd34303e25b5be31af90e750c7b4ed3 Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Thu, 24 Sep 2026 17:37:05 -0700 Subject: [PATCH 4/5] Mark pending Dragon executor behavior in the guide Signed-off-by: Trent Nelson --- docs/dragon.md | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/docs/dragon.md b/docs/dragon.md index 80df2266..2b019c96 100644 --- a/docs/dragon.md +++ b/docs/dragon.md @@ -17,6 +17,9 @@ extensions are under review; their commands require the corresponding PRs: | [Repeated XPOIS rounds (#50)](https://github.com/NVIDIA/cuPhoton/pull/50) | `--warmup-rounds` and `--measure-rounds` reuse workers and CUDA contexts. Each worker has a command queue on its own host and repeats its assigned shard. | | [Shared executors (#54)](https://github.com/NVIDIA/cuPhoton/pull/54) and [component commands (#56)](https://github.com/NVIDIA/cuPhoton/pull/56) | xFit, XScan and `xscan run-pipeline` use the shared Dragon/MPI lifecycle. Dragon uses consumer-local command queues for both a single pass and repeated rounds; component workers retain their input/model context. | +### Pending persistent behavior (#50, #54 and #56) + +The following behavior requires the open PRs linked above. Persistence does not remove all per-item work. XPOIS still reads, transfers and writes ordinary scientific artifacts each round. Standalone xFit retains its loaded host input; XScan retains metadata, its model and loader across tasks. @@ -26,6 +29,8 @@ pipeline keeps intermediate arrays inside each GPU worker while it processes an image pair. Each GPU handles complete items; these paths do not split one image across GPUs. +### Launch descriptors on main + Dragon workers load immutable, hash-checked launch descriptors from the shared filesystem. Launch arguments carry descriptor references rather than the full manifest, with a 96 KiB size guard. This bounds process-launch payloads; queue @@ -66,8 +71,11 @@ output validation and cleanup checks. The Python TCP launch in the XPOIS guide remains an explicit alternative. Transport availability and the fastest queue layout depend on the Dragon build and workload. -## Place command queues with their consumers +## Pending command-queue protocol (#50 and #54) +This section describes the implementations under review in +[#50](https://github.com/NVIDIA/cuPhoton/pull/50) and +[#54](https://github.com/NVIDIA/cuPhoton/pull/54). They are not yet on `main`. The repeated XPOIS and shared Dragon executors use a READY/command/result protocol. Each worker sends READY after initialization, then waits on its own command queue. The coordinator waits for all READY messages before releasing @@ -114,7 +122,7 @@ uses only a result queue and needs no command-queue relocation. Neither path automatically changes the transport or raises the Python TCP thread ceiling; those remain launcher settings. -The shared Dragon executor publishes terminal success only after worker +The shared Dragon executor in pending #54 publishes terminal success only after worker shutdown and lifecycle audits. Per-round success does not establish a successful complete run. Silent worker exits and shutdown failures fail the run. After a reported round failure, healthy peers can finish and retain their @@ -194,14 +202,24 @@ worker CPU budget and runtime versions constant. Rotate run order and retain the first command round as well as subsequent rounds. Record separately: - Launcher-to-exit wall time, including startup, warmup and shutdown. -- Time from harness entry until all workers are ready. +- Time from harness entry until all workers are ready. The product READY + protocol is introduced by pending #50 and #54; `main` does not expose this + boundary. - Whole-batch time from release through collection of worker completions. - Worker execution time, with the treatment of output and receipt writes. - Coordinator artifact audits, scientific finalization and shutdown. -In persistent-round reports, `batch_wall_sec` stops at completion collection; -coordinator artifact audits and scientific finalization follow it. The shared -executor records component merging separately as `finalization_sec`. Keep +### Pending executor timing fields (#50 and #54) + +The following report fields require the open executor PRs. On `main`, +`summary.json` reports coordinator phases (`discovery`, `partition`, `setup`, +`launch`, `worker_join`, `result_collection`, `cleanup`, `artifact_visibility`) +and per-item timings; it has no `batch_wall_sec` or `finalization_sec`. + +In persistent-round reports introduced by #50 and #54, `batch_wall_sec` stops +at completion collection; coordinator artifact audits and scientific +finalization follow it. The shared executor in #54 records component merging +separately as `finalization_sec`. Keep readiness and shutdown outside this batch interval, and measure external launcher-to-exit time independently. MPI finalization and process exit are outside the executor's reported coordinator time. @@ -213,6 +231,8 @@ aggregate. A numerical warmup performed before READY in a separate harness does not necessarily warm the first command round. Match these policies before comparing results. +### Interpreting comparisons + Subtracting the longest worker duration from the whole-batch duration combines start skew, synchronization, receipt writes, serialization and collection. It cannot isolate network time. Validate complete output identities and scientific From 3195ddc25f6edaad802e6fabcd397881086ad3e0 Mon Sep 17 00:00:00 2001 From: Trent Nelson Date: Thu, 24 Sep 2026 18:22:11 -0700 Subject: [PATCH 5/5] Describe merged Dragon executor behavior Signed-off-by: Trent Nelson --- docs/components/xpois.md | 2 +- docs/dragon.md | 55 ++++++++++++++++------------------------ 2 files changed, 23 insertions(+), 34 deletions(-) diff --git a/docs/components/xpois.md b/docs/components/xpois.md index 0f06060e..1f8afbc8 100644 --- a/docs/components/xpois.md +++ b/docs/components/xpois.md @@ -717,7 +717,7 @@ transport of an already running Dragon session. The worker wall-time limit covers the entire invocation, including warmups and all measured rounds. See [Dragon transport and coordination](../dragon.md) for native HSTA TCP -launch options, consumer-local queues in the pending persistent executors, +launch options, consumer-local queues in the persistent executors, and measurement boundaries. ### Results and limits diff --git a/docs/dragon.md b/docs/dragon.md index 2b019c96..e775e1bb 100644 --- a/docs/dragon.md +++ b/docs/dragon.md @@ -6,20 +6,19 @@ Dragon launcher. It does not change cuPhoton's numerical backend or fit options. ## Execution models -The [XPOIS batch executor](components/xpois.md#launch-with-dragon) on `main` +The [XPOIS batch executor](components/xpois.md#launch-with-dragon) assigns a complete shard to each worker at launch. Workers process their -assigned items and send one terminal result to the coordinator. The following -extensions are under review; their commands require the corresponding PRs: +assigned items and send one terminal result to the coordinator in single-pass +mode. Repeated rounds and component commands retain workers as follows: | Path | Coordination and retained state | |---|---| | Single-pass XPOIS | One assigned shard and one terminal result per worker; no per-round command queue. | -| [Repeated XPOIS rounds (#50)](https://github.com/NVIDIA/cuPhoton/pull/50) | `--warmup-rounds` and `--measure-rounds` reuse workers and CUDA contexts. Each worker has a command queue on its own host and repeats its assigned shard. | -| [Shared executors (#54)](https://github.com/NVIDIA/cuPhoton/pull/54) and [component commands (#56)](https://github.com/NVIDIA/cuPhoton/pull/56) | xFit, XScan and `xscan run-pipeline` use the shared Dragon/MPI lifecycle. Dragon uses consumer-local command queues for both a single pass and repeated rounds; component workers retain their input/model context. | +| [Repeated XPOIS rounds](components/xpois.md#repeat-a-batch-in-persistent-workers) | `--warmup-rounds` and `--measure-rounds` reuse workers and CUDA contexts. Each worker has a command queue on its own host and repeats its assigned shard. | +| [Shared executors](../src/cuphoton/core/executors.py) and [component commands](components/xscan.md#distributed-inference) | xFit, XScan and `xscan run-pipeline` use the shared Dragon/MPI lifecycle. Dragon uses consumer-local command queues for both a single pass and repeated rounds; component workers retain their input/model context. | -### Pending persistent behavior (#50, #54 and #56) +### Persistent behavior -The following behavior requires the open PRs linked above. Persistence does not remove all per-item work. XPOIS still reads, transfers and writes ordinary scientific artifacts each round. Standalone xFit retains its loaded host input; XScan retains metadata, its model and loader across tasks. @@ -29,7 +28,7 @@ pipeline keeps intermediate arrays inside each GPU worker while it processes an image pair. Each GPU handles complete items; these paths do not split one image across GPUs. -### Launch descriptors on main +### Launch descriptors Dragon workers load immutable, hash-checked launch descriptors from the shared filesystem. Launch arguments carry descriptor references rather than the full @@ -71,11 +70,8 @@ output validation and cleanup checks. The Python TCP launch in the XPOIS guide remains an explicit alternative. Transport availability and the fastest queue layout depend on the Dragon build and workload. -## Pending command-queue protocol (#50 and #54) +## Place command queues with their consumers -This section describes the implementations under review in -[#50](https://github.com/NVIDIA/cuPhoton/pull/50) and -[#54](https://github.com/NVIDIA/cuPhoton/pull/54). They are not yet on `main`. The repeated XPOIS and shared Dragon executors use a READY/command/result protocol. Each worker sends READY after initialization, then waits on its own command queue. The coordinator waits for all READY messages before releasing @@ -116,13 +112,13 @@ its idle remote receives and higher control latency. Native HSTA has a different progress implementation; the Python TCP thread setting still applies to a Python TCP overlay. Consumer-local queues are not universally faster under HSTA. -The pending repeated-round and shared executors already use consumer-local +The repeated-round and shared executors use consumer-local command queues. The [single-pass XPOIS path](../src/cuphoton/xpois/dragon.py) uses only a result queue and needs no command-queue relocation. Neither path automatically changes the transport or raises the Python TCP thread ceiling; those remain launcher settings. -The shared Dragon executor in pending #54 publishes terminal success only after worker +The shared Dragon executor publishes terminal success only after worker shutdown and lifecycle audits. Per-round success does not establish a successful complete run. Silent worker exits and shutdown failures fail the run. After a reported round failure, healthy peers can finish and retain their @@ -162,8 +158,8 @@ a separate harness experiment, with one invocation per configuration. Later two-node/eight-GPU product checks covered XPOIS, standalone xFit and XScan, and the combined pipeline under Dragon and MPI. They established numerical parity, persistent identities and cleanup for the tested revisions. -They predate the latest lifecycle, loader and input-ownership fixes in -#50/#54/#56; those updated revisions still need distributed GPU requalification. +They predate later lifecycle, loader and input-ownership fixes; they do not +qualify the final implementations in #50/#54/#56. ### 256-GPU follow-up @@ -202,24 +198,16 @@ worker CPU budget and runtime versions constant. Rotate run order and retain the first command round as well as subsequent rounds. Record separately: - Launcher-to-exit wall time, including startup, warmup and shutdown. -- Time from harness entry until all workers are ready. The product READY - protocol is introduced by pending #50 and #54; `main` does not expose this - boundary. +- Time from harness or executor entry until all workers report READY. - Whole-batch time from release through collection of worker completions. - Worker execution time, with the treatment of output and receipt writes. - Coordinator artifact audits, scientific finalization and shutdown. -### Pending executor timing fields (#50 and #54) +### Executor timing fields -The following report fields require the open executor PRs. On `main`, -`summary.json` reports coordinator phases (`discovery`, `partition`, `setup`, -`launch`, `worker_join`, `result_collection`, `cleanup`, `artifact_visibility`) -and per-item timings; it has no `batch_wall_sec` or `finalization_sec`. - -In persistent-round reports introduced by #50 and #54, `batch_wall_sec` stops -at completion collection; coordinator artifact audits and scientific -finalization follow it. The shared executor in #54 records component merging -separately as `finalization_sec`. Keep +In persistent-round reports, `batch_wall_sec` stops at completion collection; +coordinator artifact audits and scientific finalization follow it. The shared +executor records component merging separately as `finalization_sec`. Keep readiness and shutdown outside this batch interval, and measure external launcher-to-exit time independently. MPI finalization and process exit are outside the executor's reported coordinator time. @@ -240,11 +228,12 @@ results, worker exits and runtime cleanup alongside timings. The control probe also observed an external Dragon launcher exit of zero after an application failure; inspect the application's terminal result as well as the launcher. -[The pipeline/stage benchmark (#55)](https://github.com/NVIDIA/cuPhoton/pull/55) +[The pipeline/stage benchmark](components/pipeline-stage-benchmark.md) compares a resident device pipeline with fresh processes and intermediate -files on one GPU. It measures workflow reuse and process/file costs, including -additional intermediate outputs; it does not compare Dragon transports or -reproduce the stock component CLI chain. +files on one GPU. It measures workflow reuse and process/file costs, retaining +raw timings and separately reporting the cost of additional hash verification. +It does not compare Dragon transports or reproduce the stock component CLI +chain. Free-threaded Python does not free an executor thread blocked in a native receive: that receive already releases the GIL. A separate serialization or