diff --git a/docs/README.md b/docs/README.md index 7a65283..b17bc4c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -38,6 +38,8 @@ intend to adapt. ## Adapting and validating - [Architecture](architecture.md): package boundaries and execution model. +- [Dragon transport and coordination](dragon.md): execution models, + consumer-local queues, transport selection, and timing boundaries. - [Adapting the workflows](adapting-workflows.md): a practical extension process. - [Data and artifact contracts](data-artifacts.md): shapes, files, metadata, diff --git a/docs/components/xpois.md b/docs/components/xpois.md index 8f132cd..1f8afbc 100644 --- a/docs/components/xpois.md +++ b/docs/components/xpois.md @@ -716,6 +716,10 @@ selection remains a launcher option; `--executor dragon` cannot change the transport of an already running Dragon session. The worker wall-time limit covers the entire invocation, including warmups and all measured rounds. +See [Dragon transport and coordination](../dragon.md) for native HSTA TCP +launch options, consumer-local queues in the persistent executors, +and measurement boundaries. + ### Results and limits Both routes print a compact result containing the executor, run ID, run diff --git a/docs/dragon.md b/docs/dragon.md new file mode 100644 index 0000000..e775e1b --- /dev/null +++ b/docs/dragon.md @@ -0,0 +1,242 @@ +# Dragon transport and coordination + +Dragon places and manages cuPhoton workers; the selected numerical backend +runs each image pair inside its worker. Transport selection belongs to the +Dragon launcher. It does not change cuPhoton's numerical backend or fit options. + +## Execution models + +The [XPOIS batch executor](components/xpois.md#launch-with-dragon) +assigns a complete shard to each worker at launch. Workers process their +assigned items and send one terminal result to the coordinator in single-pass +mode. Repeated rounds and component commands retain workers as follows: + +| Path | Coordination and retained state | +|---|---| +| Single-pass XPOIS | One assigned shard and one terminal result per worker; no per-round command queue. | +| [Repeated XPOIS rounds](components/xpois.md#repeat-a-batch-in-persistent-workers) | `--warmup-rounds` and `--measure-rounds` reuse workers and CUDA contexts. Each worker has a command queue on its own host and repeats its assigned shard. | +| [Shared executors](../src/cuphoton/core/executors.py) and [component commands](components/xscan.md#distributed-inference) | xFit, XScan and `xscan run-pipeline` use the shared Dragon/MPI lifecycle. Dragon uses consumer-local command queues for both a single pass and repeated rounds; component workers retain their input/model context. | + +### Persistent behavior + +Persistence does not remove all per-item work. XPOIS still reads, transfers and +writes ordinary scientific artifacts each round. Standalone xFit retains its +loaded host input; XScan retains metadata, its model and loader across tasks. +Distributed XScan defaults to `--num-workers 0`. Explicitly requested loader +processes start before READY and persist across tasks and rounds. The combined +pipeline keeps intermediate arrays inside each GPU worker while it +processes an image pair. Each GPU handles complete items; these paths do not +split one image across GPUs. + +### Launch descriptors + +Dragon workers load immutable, hash-checked launch descriptors from the shared +filesystem. Launch arguments carry descriptor references rather than the full +manifest, with a 96 KiB size guard. This bounds process-launch payloads; queue +placement addresses a separate coordination cost. + +## Select the application and overlay transports + +Dragon has separate application and infrastructure overlay transports. +`-t tcp -o tcp` selects its Python TCP transport for both. With a Dragon 0.14.2 +installation that includes native HSTA, select native HSTA for application +traffic and retain Python TCP for the overlay: + +```bash +DRAGON_HSTA_FORCE_BACKEND=tcp \ + .venv/bin/dragon -m -N 2 -w slurm -t hsta -o tcp \ + examples/xpois/dragon_batch.py \ + --backend cupy \ + --manifest /shared/manifests/fixed-32.yaml \ + --output-dir /shared/results/xpois-hsta \ + --name fixed-32-hsta16 \ + --max-workers 16 \ + --worker-timeout-sec 3600 +``` + +As in the XPOIS launch example, these paths must be accessible to the allocated +nodes. The manifest needs enough image pairs for the selected workers. Adapt +the node and worker counts to the allocation. + +`DRAGON_HSTA_FORCE_BACKEND=tcp` selects HSTA's native TCP implementation. This +configuration uses TCP rather than UCX or RDMA. Inspect the launcher and +transport logs to confirm HSTA started on each node and exited successfully; +record the Dragon version, arguments and environment with the run. INFO logs +can confirm native HSTA startup without providing network byte counters. + +HSTA TCP is a useful candidate for workloads with frequent small messages. +Qualify it on the target installation with representative numerical work, +output validation and cleanup checks. The Python TCP launch in the XPOIS guide +remains an explicit alternative. Transport availability and the fastest queue +layout depend on the Dragon build and workload. + +## Place command queues with their consumers + +The repeated XPOIS and shared Dragon executors use a READY/command/result +protocol. Each worker sends READY after initialization, then waits on its own +command queue. The coordinator waits for all READY messages before releasing +a round. Custom coordinators with the same protocol need to consider queue +placement. + +In Dragon 0.14.2's Python TCP transport, remote receives, sends and polls share +an executor. A receive waiting on an empty remote queue can occupy a thread +until a message arrives or its timeout expires. If command queues are all on +the coordinator's node, enough idle remote workers can consume the threads +needed to deliver the remaining READY messages. Both sides then wait. + +For this pattern with Python TCP, place each command queue on its consumer's +host. Keep the result queue on its consumer, the coordinator: + +```python +from dragon.infrastructure.policy import Policy +from dragon.native.machine import Node +from dragon.native.queue import Queue + +# worker_node_id comes from the allocation's discovered Dragon node IDs. +worker_host = Node(worker_node_id).hostname +worker_policy = Policy( + placement=Policy.Placement.HOST_NAME, + host_name=worker_host, +) +command_queue = Queue(maxsize=1, policy=worker_policy) +``` + +Use the hostname reported by Dragon's node discovery for both the queue and +worker placement. This makes the worker's idle receive local and the +coordinator's command send remote. It removes the central population of idle +receive waits while retaining command traffic and result collection. + +`DRAGON_TRANSPORT_TCP_MAX_THREADS` can increase the Python TCP executor ceiling. +It allowed the centralized layout to make progress in testing, but preserved +its idle remote receives and higher control latency. Native HSTA has a different +progress implementation; the Python TCP thread setting still applies to a +Python TCP overlay. Consumer-local queues are not universally faster under HSTA. + +The repeated-round and shared executors use consumer-local +command queues. The [single-pass XPOIS path](../src/cuphoton/xpois/dragon.py) +uses only a result queue and needs no command-queue relocation. Neither path +automatically changes the transport or raises the Python TCP thread ceiling; +those remain launcher settings. + +The shared Dragon executor publishes terminal success only after worker +shutdown and lifecycle audits. Per-round success does not establish a +successful complete run. Silent worker exits and shutdown failures fail the +run. After a reported round failure, healthy peers can finish and retain their +artifacts. Check the root terminal summary before consuming a benchmark result. + +## Evidence and timing boundaries + +A Dragon 0.14.2 control experiment used two nodes, with the coordinator on one +and 64 CPU worker processes on the other. Each worker returned a 1024-byte +payload per round; no GPU work or timed receipt-file writes occurred. Each +successful invocation had 12 rounds. Two invocations per configuration reversed +the queue-layout order. The table gives the median of rounds 2–12 for each +invocation; the first round was retained separately. + +| Application transport | Command queue host | Python TCP ceiling | Later-round medians | +|---|---|---:|---:| +| Python TCP | Coordinator | 1024 | 47.235 / 45.307 ms | +| Python TCP | Consumer | 1024 | 33.604 / 28.123 ms | +| Native HSTA TCP | Coordinator | 32 for overlay | 2.371 / 2.545 ms | +| Native HSTA TCP | Consumer | 32 for overlay | 3.027 / 3.093 ms | + +A separate invocation with coordinator queues and a 32-thread Python TCP +ceiling stalled after 32 of 64 READY messages. Consumer queues completed all +12 rounds at the same ceiling. HSTA's first rounds took 5.6–6.3 ms, compared +with the 2.4–3.1 ms later medians above. These are whole control-round timings, +not network-only measurements or predicted application speedups. + +A separate persistent-worker imaging harness compared MPI and both queue +layouts under Python TCP and HSTA TCP on eight GB200 GPUs. It processed +16 image-pair occurrences per round from two base pairs, for three rounds per +configuration. All 240 measured scientific outputs agreed. Batch medians +ranged from 7.515 to 7.719 seconds: the large control-only improvement did not +produce a comparable whole-pipeline gain. These results do not qualify a +512-GPU speedup or establish the best layout at larger node counts. This was +a separate harness experiment, with one invocation per configuration. + +Later two-node/eight-GPU product checks covered XPOIS, standalone xFit and +XScan, and the combined pipeline under Dragon and MPI. They established +numerical parity, persistent identities and cleanup for the tested revisions. +They predate later lifecycle, loader and input-ownership fixes; they do not +qualify the final implementations in #50/#54/#56. + +### 256-GPU follow-up + +A later instrumented harness compared MPI and Dragon on 256 GPUs across +64 nodes. Fixed batches contained 512 image-pair occurrences (two per GPU); +weak batches contained 4,096 (16 per GPU). All Dragon treatments used the +1024-thread Python TCP ceiling for the applicable application or overlay +transport. The table shows the range of +batch duration minus longest worker duration across all three rounds of each +accepted Dragon invocation, including the first command round: + +| Dragon treatment | Fixed batch remainder | Weak batch remainder | +|---|---:|---:| +| Python TCP, coordinator queues | 209.590–357.407 ms | 361.694–462.434 ms | +| Python TCP, consumer queues | 24.325–32.729 ms | 39.762–45.234 ms | +| Native HSTA TCP, coordinator queues | 8.126–8.629 ms | 7.863–9.451 ms | + +The fixed-batch consumer results combine two accepted launches; each other +table cell comes from one. The complete MPI/Dragon campaign retained 10 accepted +launches, with 30 measured rounds and 58,368 validated image-pair outputs, +alongside two startup failures and four unrun launches. Consumer-local queues +and HSTA reduced the observed coordination remainder, which also includes +dispatch, serialization and collection. It is not a network-only timer. + +The coordinator-queue control also lacked the historical multi-second spike. +Allocation, scheduler segmentation and logging differed from the earlier +campaign, so the spike's cause remains unresolved. There was no 512-GPU +retest. These measurements exercised the retained harness, not the latest +product PR revisions; they do not establish large-scale performance or launch +reliability for those revisions. + +### Compare timing boundaries + +For a new comparison, hold the corpus, item count, numerical configuration, +worker CPU budget and runtime versions constant. Rotate run order and retain +the first command round as well as subsequent rounds. Record separately: + +- Launcher-to-exit wall time, including startup, warmup and shutdown. +- Time from harness or executor entry until all workers report READY. +- Whole-batch time from release through collection of worker completions. +- Worker execution time, with the treatment of output and receipt writes. +- Coordinator artifact audits, scientific finalization and shutdown. + +### Executor timing fields + +In persistent-round reports, `batch_wall_sec` stops at completion collection; +coordinator artifact audits and scientific finalization follow it. The shared +executor records component merging separately as `finalization_sec`. Keep +readiness and shutdown outside this batch interval, and measure external +launcher-to-exit time independently. MPI finalization and process exit are +outside the executor's reported coordinator time. + +Requested warmup rounds exercise real dispatch, work and collection. Their +outputs and timings remain in the report. Only measured rounds enter timing +statistics; failed warmup rounds, measured rounds or cleanup invalidate the +aggregate. A numerical warmup performed before READY in a separate harness +does not necessarily warm the first command round. Match these policies before +comparing results. + +### Interpreting comparisons + +Subtracting the longest worker duration from the whole-batch duration combines +start skew, synchronization, receipt writes, serialization and collection. It +cannot isolate network time. Validate complete output identities and scientific +results, worker exits and runtime cleanup alongside timings. The control probe +also observed an external Dragon launcher exit of zero after an application +failure; inspect the application's terminal result as well as the launcher. + +[The pipeline/stage benchmark](components/pipeline-stage-benchmark.md) +compares a resident device pipeline with fresh processes and intermediate +files on one GPU. It measures workflow reuse and process/file costs, retaining +raw timings and separately reporting the cost of additional hash verification. +It does not compare Dragon transports or reproduce the stock component CLI +chain. + +Free-threaded Python does not free an executor thread blocked in a native +receive: that receive already releases the GIL. A separate serialization or +dispatch profile is needed to establish whether Python execution is a remaining +bottleneck. The experiments above used conventional CPython 3.12; they do not +qualify a free-threaded Dragon runtime.