Compare persistent pipeline and separate stage execution

docs/components/pipeline-stage-benchmark.md

@@ -0,0 +1,148 @@
+# Pipeline versus separate-stage benchmark
+
+This benchmark compares one persistent GPU worker running xPOIS, xFit and
+XScan with three fresh processes that exchange intermediate arrays through
+files. Both treatments use the same device numerical APIs, inputs,
+checkpoint and per-image candidate batches. The comparison measures the
+combined cost of process lifetime, transfers and intermediate artifacts.
+
+The separate-stage treatment uses benchmark-specific entry points. It does
+not invoke the stock component CLI commands: those use different host
+materialization, coefficient-solving and probability-calculation paths.
+
+## Run the comparison
+
+Use a CUDA 13 GPU with both CuPy and PyTorch available. From the repository
+root:
+
+```bash
+uv sync --locked --extra gpu
+
+uv run --locked --extra gpu cuphoton xscan benchmark-pipeline \
+  --output /tmp/cuphoton-pipeline-forward \
+  --images 4 --image-size 256 --candidates 9 --stamp-size 17 \
+  --seed 2026 --device cuda:0 --warmup 1 --repeat 3 \
+  --order pipeline-first
+```
+
+Each output directory must be new. The default fixture contains four
+independent image pairs and nine candidates per image, so each measured
+round processes four pairs and 36 candidate stamps. Candidates are supplied
+positions; this benchmark does not include source detection. All images
+run serially on the selected GPU.
+
+Reverse the treatment order using the *same generated fixture*:
+
+```bash
+uv run --locked --extra gpu cuphoton xscan benchmark-pipeline \
+  --output /tmp/cuphoton-pipeline-reverse \
+  --config /tmp/cuphoton-pipeline-forward/input/config.json \
+  --items /tmp/cuphoton-pipeline-forward/input/items.json \
+  --warmup 1 --repeat 3 --order staged-first
+```
+
+`--config` and `--items` must be supplied together. They contain
+`DevicePipelineConfig.to_payload()` and an ordered list of
+`DevicePipelineItem.to_payload()` values, including absolute file paths and
+hashes. With these options, the manifests determine the device and workload;
+fixture-generation options do not replace them. Preserve the referenced
+files when running the reversed comparison. Use the same CPU affinity,
+thread settings and GPU, without another workload running concurrently.
+`--timeout` bounds each child invocation; its default is 600 seconds.
+
+## What each treatment runs
+
+The pipeline initializes `DeviceWorkerContext` once, performs its warmup
+rounds, then retains the model and CUDA context for measured rounds. Each
+item flows through constant-kernel subtraction, stamp extraction, Gaussian
+difference fitting, xFit feature conversion and model inference. Device
+arrays pass to PyTorch through DLPack. The result contains compact scientific
+evidence and predictions.
+
+The separate-stage treatment starts one xPOIS child, one xFit child and one
+XScan child per complete round. Each child processes all image items in
+manifest order, keeping the candidate batch for each image unchanged. It
+writes lossless, uncompressed NPY arrays between stages: extracted difference
+stamps for xFit, triplets and features for XScan, and the compact scientific
+outputs shared with the pipeline. Unused full subtraction images, basis
+kernels and fit residuals are not transferred or written. Its preliminary
+rounds prepare caches; subsequent measured rounds still create fresh processes.
+
+To inspect a manual round directly, invoke the command three times with
+`--stage xpois`, `--stage xfit` and `--stage xscan`, in that order. Supply the
+same `--config`, `--items` and `--output` directory each time. The stage
+commands reject changed upstream artifacts and refuse to overwrite a stage.
+
+Both treatments use float64 subtraction and xFit inputs, unweighted xFit
+stamps, float32 triplets/features, and the same GPU sigmoid. Inference forces
+AMP, TF32, compilation and cuDNN benchmarking off. An xPOIS variance plane
+does not become an xFit variance plane.
+
+## Read the timers
+
+| Measurement | Boundary |
+| --- | --- |
+| Pipeline `setup_seconds` / `context_load_seconds` | Worker setup and context/model initialization, recorded separately from numerical warmup and measured batches. The external invocation also includes interpreter startup. |
+| Pipeline `batch_seconds` | One ordered image batch through completed device work and per-item result JSON writes. Measured batches reuse the initialized, warmed worker. |
+| Pipeline `invocation_external_seconds` | Parent-observed process duration, including imports, setup, all warmup and measured rounds, final summary and process exit. |
+| Staged `batch_seconds` | Parent clock before launching xPOIS through successful XScan process exit, including all three process lifetimes and intermediate files. |
+| Staged `extra_hashing_seconds` | Intermediate-artifact SHA-256 reads/writes and original-input rechecks repeated by xFit/XScan, measured inside the raw batch timer. |
+| Staged `batch_seconds_without_extra_hashing` | Raw batch time minus that additional verification time; retains process startup, numerical work, transfers and file I/O. |
+| Staged per-process `external_seconds` | Parent-observed duration of that individual child, including imports and shutdown. |
+
+Stage summaries also record internal setup, reads, uploads, computation,
+downloads and writes. Their GPU operations synchronize explicitly; the
+pipeline's existing component timers are asynchronous host elapsed times,
+with pending work completed at the terminal copy. Comparing those component
+timers as isolated GPU kernel durations would be misleading. Use the
+completed batch timers for the main workflow comparison.
+
+The report's `fresh_stages_over_warm_pipeline_ratio` divides the median
+staged batch time **without extra hashing** by the median warm pipeline batch
+time. Both arms still include their common original-input hashes on read and
+once after execution; checkpoint/schema verification remains in setup. All
+verification still executes. The adjustment subtracts measured hash time; it
+is not a separate hash-disabled run and cannot undo cache effects caused by
+verification. `raw_fresh_stages_over_warm_pipeline_ratio` preserves the ratio
+of unadjusted medians. Both include repeated startup and file costs. Keep setup and
+whole-invocation measurements alongside that ratio when discussing a service
+that may process only a few batches.
+
+The benchmark preserves existing filesystem and CUDA caches. Fresh
+processes can benefit from both. Writes close files without `fsync`, so
+elapsed time does not measure durable storage completion. Fixture preparation
+and the final parity audit run outside the treatment timers.
+
+## Acceptance and artifacts
+
+The benchmark checks every warmup and measured item against the other
+treatment. Acceptance requires exact equality of all 22 compact scientific
+arrays, including shapes, dtypes and NaN locations, plus matching candidate
+identity/order, fit metadata, subtraction diagnostics and predictions. It
+verifies retained file hashes and rechecks the original inputs. This checks
+equivalence between two compositions of the same algorithms; it does not
+independently establish their astronomical accuracy. Only arrays needed by
+the next stage or by this compact parity contract are retained.
+
+`manifest.json` records the configuration, ordered input descriptors and
+runtime settings. Treatment subdirectories retain each attempted round and
+child log. `parity.json` records numerical comparisons; a successful
+`report.json` contains all round times and the measured medians. Report-level
+provenance includes the selected GPU name/UUID, NVIDIA driver version, CUDA
+driver API/runtime versions and the imported CuPy version. Package discovery
+also records the installed distribution providing `cupy`. Driver-query
+failures are retained explicitly rather than silently omitting the field. A failure
+stops the run and leaves its logs and `failure.json` for inspection. A
+successful process exit alone does not satisfy the parity check.
+
+The default inputs are synthetic textured images with planted dipoles, a
+known convolution kernel, noise and an exclusion mask around the candidates.
+The fitted 15×15 kernel uses Gaussian sigmas 1.5/3/6 and degrees 2/1/0,
+a constant background and no flux constraint.
+A small, randomly initialized triplet model has a nonzero xFit fusion branch.
+A separate CPU fit creates the canonical feature-schema artifacts before
+measurement. Input arrays and model weights repeat for the same seed;
+provenance paths and timestamps can change artifact hashes across newly
+prepared fixtures. This fixture exercises the workflow and numerical
+boundaries. It provides no trained-classifier accuracy result or claim of
+representative production throughput.

docs/components/xscan.md

@@ -296,6 +296,11 @@ outputs, and merging and validation occur after the timed worker phase.
 
 ## Persistent XPOIS, xFit and XScan pipeline
 
+For a reproducible comparison with separately launched stages and intermediate
+files, see the [pipeline stage benchmark](pipeline-stage-benchmark.md). It
+checks the same scientific outputs while reporting startup and warm execution
+separately.
+
 The Python API in `cuphoton.xscan.device_pipeline` runs complete image pairs
 through constant-kernel XPOIS, stamp extraction, Gaussian difference-mode
 xFit, feature conversion and triplet XScan inference. A `DeviceWorkerContext`

src/cuphoton/xscan/commands.py

@@ -7,14 +7,17 @@
 from __future__ import annotations
 
 import json
+import math
 import os
 from pathlib import Path
+from types import SimpleNamespace
 from typing import Any, Callable, TypeVar
 
 from cuphoton.core.bulk import validate_identifier
 from cuphoton.core.cli import (
     BoolInvariant,
     CommandError,
+    FloatInvariant,
     InvariantAwareCommand,
     NonNegativeIntegerInvariant,
     PositiveIntegerInvariant,
@@ -2156,3 +2159,132 @@ def run(self) -> None:
             run_name=self.run_name or None,
         )
         self._emit_json({"run_dir": str(result.run_dir), **result.summary})
+
+
+class BenchmarkPipelineCommand(XScanCommand):
+    """Compare the device pipeline with separate file-mediated stages."""
+
+    _name_ = "benchmark-pipeline"
+
+    output = None
+    config = None
+    items = None
+    images = None
+    image_size = None
+    candidates = None
+    stamp_size = None
+    seed = None
+    device = None
+    warmup = None
+    repeat = None
+    timeout = None
+    order = None
+    stage = None
+
+    class OutputArg(PathSpecInvariant):
+        _arg = "--output"
+        _help = "New output directory for benchmark receipts and artifacts."
+        _mandatory = True
+
+    class ConfigArg(PathSpecInvariant):
+        _arg = "--config"
+        _help = "Existing pipeline config JSON; requires --items."
+        _default = None
+
+    class ItemsArg(PathSpecInvariant):
+        _arg = "--items"
+        _help = "Existing pipeline items JSON; requires --config."
+        _default = None
+
+    class ImagesArg(PositiveIntegerInvariant):
+        _arg = "--images"
+        _help = "Synthetic image-pair count. [default: %default]"
+        _default = 4
+
+    class ImageSizeArg(PositiveIntegerInvariant):
+        _arg = "--image-size"
+        _help = "Synthetic square image width in pixels. [default: %default]"
+        _default = 256
+
+    class CandidatesArg(PositiveIntegerInvariant):
+        _arg = "--candidates"
+        _help = "Candidates per synthetic image. [default: %default]"
+        _default = 9
+
+    class StampSizeArg(PositiveIntegerInvariant):
+        _arg = "--stamp-size"
+        _help = "Square candidate stamp width in pixels. [default: %default]"
+        _default = 17
+
+    class SeedArg(NonNegativeIntegerInvariant):
+        _arg = "--seed"
+        _help = "Synthetic fixture random seed. [default: %default]"
+        _default = 2026
+
+    class DeviceArg(StringInvariant):
+        _arg = "--device"
+        _help = "CUDA device for the synthetic fixture. [default: %default]"
+        _default = "cuda:0"
+
+    class WarmupArg(PositiveIntegerInvariant):
+        _arg = "--warmup"
+        _help = "Recorded warmup rounds per treatment. [default: %default]"
+        _default = 1
+
+    class RepeatArg(PositiveIntegerInvariant):
+        _arg = "--repeat"
+        _help = "Measured rounds per treatment. [default: %default]"
+        _default = 3
+
+    class TimeoutArg(FloatInvariant):
+        _arg = "--timeout"
+        _help = (
+            "Positive child-process timeout in seconds. [default: %default]"
+        )
+        _default = 600.0
+
+        @classmethod
+        def validate(cls, value: Any) -> float | None:
+            converted = super().validate(value)
+            if converted is not None and (
+                not math.isfinite(converted) or converted <= 0
+            ):
+                raise ValueError("must be finite and greater than zero")
+            return converted
+
+    class OrderArg(SetInvariant):
+        _arg = "--order"
+        _help = "Treatment launch order. [default: %default]"
+        _set = {"pipeline-first", "staged-first"}
+        _default = "pipeline-first"
+
+    class StageArg(SetInvariant):
+        _arg = "--stage"
+        _help = (
+            "Comparison or individual subprocess stage. [default: %default]"
+        )
+        _set = {"compare", "pipeline", "xpois", "xfit", "xscan"}
+        _default = "compare"
+
+    def run(self) -> None:
+        from .pipeline_benchmark.runner import run_benchmark
+
+        self._call(
+            run_benchmark,
+            SimpleNamespace(
+                output=Path(self.output).expanduser(),
+                config=self._path(self.config),
+                items=self._path(self.items),
+                images=self.images,
+                image_size=self.image_size,
+                candidates=self.candidates,
+                stamp_size=self.stamp_size,
+                seed=self.seed,
+                device=self.device,
+                warmup=self.warmup,
+                repeat=self.repeat,
+                timeout=self.timeout,
+                order=self.order,
+                stage=self.stage,
+            ),
+        )

src/cuphoton/xscan/pipeline_benchmark/__init__.py

@@ -0,0 +1,5 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Reproducible pipeline versus file-mediated stage measurement helpers."""