Skip to content

madspace: run the unit test suite on CUDA and HIP - #265

Merged
oliviermattelaer merged 11 commits into
mainfrom
feat-madspace-gpu-tests
Oct 14, 2026
Merged

oliviermattelaer merged 11 commits into
mainfrom
feat-madspace-gpu-tests

Conversation

@theoheimel

@theoheimel theoheimel commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Runs the full madspace unit test suite on CUDA and HIP in the CECI GPU CI, in addition to the existing CPU tests.

Changes

  • Tests on torch. madspace/tests is converted from numpy to torch. Inputs are torch tensors, so madspace runs each function on the device of its inputs, and the assertions use torch.testing.assert_close. numpy remains only in host-side code.
  • MADSPACE_TEST_DEVICE=cpu|cuda|hip (default cpu). conftest.py sets the default torch device and dtype. If a GPU is requested but torch has none, the run fails rather than skipping silently. The helpers in madspace_testing.py are DEVICE, default_context(), new_context(), and generator/rand/randn/randint.
  • Same random numbers on every device. Random numbers are drawn on the CPU and then moved to the device, so the statistical tests see identical inputs on CPU, CUDA and HIP.
  • cpu_only marker for host-side tests, which are skipped on a GPU: test_lhe, test_systematics, test_event_histograms, test_survey_job_failure, and the yaml signature check in test_cut_kernels.
  • GPU checks merged into the suite. The checks in .github/gpu_ci/madspace_checks.py are now regular tests. The large histogram against the CPU runtime is in test_histogram_op.py, and the rank-4 kernel check is in test_rank4_kernels.py. madspace_checks.py and madspace_ops_gpu.sh are removed.
  • GPU CI. In gpu_runner_ci.yml, the madspace_ops job is replaced by madspace_tests. It runs MADSPACE_TEST_DEVICE=$BACKEND pytest madspace/tests with the python3 of the modules, through .github/gpu_ci/madspace_tests_gpu.sh. If the modules lack pytest or scipy, they are installed into the cluster cache. The log and junit.xml are uploaded as an artifact.
  • CPU CI. madspace.yml sets MADSPACE_TEST_DEVICE=cpu explicitly.
  • CUDA torch on manneback. The only PyTorch module on manneback is a CPU build. In that case madspace_tests_gpu.sh installs a torch==2.8.0 cu128 wheel into the cluster cache and puts it ahead of the module's torch.

madspace fixes found by the GPU runs

  • GPU runtime: functions without instructions (src/gpu/runtime.cu). Such a function got 0 streams, so run() indexed an empty stream vector. This happens with a Cuts whose only cut has nothing to act on, for example an SFOS-pair cut without an SFOS pair. A function now always has at least the main stream.
  • Kinematics: precision loss under FMA contraction (p1com_from_tabs_cs, src/kernels/kinematics.hpp).
    • The transverse momentum of p1 relative to pa was computed as sqrt(pp² − p1z²). Near the ends of the t range, where the t sampling puts most points, this cancels, so an O(eps) rounding difference grew to O(eps/(pt/pp)²).
    • nvcc's default FMA contraction rounds this differently from the CPU and HIP. As a result the s23 range of the 2→3 blocks, and the color-ordered round trip with cuts, were off by up to 1e-4 on CUDA.
    • p1z and pt are now computed from the distances to the two ends of the |t| range, pt² = (|t|−t_lo)(t_hi−|t|)/(4|pa|²), with t_lo factorized so that it is exactly 0 for massless particles.
    • Found with a per-instruction CUDA-vs-CPU comparison on the cluster. A CUDA build with --fmad=false passed everything, which confirmed contraction as the cause.
    • Remaining: in the inverse, the lower end of the s23 range (center − half_width) still differs between CUDA and CPU by about 3e-7 at q999. This is within the test tolerances and could be a follow-up.
  • Test tolerances below rounding (Linux x86 CPU):
    • test_t_channel momentum conservation: the absolute tolerance now scales with CM_ENERGY.
    • test_composite_round_trip: the bound on the single worst point is now 1e-3.

Status

result
CPU (madspace.yml, Linux x86 + macOS) pass
HIP, lemaitre4 (AMD MI300A) 1720 passed, 31 skipped
CUDA, manneback (NVIDIA A10) 1720 passed, 31 skipped

The 31 skips are made up of 20 cpu_only tests and 11 PDF/LHAPDF tests that the clusters cannot run because they lack LHAPDF or the PDF set.

Seed changes. These statistical tests were marginal and failed with the new torch random stream:

  • test_t_channel (1234→43).
  • test_decay_topology (+1).
  • test_pair_mass_floor_subtree (SEED+1→SEED+2). The any-mode W-channel weights are very heavy-tailed: the mapping mean ranges from 1.4e8 to 7e8 across seeds. This deserves a separate look.

The GPU workflows were run from this branch through a temporary ceci-gpu deployment-branch rule.

…UDA and HIP in the GPU CI

- tests converted from numpy to torch; conftest sets the default torch device from
  MADSPACE_TEST_DEVICE (cpu, cuda or hip) and fails if a GPU is requested but missing
- random numbers drawn on the CPU and moved to the device: same inputs on every device
- host-side tests (LHE, systematics, event histograms, survey jobs, yaml signatures)
  marked cpu_only, skipped on a GPU
- the GPU checks of .github/gpu_ci/madspace_checks.py (histogram, rank-4 kernels) are
  now regular tests
- GPU CI: job madspace_ops replaced by madspace_tests (pytest on the GPU)
…t instructions

A Cuts function whose only cut has nothing to act on (no SFOS pair) has no
instructions; with zero streams GpuRuntime::run indexed an empty stream vector.
- t_channel momentum conservation: absolute tolerance relative to CM_ENERGY
- composite round trip: the single worst point may lose 1e-4
… ends of the t range

pt = sqrt(pp^2 - p1_z^2) cancelled near an end of the t range, where the t
sampling puts most points: an O(eps) difference in the rounding, e.g. FMA
contraction by nvcc, became O(eps / (pt/pp)^2) in pt and in the s23 range of
the 2->3 blocks, and broke the CUDA round trip of ColorOrderedMapping with cuts.
@theoheimel
theoheimel marked this pull request as ready for review October 13, 2026 09:28
@oliviermattelaer

Copy link
Copy Markdown
Contributor

Relevant findings by claude (I dropped many other I can post them if you want):

  • Kinematics precision: the new formula in madspace/src/kernels/kinematics.hpp (p1com_from_tabs_cs) loses precision for a spacelike incoming momentum with negative energy. That happens in later steps of a t-channel chain that alternates sides.
  • Kinematics NaN: the same formula gives 0/0 = NaN at the massless threshold, where the old code stayed finite.
  • Seed choice: three test seems to be fail/pass depending of the seed choice. This need to be investigate (but can be in another PR)
  • Note that we need to be light on such type of CI (at least for the moment where we trigger a lot of them). Manneback is likely going to hit the wall time already

@oliviermattelaer
oliviermattelaer merged commit 63ed805 into main Oct 14, 2026
140 checks passed
@oliviermattelaer
oliviermattelaer deleted the feat-madspace-gpu-tests branch October 14, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants