Repository navigation
ci: use GPU-aware MPI halos on GPU test runs (opt-out input) - #87
Merged
Merged
Conversation
MPAS host-stages every halo exchange (acc update self/device around the exchange) unless config_gpu_aware_mpi = .true. is set in &development, so the GPU CI runs spend most of their time stepping on host<->device copies. Add a gpu-aware-mpi input (default false) to the run-mpas and run-perturb-mpas actions. When true it sets config_gpu_aware_mpi in the namelist (creating &development if missing, idempotent) and the MPI library's GPU-support environment (MPICH_GPU_SUPPORT_ENABLED=1, or the OpenMPI UCX/CUDA settings), via two small shared scripts. The GPU test workflows (_test-gpu ECT members, _test-bfb and _test-perf GPU legs) get a gpu-aware-mpi workflow input defaulting to true, so GPU runs use device-resident halos; CPU legs are unchanged. Set it to false to fall back to host-staged halos, e.g. if a container's MPI lacks CUDA support (MPI then aborts rather than silently staging). On Derecho (Cray MPICH) the namelist change alone gives bit-identical output at 30km on 4 GPUs and 2.2-2.7x faster time stepping. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Namelist editing can silently fail to enable GPU-aware MPI, and dispatch workflows do not expose the fallback input.
2 open findings
What changed in this PR
Enables GPU-aware MPI halo exchanges in GPU CI runs to reduce host–device transfers while leaving CPU legs unchanged.
Changes:
- Adds optional GPU-aware MPI inputs to both run actions.
- Introduces shared namelist and MPI environment helpers.
- Defaults GPU test workflows to GPU-aware MPI.
| File | Description |
|---|---|
.github/workflows/_test-perf.yml |
Enables GPU-aware MPI for GPU performance legs. |
.github/workflows/_test-gpu.yml |
Passes the setting to GPU ensemble runs. |
.github/workflows/_test-bfb.yml |
Enables the setting for GPU comparison variants. |
.github/scripts/gpu-aware-mpi-env.sh |
Configures MPI GPU-support settings. |
.github/scripts/enable-gpu-aware-mpi.sh |
Enables GPU-aware MPI in the namelist. |
.github/actions/run-perturb-mpas/action.yml |
Applies configuration to ensemble members. |
.github/actions/run-mpas/action.yml |
Adds optional GPU-aware MPI configuration. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+31
to
+35
| gpu-aware-mpi: | ||
| description: 'Use GPU-aware MPI halo exchanges on the GPU (OpenACC) runs (config_gpu_aware_mpi + MPI GPU-support env). Needs CUDA-enabled MPI in the container; set false to fall back to host-staged halos.' | ||
| required: false | ||
| type: boolean | ||
| default: true |
…les comments and case-sensitive groups' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The opt-out only existed on the reusable workflows (_test-gpu, _test-bfb, _test-perf), so a dispatched GPU run always got the new default (true) and could not select host-staged halos when a container's MPI lacks CUDA support. Add a boolean gpu_aware_mpi dispatch input (default true) to perf-cpu-vs-gpu, bfb-decomp-gpu, bfb-io-gpu, bfb-nvhpc-cpu-vs-gpu, test-gpu-mpich and test-gpu-openmpi and forward it as gpu-aware-mpi. test-cross-repo has its GPU job commented out and is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The check-gpu-aware-mpi workflow showed that the CI OpenMPI GPU container
(OpenMPI 5.0.9 --with-cuda: accelerator:cuda + btl:smcuda) has no ucx pml/osc
component, so forcing --mca pml ucx / OMPI_MCA_pml=ucx aborts MPI_Init
("Framework: pml, Component: ucx not found"). That would have broken every
OpenMPI GPU job now that GPU-aware halos are the default. OpenMPI is CUDA-aware
on a node by default in that image (the default environment passes a device
buffer exchange), so set nothing for it.
For MPICH (4.3.2, ch4:ucx, --with-cuda) set MPIR_CVAR_ENABLE_GPU=1, the stock
MPICH variable, in addition to the Cray MPICH one; both passed.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MPAS host-stages every halo exchange (acc update self/device around the exchange) unless config_gpu_aware_mpi = .true. is set in &development, so the GPU CI runs spend most of their time stepping on host<->device copies.
Add a gpu-aware-mpi input (default false) to the run-mpas and run-perturb-mpas actions. When true it sets config_gpu_aware_mpi in the namelist (creating &development if missing, idempotent) and the MPI library's GPU-support environment (MPICH_GPU_SUPPORT_ENABLED=1, or the OpenMPI UCX/CUDA settings), via two small shared scripts.
The GPU test workflows (_test-gpu ECT members, _test-bfb and _test-perf GPU legs) get a gpu-aware-mpi workflow input defaulting to true, so GPU runs use device-resident halos; CPU legs are unchanged. Set it to false to fall back to host-staged halos, e.g. if a container's MPI lacks CUDA support (MPI then aborts rather than silently staging).
On Derecho (Cray MPICH) the namelist change alone gives bit-identical output at 30km on 4 GPUs and 2.2-2.7x faster time stepping.