Skip to content

Add gnu-mphipcc CIME compiler to the HIP image, and build-test it in CI - #125

Merged
bartgol merged 1 commit into
bartgol/vibrant-goodall-v7s4lrfrom
bartgol/hip-cime-compiler
Oct 28, 2026
Merged

bartgol merged 1 commit into
bartgol/vibrant-goodall-v7s4lrfrom
bartgol/hip-cime-compiler

Conversation

@bartgol

@bartgol bartgol commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #124 (base is its branch; retarget to main once #124 merges). Gives the HIP image a CIME compiler, gnu-mphipcc, and adds the CI that builds E3SM with it.

What changes

  • Compiler gnu-mphipcc: GNU Fortran and C, hipcc for C++, the name E3SM's craygnu-mphipcc uses on Frontier. The HIP image's CIME machine now lists gnu-mphipcc,gnu (the first is the default), and the CPU gnu stays available.
  • Macro file ghci/env/cime/e3sm-container_gnu-mphipcc.cmake: E3SM has no generic gnu-mphipcc.cmake, so it starts from E3SM's gnu.cmake, included from E3SM's source tree through ${SRCROOT} (as Macros.cmake does for its post-process file). CIME copies only the macro files a case names into the case, and follows the includes of the ones it copies from E3SM, not of the ones from ~/.cime, so a plain ${CMAKE_CURRENT_LIST_DIR}/gnu.cmake include is not found. It then adds the Kokkos HIP options (modelled on Frontier's), for MI200 by default; E3SM_KOKKOS_HIP_ARCH overrides it, e.g. AMD_GFX942, and the hipcc target follows from it.
  • Machine XML: MPICH_CXX=hipcc and BLA_VENDOR for the compiler, and 2 MPI tasks per node as for gnugpu.
  • CI: e3sm-case gets the HIP image (x86_64), build only like CUDA's since hosted runners have no GPU. It builds the wcycl and aqp cases with gnu-mphipcc. The EAMxx standalone tests are skipped, as for CUDA.

Checked / not checked

  • The first e3sm-case run failed at CMake configure because of that gnu.cmake include; fixed above. A local cmake run of the macro file with only it in ~/.cime and gnu.cmake/universal.cmake read from an E3SM checkout gives the expected SCXX, KOKKOS_OPTIONS, flags and arch override.
  • Not checked yet: that E3SM compiles and links with it. That is what this PR's e3sm-case HIP jobs are for, so expect more iteration. They are not part of required-check, like the rest of e3sm-case.
  • No GPU in CI, so nothing runs on a device.
  • E3SM's own ghci machine entry is not touched; making E3SM's CI use the HIP image would be a separate PR in E3SM, with the same compiler name.

🤖 Generated with Claude Code

https://claude.ai/code/session_019UTqLci8P681gR1erTA488

@bartgol
bartgol requested a review from mahf708 October 28, 2026 05:03
@bartgol
bartgol force-pushed the bartgol/hip-cime-compiler branch from 01f9700 to 52d5f23 Compare October 28, 2026 05:33
@bartgol
bartgol force-pushed the bartgol/hip-cime-compiler branch from 52d5f23 to 56e0552 Compare October 28, 2026 05:47
mahf708
mahf708 previously approved these changes Oct 28, 2026
@mahf708

mahf708 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

just one tiny note ... is there a reason we name them differently?

  • ghci/env/cime/e3sm-container_gnu-mphipcc.cmake
  • ghci/env/cime/e3sm-container_gnugpu.cmake
    <MAX_MPITASKS_PER_NODE compiler="gnugpu">2</MAX_MPITASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE compiler="gnu-mphipcc">2</MAX_MPITASKS_PER_NODE>

why don't we use gnumphipcc or gnu-gpu?

i never followed the compiler naming stuff in e3sm, so i may be missing something

@mahf708

mahf708 commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

tho, tbc, I don't mind if you merge this without explaining the compiler stuff 😆 approved ✅ 🚀 🌕

@bartgol

bartgol commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

just one tiny note ... is there a reason we name them differently?

  • ghci/env/cime/e3sm-container_gnu-mphipcc.cmake
  • ghci/env/cime/e3sm-container_gnugpu.cmake
    <MAX_MPITASKS_PER_NODE compiler="gnugpu">2</MAX_MPITASKS_PER_NODE>
    <MAX_MPITASKS_PER_NODE compiler="gnu-mphipcc">2</MAX_MPITASKS_PER_NODE>

why don't we use gnumphipcc or gnu-gpu?

i never followed the compiler naming stuff in e3sm, so i may be missing something

I actually had a convo with Mr C on this. The main rationale is that this is the way E3SM named the compilers on frontier, which is the main machine where we use HIP. More broadly though, I do NOT like the name gnugpu, as it does not answer "what kind of GPU?"... But in general, we only have ONE type of GPU on each HPC machine, so gnugpu means gnu+"whatever GPU backend makes sense on this machine". Unfortunately, the ghci machine in E3SM is an "umbrella" machine: we technically have 3 diff machines (one per compiler) but I put everything in one machine to avoid code duplication.

Ideally, I would like the compiler name to be <hostcompiler>-<gpu-backend> (where the second part is omitted for host-only compilers). So for us: gnu, intel, gnu-cuda, gnu-hip (and, for intel gpus, intel-sycl). Alas, I did not pick the e3sm names. I think Claude's idea of sticking with the same pattern as used in E3SM is probably the best one.

@bartgol
bartgol marked this pull request as ready for review October 28, 2026 15:41
Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

3 open issues
What changed in this PR

Adds HIP (ROCm) support to the E3SM container CIME configuration and CI matrix, introducing a dedicated gnu-mphipcc compiler setup and updating documentation/comments accordingly.

Changes:

  • Add a new CIME compiler macro file for GNU + hipcc (gnu-mphipcc) and wire it into machine config.
  • Extend Dockerfile compiler selection and GitHub Actions matrix to include HIP images/jobs.
  • Update README/test script/workflow comments to reflect HIP behavior (build-only in CI).
File Description
ghci/​tests/​e3sm-case.sh Updates header comment to include HIP in build-only CI behavior.
ghci/​env/​cime/​e3sm-container_gnu-mphipcc.cmake Adds new CIME compiler macro for HIP builds (GNU + hipcc, arch selection, flags, linking).
ghci/​env/​cime/​config_machines.xml Registers gnu-mphipcc limits and sets MPICH_CXX=hipcc for that compiler.
ghci/​env/​Dockerfile Selects gnu-mphipcc compiler variant when building HIP images.
ghci/​README.md Documents HIP default compiler and HIP arch override behavior.
.github/​workflows/​e3sm-case.yaml Adds HIP env to CI matrix; marks HIP as build-only like CUDA.

Review effort: Lite

Comment thread ghci/env/Dockerfile Resolved
The HIP image's CIME machine now lists gnu-mphipcc (GNU Fortran and C, hipcc for
C++, the name E3SM's craygnu-mphipcc uses on Frontier) ahead of gnu. Its macro file
includes E3SM's own gnu.cmake from E3SM's source tree (CIME copies only the macro files
a case names, and follows includes only for those it copies from E3SM, not from
~/.cime), and adds the Kokkos HIP options for MI200 (E3SM_KOKKOS_HIP_ARCH overrides
the arch).

e3sm-case builds the two CIME cases on the HIP image, build only like CUDA's, since
hosted runners have no GPU.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UTqLci8P681gR1erTA488
@bartgol
bartgol added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit d021546 Oct 28, 2026
60 checks passed
@bartgol
bartgol deleted the bartgol/hip-cime-compiler branch October 28, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants