From 4e30bbabf8050cd6b27fc9e66fdd7fbd8d1d1284 Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Wed, 21 Oct 2026 12:32:47 -0400 Subject: [PATCH 1/7] GMAO: add -T template option to batch_install.sh, discover-gmao packages fix, README updates --- configs/sites/tier1/discover/README.md | 7 +- configs/sites/tier1/nas-toss5/README.md | 86 +++++++++++++++++++ configs/sites/tier1/nas/README.md | 86 +++++++++++++++++++ configs/sites/tier2/bucy/README.md | 70 +++++++++++++-- configs/sites/tier2/discover-gmao/README.md | 1 + .../sites/tier2/discover-gmao/packages.yaml | 9 ++ util/gmao/README.md | 6 +- util/gmao/batch_install.sh | 15 +++- 8 files changed, 267 insertions(+), 13 deletions(-) diff --git a/configs/sites/tier1/discover/README.md b/configs/sites/tier1/discover/README.md index 1f8b2fb571..b421ae1b8a 100644 --- a/configs/sites/tier1/discover/README.md +++ b/configs/sites/tier1/discover/README.md @@ -293,10 +293,10 @@ The script creates one environment per compiler under `envs/` (relative to the r | Mode | Environment names created | |---|---| -| `build` | `ue-gcc-14.2.0-build`, `ue-gcc-15.2.0-build`, `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build` | -| `install` | `ue-gcc-14.2.0`, `ue-gcc-15.2.0`, `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0` | +| `build` | `ue-gcc-14.2.0-build`, `ue-gcc-15.2.0-build`, `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0-build`) | +| `install` | `ue-gcc-14.2.0`, `ue-gcc-15.2.0`, `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0`) | -The `ue-` prefix comes from the `unified-dev` template used by this site. +The `ue-` prefix comes from the `unified-dev` template and `ge-` from `geos-dev`. Both templates are built by default on this site; use `-T unified-dev` or `-T geos-dev` to build only one. --- @@ -477,6 +477,7 @@ The following flags are less commonly used. The descriptions below reflect our b | `-p` / `--partition` | Override the SLURM partition. Defaults to `preops` when account is `s1873`, otherwise unset. | | `-q` / `--qos` | Override the SLURM QOS. Defaults to `benchmark` when account is `s1873`, otherwise unset. | | `--constraint` | Override the SLURM node constraint. Defaults to `mil` (Milan nodes). No short form (`-c`/`-C` are taken). | +| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates (`unified-dev` and `geos-dev` on `discover`). | | `--datestamp` | Append `-YYYYMMDD` to environment names. Off by default. | | `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to newly created environments. Expensive; off by default. | | `--suffix=LABEL` | Append `-LABEL` to environment names. Off by default. | diff --git a/configs/sites/tier1/nas-toss5/README.md b/configs/sites/tier1/nas-toss5/README.md index 30f1b5d704..d8403b7fe2 100644 --- a/configs/sites/tier1/nas-toss5/README.md +++ b/configs/sites/tier1/nas-toss5/README.md @@ -12,6 +12,13 @@ This guide documents how to build **spack-stack** on NASA NAS TOSS5 systems. Use - [Obtain an Interactive Compute Node](#obtain-an-interactive-compute-node) - [Setup spack-stack](#setup-spack-stack) - [Using batch_install.sh (recommended)](#using-batch_installsh-recommended) + - [Roles and modes](#roles-and-modes) + - [The -H flag](#the--h-flag) + - [The -e flag](#the--e-flag) + - [The --datestamp and --suffix flags](#the---datestamp-and---suffix-flags) + - [Environment names](#environment-names) + - [Loading the stack](#loading-the-stack) + - [Additional options reference](#additional-options-reference) - [Create Environments](#create-environments) - [oneAPI - ifx Environment](#oneapi---ifx-environment) - [oneAPI - ifort Environment](#oneapi---ifort-environment) @@ -127,6 +134,85 @@ The `-p`, `-q`, and `--constraint` options are Discover-specific and do not appl --- +### Roles and modes + +The script requires two key flags on every invocation: + +- **`-r ROLE`** — either `dev` or `ops` +- **`-m MODE`** — either `build` or `install` + +| Role + Mode | What it does | +|---|---| +| `-r dev -m build` | For SI Team developers with write access to the shared mirrors. Builds packages from source and writes results to the **default shared build cache**. | +| `-r dev -m install` | Installs packages from the shared build cache (no compilation) and generates module files. | +| `-r ops -m build` | For operational partners **without** write access to the shared mirrors. Builds packages and writes to a **custom build cache** specified with `-c BUILDCACHE_DIR`. Allows partners to maintain their own binary cache independently. | +| `-r ops -m install` | Installs packages from the shared build cache and generates module files. | + +The `-m install` step is **required** to generate usable module files. A `-m build` run alone does not produce modules. + +--- + +### The -H flag + +Always pass `-H nas-toss5` explicitly on NAS TOSS5. Without `-H`, the script autodetects the site from `$HOSTNAME`; explicit `-H` avoids any ambiguity. + +--- + +### The -e flag + +The `-e` flag tells the script to continue even if the environment directories already exist. Without it, the script exits with an error if any target environment directory is present. + +- On the **very first run**, you do not need `-e` (the directories do not exist yet). +- On **every subsequent run** (rebuilds, retries, installs), you must pass `-e`. + +--- + +### Environment names + +The script creates one environment per compiler under `envs/` (relative to the repository root, unless overridden with `-d`): + +| Mode | Environment names created | +|---|---| +| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.3.0-build` | +| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.3.0` | + +The `ue-` prefix comes from the `unified-dev` template used by this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates. + +--- + +### Loading the stack + +After the `install` step completes, point your shell to the generated module files: + +```bash +module use -a /path/to/spack-stack/envs/ue-oneapi-2024.2.0/install/modulefiles/Core +module load stack-intel-oneapi-compilers stack-intel-oneapi-mpi +``` + +Replace `ue-oneapi-2024.2.0` with the appropriate environment name for your compiler of choice. + +--- + +### Additional options reference + +| Flag | Description | +|---|---| +| `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build`. | +| `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). | +| `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `oneapi@=2024.2.0,gcc@=15.3.0`). Overrides the site defaults. | +| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `unified-dev`). Overrides the host's default templates. | +| `-a ACCOUNT` | Override the PBS account (default: `s1873`). | +| `-o` / `--concretize-only` | Concretize environments and stop before installation. | +| `-n` | Dry run: print what would be executed. | +| `-L` / `--disable-locks` | Disable Spack install locks and serialize package installs for an isolated recovery run only. | +| `-t` | Run tests for specific third-party dependencies after installation (list hardcoded in `batch_install.sh`). | +| `-u` | Populate bootstrap, source, and cargo mirrors. Requires `-r dev -m build`. First run or cache refresh only. | +| `--datestamp` | Append `-YYYYMMDD` to environment names. Off by default. | +| `--suffix=LABEL` | Append `-LABEL` to environment names. Off by default. | +| `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to newly created environments. Expensive; off by default. | + +--- + ### The --datestamp and --suffix flags By default, environments are named `--` (e.g. `ge-gcc-15.3.0`). To build an environment that others may be using and that you are less likely to remove by accident, tag its name: diff --git a/configs/sites/tier1/nas/README.md b/configs/sites/tier1/nas/README.md index 5035a5dd63..0081f2fa45 100644 --- a/configs/sites/tier1/nas/README.md +++ b/configs/sites/tier1/nas/README.md @@ -12,6 +12,13 @@ This guide documents how to build **spack-stack** on NASA NAS systems. Use `util - [Obtain an Interactive Compute Node](#obtain-an-interactive-compute-node) - [Setup spack-stack](#setup-spack-stack) - [Using batch_install.sh (recommended)](#using-batch_installsh-recommended) + - [Roles and modes](#roles-and-modes) + - [The -H flag](#the--h-flag) + - [The -e flag](#the--e-flag) + - [The --datestamp and --suffix flags](#the---datestamp-and---suffix-flags) + - [Environment names](#environment-names) + - [Loading the stack](#loading-the-stack) + - [Additional options reference](#additional-options-reference) - [Create Environments](#create-environments) - [oneAPI - ifx Environment](#oneapi---ifx-environment) - [oneAPI - ifort Environment](#oneapi---ifort-environment) @@ -129,6 +136,85 @@ The `-p`, `-q`, and `--constraint` options are Discover-specific and do not appl --- +### Roles and modes + +The script requires two key flags on every invocation: + +- **`-r ROLE`** — either `dev` or `ops` +- **`-m MODE`** — either `build` or `install` + +| Role + Mode | What it does | +|---|---| +| `-r dev -m build` | For SI Team developers with write access to the shared mirrors. Builds packages from source and writes results to the **default shared build cache**. | +| `-r dev -m install` | Installs packages from the shared build cache (no compilation) and generates module files. | +| `-r ops -m build` | For operational partners **without** write access to the shared mirrors. Builds packages and writes to a **custom build cache** specified with `-c BUILDCACHE_DIR`. Allows partners to maintain their own binary cache independently. | +| `-r ops -m install` | Installs packages from the shared build cache and generates module files. | + +The `-m install` step is **required** to generate usable module files. A `-m build` run alone does not produce modules. + +--- + +### The -H flag + +Always pass `-H nas` explicitly on NAS. Without `-H`, the script autodetects the site from `$HOSTNAME`; explicit `-H` avoids any ambiguity. + +--- + +### The -e flag + +The `-e` flag tells the script to continue even if the environment directories already exist. Without it, the script exits with an error if any target environment directory is present. + +- On the **very first run**, you do not need `-e` (the directories do not exist yet). +- On **every subsequent run** (rebuilds, retries, installs), you must pass `-e`. + +--- + +### Environment names + +The script creates one environment per compiler under `envs/` (relative to the repository root, unless overridden with `-d`): + +| Mode | Environment names created | +|---|---| +| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.2.0-build` | +| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.2.0` | + +The `ue-` prefix comes from the `unified-dev` template used by this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates. + +--- + +### Loading the stack + +After the `install` step completes, point your shell to the generated module files: + +```bash +module use -a /path/to/spack-stack/envs/ue-oneapi-2024.2.0/install/modulefiles/Core +module load stack-intel-oneapi-compilers stack-intel-oneapi-mpi +``` + +Replace `ue-oneapi-2024.2.0` with the appropriate environment name for your compiler of choice. + +--- + +### Additional options reference + +| Flag | Description | +|---|---| +| `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build`. | +| `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). | +| `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `oneapi@=2024.2.0,gcc@=15.2.0`). Overrides the site defaults. | +| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `unified-dev`). Overrides the host's default templates. | +| `-a ACCOUNT` | Override the PBS account (default: `s1873`). | +| `-o` / `--concretize-only` | Concretize environments and stop before installation. | +| `-n` | Dry run: print what would be executed. | +| `-L` / `--disable-locks` | Disable Spack install locks and serialize package installs for an isolated recovery run only. | +| `-t` | Run tests for specific third-party dependencies after installation (list hardcoded in `batch_install.sh`). | +| `-u` | Populate bootstrap, source, and cargo mirrors. Requires `-r dev -m build`. First run or cache refresh only. | +| `--datestamp` | Append `-YYYYMMDD` to environment names. Off by default. | +| `--suffix=LABEL` | Append `-LABEL` to environment names. Off by default. | +| `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to newly created environments. Expensive; off by default. | + +--- + ### The --datestamp and --suffix flags By default, environments are named `--` (e.g. `ge-gcc-15.3.0`). To build an environment that others may be using and that you are less likely to remove by accident, tag its name: diff --git a/configs/sites/tier2/bucy/README.md b/configs/sites/tier2/bucy/README.md index bcfc3bbfdb..3cfe18a311 100644 --- a/configs/sites/tier2/bucy/README.md +++ b/configs/sites/tier2/bucy/README.md @@ -23,12 +23,75 @@ the site NAG module that sets it: export NAG_KUSARI_FILE=/ford1/share/gmao_SIteam/nag/nag.key ``` -For a direct local build, use: +--- + +## Using batch_install.sh script + +On Bucy the `batch_install.sh` script runs in its one-step `local` mode: it builds directly into the final environment, generates modules, and writes to the shared build cache. There is no batch scheduler on Bucy, so `-s`, `-r`, `-p`, `-q`, `--constraint`, and `-a` do not apply. + +### Help and dry-run + +```bash +./util/gmao/batch_install.sh -h +``` + +Preview what the script would do without making any changes: + +```bash +./util/gmao/batch_install.sh -m local -H bucy -n +``` + +### The -H flag + +Always pass `-H bucy` explicitly. Without it, the script autodetects the site from `$HOSTNAME`; explicit `-H` avoids any ambiguity. + +### First run + +On the very first run, use `-u` to populate the bootstrap, source, and cargo mirrors: ```bash ./util/gmao/batch_install.sh -m local -H bucy -u ``` +### The -e flag + +On every subsequent run (rebuilds, retries), pass `-e` so the script continues in the existing environment directories instead of exiting with an error: + +```bash +./util/gmao/batch_install.sh -m local -H bucy -e +``` + +### Selecting compilers and templates + +Use `-C` to build only some compilers and `-T` to build only some templates (both comma-separated). By default all compilers in the matrix above are built with the `geos-dev` and `geos-dev-nag` templates (`geos-dev-nag` only applies to NAG): + +```bash +./util/gmao/batch_install.sh -m local -H bucy -e -C gcc@=15.2.0,oneapi@=2024.2.0 -T geos-dev +``` + +### Other useful flags + +- `-d ENV_DIRS`: override the environment directory (default: `envs/` in the repo root). +- `-c BUILDCACHE_DIR`: use a custom build cache instead of the shared one. +- `-o` / `--concretize-only`: stop after concretization. +- `-L` / `--disable-locks`: disable Spack install locks and serialize package installs; only for an isolated recovery run when no other process modifies the install tree. +- `-t`: run tests for specific third-party dependencies after installation. +- `--datestamp` / `--suffix=LABEL`: see below. +- `--with-ai`: add `ai-env` (py-torch + `jedi-base-env`) to newly created environments; expensive, off by default. + +### Environment names + +Environments are created directly under `envs/` (not `envs/bucy`), one per compiler, named `--`, e.g. `ge-gcc-15.2.0`, `ge-oneapi-2024.2.0`, `ge-nag-7.2.43`. The `ge-` prefix comes from the `geos-dev` and `geos-dev-nag` templates. + +To keep an environment others may be using from being replaced, tag its name +with `--datestamp` (appends `-YYYYMMDD`) and/or `--suffix=LABEL` (appends +`-LABEL`), e.g. `./util/gmao/batch_install.sh -m local -H bucy --datestamp --suffix=swelltest` +gives `ge-gcc-15.2.0-YYYYMMDD-swelltest`. Both are off by default. Use the same options together with `-e` to continue a tagged environment. + +--- + +## Site configuration notes + The shared installation, source, bootstrap, and cache locations are under `/ford1/share/gmao_SIteam/spack-stack`. Tcl module generation is intentionally not enabled in this first pass; it can be added after the Lmod hierarchy is @@ -45,8 +108,3 @@ gmake, and so on) as externals, and forces the system `tar`, because `tar@1.35` fails to build against the system libacl headers. Environments are created directly under `envs/` (not `envs/bucy`). - -To keep an environment others may be using from being replaced, tag its name -with `--datestamp` (appends `-YYYYMMDD`) and/or `--suffix=LABEL` (appends -`-LABEL`), e.g. `./util/gmao/batch_install.sh -m local -H bucy --datestamp --suffix=swelltest` -gives `ge-gcc-15.2.0-YYYYMMDD-swelltest`. Both are off by default. diff --git a/configs/sites/tier2/discover-gmao/README.md b/configs/sites/tier2/discover-gmao/README.md index e5c26c9346..fb87681f9c 100644 --- a/configs/sites/tier2/discover-gmao/README.md +++ b/configs/sites/tier2/discover-gmao/README.md @@ -460,6 +460,7 @@ The following flags are less commonly used. The descriptions below reflect our b | `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build` for operational partners who do not have write access to the shared mirrors. | | `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). | | `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `gcc@=15.2.0,oneapi@=2025.3.0`). Overrides the site defaults. | +| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates. | | `-p` / `--partition` | Override the SLURM partition. Defaults to `preops` when account is `s1873`, otherwise unset. | | `-q` / `--qos` | Override the SLURM QOS. Defaults to `benchmark` when account is `s1873`, otherwise unset. | | `--constraint` | Override the SLURM node constraint. Defaults to `mil` (Milan nodes). No short form (`-c`/`-C` are taken). | diff --git a/configs/sites/tier2/discover-gmao/packages.yaml b/configs/sites/tier2/discover-gmao/packages.yaml index 56d60208dc..6cdbc101ce 100644 --- a/configs/sites/tier2/discover-gmao/packages.yaml +++ b/configs/sites/tier2/discover-gmao/packages.yaml @@ -1,4 +1,6 @@ packages: + all: + target: [x86_64_v3] ### Modification of common packages ewok-env: require: @@ -26,6 +28,13 @@ packages: variants: ~shared met: variants: +python +grib2 +graphics +lidar2nc +modis + # mimalloc is vendored and downloaded via FetchContent; fails on air-gapped compute nodes + arrow: + require: + - '~mimalloc' + gdal: + require: + - '~arrow' ### All other external packages listed alphabetically autoconf: externals: diff --git a/util/gmao/README.md b/util/gmao/README.md index 1e03b5b008..3e27fc78b9 100644 --- a/util/gmao/README.md +++ b/util/gmao/README.md @@ -23,7 +23,7 @@ The primary script for building and installing spack-stack environments. It auto ### Usage ``` -batch_install.sh -r -m [-d ] [-c ] [-H ] +batch_install.sh -r -m [-d ] [-c ] [-C ] [-T ] [-H ] ``` Run with `-h` for full usage: @@ -37,7 +37,7 @@ Run with `-h` for full usage: | Flag | Description | |---|---| | `-r ROLE` | Role: `dev` (SI Team developers with write access to shared mirrors, writes to default build cache) or `ops` (operational partners without write access, requires `-c` to specify their own build cache). | -| `-m MODE` | Mode: `build` (compile + populate build cache) or `install` (install from cache + generate modules). | +| `-m MODE` | Mode: `build` (compile + populate build cache), `install` (install from cache + generate modules), or `local` (one-step per-machine install and cache update; `macos.gmao` and `bucy`; `-r` not required). | | `-H HOSTNAME` | Override hostname autodetection. Usually required (e.g. `-H discover-gmao`, `-H macos.gmao`). | | `-s` | Submit the install step to the batch scheduler (SLURM/PBS) instead of running interactively. | | `-e` | Continue if environment directories already exist. Required on all runs after the first. | @@ -51,6 +51,8 @@ Run with `-h` for full usage: | `-c BUILDCACHE_DIR` | Custom build cache path. Required with `-r ops -m build`. | | `-d ENV_DIRS` | Override the default environment directory. | | `-C COMPILERS` | Comma-separated compiler list, overrides site defaults. | +| `-T TEMPLATES` | Comma-separated template list (e.g. `geos-dev` or `unified-dev,geos-dev`), overrides the host's default templates. | +| `-L` / `--disable-locks` | Disable Spack install locks and serialize package installs. Use only when this is the sole process modifying the install tree. | | `-N NAGFOR_PATH` | Path to `nagfor` executable for NAG compiler detection (macOS only). | | `-t` | Run tests for hardcoded third-party packages after installation. | | `--datestamp` | Append `-YYYYMMDD` (today) to environment names, e.g. `ge-gcc-15.3.0-20261007`. Off by default. | diff --git a/util/gmao/batch_install.sh b/util/gmao/batch_install.sh index 85da989d60..dcf58c9fee 100755 --- a/util/gmao/batch_install.sh +++ b/util/gmao/batch_install.sh @@ -98,6 +98,8 @@ usage() { echo " by default, exit with an error if already exist" echo " -C Set a comma-separated list of compilers to use (e.g. gcc@=15.3.0,nag@=7.2.7243);" echo " overrides the default compilers for the site" + echo " -T Set a comma-separated list of templates to use (e.g. geos-dev or unified-dev,geos-dev);" + echo " overrides the default templates for the host" echo " -N Path to nagfor executable (e.g. /opt/nag/bin/nagfor);" echo " forces NAG stack to be built using this specific compiler" echo " -a Set PBS/SLURM account (default: s1873);" @@ -214,7 +216,7 @@ while [[ $# -gt 0 ]]; do done set -- "${normalized_args[@]}" -while getopts r:m:d:c:C:N:H:a:p:q:nouestLh flag +while getopts r:m:d:c:C:T:N:H:a:p:q:nouestLh flag do case "${flag}" in r) @@ -232,6 +234,9 @@ do C) SPACK_STACK_COMPILER_OPT=${OPTARG} ;; + T) + SPACK_STACK_TEMPLATE_OPT=${OPTARG} + ;; N) SPACK_STACK_NAGFOR_PATH=${OPTARG} ;; @@ -282,6 +287,7 @@ echo " SPACK_STACK_MODE: ${SPACK_STACK_MODE:-not set echo " SPACK_STACK_ENVIRONMENT_DIRS: ${SPACK_STACK_ENVIRONMENT_DIRS:-${SPACK_STACK_DIR}/envs}" echo " SPACK_STACK_BUILDCACHE_DIR: ${SPACK_STACK_BUILDCACHE_DIR:-use default caches}" echo " SPACK_STACK_BATCH_HOST_OPT: ${SPACK_STACK_BATCH_HOST_OPT:-autodetect}" +echo " SPACK_STACK_TEMPLATE_OPT: ${SPACK_STACK_TEMPLATE_OPT:-use host default}" echo " SPACK_STACK_ENV_SUFFIX: ${SPACK_STACK_ENV_SUFFIX:-none}" echo " SPACK_STACK_ENV_DATESTAMP: ${SPACK_STACK_ENV_DATESTAMP:-false}" echo " SPACK_STACK_WITH_AI: ${SPACK_STACK_WITH_AI:-false}" @@ -361,7 +367,7 @@ case ${SPACK_STACK_BATCH_HOST} in ;; discover) SPACK_STACK_BATCH_COMPILERS=("oneapi@=2024.2.0" "oneapi@=2025.3.0" "gcc@=14.2.0" "gcc@=15.2.0") - SPACK_STACK_BATCH_TEMPLATES=("unified-dev") + SPACK_STACK_BATCH_TEMPLATES=("unified-dev" "geos-dev") SPACK_STACK_MODULE_CHOICE="lmod" SPACK_STACK_BOOTSTRAP_MIRROR="/discover/nobackup/projects/gmao/SIteam/spack-stack/bootstrap-mirror" SPACK_STACK_CARGO_MIRROR="/discover/nobackup/projects/gmao/SIteam/spack-stack/cargo-mirror" @@ -439,6 +445,11 @@ if [[ -n "${SPACK_STACK_COMPILER_OPT}" ]]; then IFS=',' read -r -a SPACK_STACK_BATCH_COMPILERS <<< "${SPACK_STACK_COMPILER_OPT}" fi +# Apply -T template override for all hosts +if [[ -n "${SPACK_STACK_TEMPLATE_OPT}" ]]; then + IFS=',' read -r -a SPACK_STACK_BATCH_TEMPLATES <<< "${SPACK_STACK_TEMPLATE_OPT}" +fi + ################################################################################################## function fix_permissions() { From ddf2ec7bd4f89d8f3d335083c427cfb46ef32c68 Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Mon, 26 Oct 2026 08:43:43 -0400 Subject: [PATCH 2/7] Add geos-dev at NAS --- configs/sites/tier1/nas-toss5/README.md | 8 ++++---- configs/sites/tier1/nas/README.md | 8 ++++---- util/gmao/batch_install.sh | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/configs/sites/tier1/nas-toss5/README.md b/configs/sites/tier1/nas-toss5/README.md index d8403b7fe2..3ca8d45b48 100644 --- a/configs/sites/tier1/nas-toss5/README.md +++ b/configs/sites/tier1/nas-toss5/README.md @@ -173,10 +173,10 @@ The script creates one environment per compiler under `envs/` (relative to the r | Mode | Environment names created | |---|---| -| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.3.0-build` | -| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.3.0` | +| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.3.0-build`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0-build`) | +| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.3.0`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0`) | -The `ue-` prefix comes from the `unified-dev` template used by this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates. +The `ue-` prefix comes from the `unified-dev` template and `ge-` from `geos-dev`; both are built by default on this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates. --- @@ -200,7 +200,7 @@ Replace `ue-oneapi-2024.2.0` with the appropriate environment name for your comp | `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build`. | | `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). | | `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `oneapi@=2024.2.0,gcc@=15.3.0`). Overrides the site defaults. | -| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `unified-dev`). Overrides the host's default templates. | +| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates (`unified-dev` and `geos-dev`). | | `-a ACCOUNT` | Override the PBS account (default: `s1873`). | | `-o` / `--concretize-only` | Concretize environments and stop before installation. | | `-n` | Dry run: print what would be executed. | diff --git a/configs/sites/tier1/nas/README.md b/configs/sites/tier1/nas/README.md index 0081f2fa45..2bd79f073f 100644 --- a/configs/sites/tier1/nas/README.md +++ b/configs/sites/tier1/nas/README.md @@ -175,10 +175,10 @@ The script creates one environment per compiler under `envs/` (relative to the r | Mode | Environment names created | |---|---| -| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.2.0-build` | -| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.2.0` | +| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.2.0-build`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0-build`) | +| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.2.0`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0`) | -The `ue-` prefix comes from the `unified-dev` template used by this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates. +The `ue-` prefix comes from the `unified-dev` template and `ge-` from `geos-dev`; both are built by default on this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates. --- @@ -202,7 +202,7 @@ Replace `ue-oneapi-2024.2.0` with the appropriate environment name for your comp | `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build`. | | `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). | | `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `oneapi@=2024.2.0,gcc@=15.2.0`). Overrides the site defaults. | -| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `unified-dev`). Overrides the host's default templates. | +| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates (`unified-dev` and `geos-dev`). | | `-a ACCOUNT` | Override the PBS account (default: `s1873`). | | `-o` / `--concretize-only` | Concretize environments and stop before installation. | | `-n` | Dry run: print what would be executed. | diff --git a/util/gmao/batch_install.sh b/util/gmao/batch_install.sh index dcf58c9fee..bdc9200e7c 100755 --- a/util/gmao/batch_install.sh +++ b/util/gmao/batch_install.sh @@ -351,7 +351,7 @@ fi case ${SPACK_STACK_BATCH_HOST} in nas) SPACK_STACK_BATCH_COMPILERS=("oneapi@=2024.2.0" "oneapi@=2025.3.0" "gcc@=15.2.0") - SPACK_STACK_BATCH_TEMPLATES=("unified-dev") + SPACK_STACK_BATCH_TEMPLATES=("unified-dev" "geos-dev") SPACK_STACK_MODULE_CHOICE="tcl" SPACK_STACK_BOOTSTRAP_MIRROR="/swbuild/gmao_SIteam/spack-stack/bootstrap-mirror-toss4" SPACK_STACK_CARGO_MIRROR="/swbuild/gmao_SIteam/spack-stack/cargo-mirror" @@ -359,7 +359,7 @@ case ${SPACK_STACK_BATCH_HOST} in ;; nas-toss5) SPACK_STACK_BATCH_COMPILERS=("oneapi@=2024.2.0" "oneapi@=2025.3.0" "gcc@=15.3.0") - SPACK_STACK_BATCH_TEMPLATES=("unified-dev") + SPACK_STACK_BATCH_TEMPLATES=("unified-dev" "geos-dev") SPACK_STACK_MODULE_CHOICE="tcl" SPACK_STACK_BOOTSTRAP_MIRROR="/swbuild/gmao_SIteam/spack-stack/bootstrap-mirror-toss5" SPACK_STACK_CARGO_MIRROR="/swbuild/gmao_SIteam/spack-stack/cargo-mirror" From 39ab699beb527918ac74a6736e171880c0edaba4 Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Mon, 26 Oct 2026 11:29:09 -0400 Subject: [PATCH 3/7] TMPDIR fix on discover --- util/gmao/batch_install.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/util/gmao/batch_install.sh b/util/gmao/batch_install.sh index bdc9200e7c..fd75f02e8b 100755 --- a/util/gmao/batch_install.sh +++ b/util/gmao/batch_install.sh @@ -1389,6 +1389,18 @@ fi export SPACK_CARGO_HOME=${cargo_mirror_path} export CARGO_NET_OFFLINE=true +# Discover's /tmp may disallow execution, which breaks configure checks such +# as FFmpeg's temporary shell script. Use a private directory per install job +# on the shared filesystem, independently of Spack's build stage setting. +if [[ "${host}" == "discover" || "${host}" == "discover-gmao" ]]; then + build_tmpdir_root=/discover/nobackup/projects/gmao/SIteam/spack-stack/build-tmpdir + mkdir -p "\${build_tmpdir_root}" + build_tmpdir=\$(mktemp -d "\${build_tmpdir_root}/\${USER}.\${SLURM_JOB_ID:-local}.XXXXXX") + export TMPDIR="\${build_tmpdir}" + trap 'rm -rf -- "\${build_tmpdir}"' EXIT + echo "INFO: using TMPDIR=\${TMPDIR}" +fi + if [[ "${SPACK_STACK_DISABLE_LOCKS}" == "true" ]]; then # Spack's package workers do not inherit command-line configuration or the # environment's config scopes. Give every worker a private user config scope From 6b76fee94966ee7d3e4d97dbadb3828b8f1591d4 Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Wed, 28 Oct 2026 09:55:39 -0400 Subject: [PATCH 4/7] util/gmao/batch_install.sh: restore private TMPDIR after module preload (OpenMPI module resets it to /tmp) --- util/gmao/batch_install.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/util/gmao/batch_install.sh b/util/gmao/batch_install.sh index fd75f02e8b..3a20c1e6bd 100755 --- a/util/gmao/batch_install.sh +++ b/util/gmao/batch_install.sh @@ -1446,6 +1446,13 @@ elif [[ "${host}" == "discover-gmao" ]] && \ module load comp/gcc/12.3.0 comp/nag/7.2-7238 mpi/openmpi/4.1.6/nag_7.2.7238-gcc_12.3.0 fi +# Some Discover MPI modules (e.g. mpi/openmpi/5.0.10/gcc-15.2.0) run +# setenv("TMPDIR", "/tmp"), so restore our private TMPDIR after module loads. +if [[ -n "\${build_tmpdir:-}" ]]; then + export TMPDIR="\${build_tmpdir}" + echo "INFO: TMPDIR after module preload: \${TMPDIR}" +fi + $(declare -p test_packages) # If no tests are required, install everything From 3ab17069297de1dceee48a6bbd3547165294c865 Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Mon, 2 Nov 2026 10:35:58 -0500 Subject: [PATCH 5/7] batch_install.sh: make shared mirrors group-writable (sispack) via fix_permissions --- repos/builtin | 2 +- util/gmao/batch_install.sh | 25 +++++++++++++++++-------- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/repos/builtin b/repos/builtin index 7e51ad8614..e03613d824 160000 --- a/repos/builtin +++ b/repos/builtin @@ -1 +1 @@ -Subproject commit 7e51ad86144fb59b25dab27931284ba71a70fe89 +Subproject commit e03613d8246aefa7f971c2b4fc8b7006c8387c9f diff --git a/util/gmao/batch_install.sh b/util/gmao/batch_install.sh index 3a20c1e6bd..2bcb3cb8d4 100755 --- a/util/gmao/batch_install.sh +++ b/util/gmao/batch_install.sh @@ -456,6 +456,7 @@ function fix_permissions() { host=$1 dir=$2 executables=$3 + group_write=${4:-0} echo "Repairing permissions for directory ${dir} on ${host} ..." set +e case ${host} in @@ -501,6 +502,14 @@ function fix_permissions() { exit 1 ;; esac + # Shared caches (mirrors) are also group-writable so that members of the + # shared group can update them; install trees never pass group_write. + if [[ ${group_write} -eq 1 && ${host} != "macos.gmao" ]]; then + shared_group=${SPACK_STACK_SHARED_GROUP:-sispack} + nice -n 19 find ${dir} ! -group ${shared_group} -print0 | xargs --null --no-run-if-empty chgrp ${shared_group} + nice -n 19 find ${dir} -type d -print0 | xargs --null chmod g+rwxs + nice -n 19 find ${dir} -type f -print0 | xargs --null chmod g+rw + fi set -e } @@ -989,20 +998,20 @@ for compiler in "${SPACK_STACK_BATCH_COMPILERS[@]}"; do if [[ "${update_build_cache}" == "true" ]]; then echo "[DRY-RUN] spack buildcache push -u " echo "[DRY-RUN] spack buildcache update-index local-binary" - echo "[DRY-RUN] fix_permissions ${host} 0" + echo "[DRY-RUN] fix_permissions ${host} 0 1" elif [[ "${publish_build_cache}" == "true" ]]; then echo "[DRY-RUN] spack buildcache push -u " echo "[DRY-RUN] spack buildcache update-index local-binary" - echo "[DRY-RUN] fix_permissions ${host} 0" + echo "[DRY-RUN] fix_permissions ${host} 0 1" else echo "[DRY-RUN] spack module ${module_choice} refresh --yes --upstream-modules" echo "[DRY-RUN] spack stack setup-meta-modules" fi if [[ "${update_source_cache}" == "true" ]]; then - echo "[DRY-RUN] fix_permissions ${host} 0" + echo "[DRY-RUN] fix_permissions ${host} 0 1" fi if [[ "${update_cargo_mirror}" == "true" ]]; then - echo "[DRY-RUN] fix_permissions ${host} ${cargo_mirror_path} 0" + echo "[DRY-RUN] fix_permissions ${host} ${cargo_mirror_path} 0 1" fi echo "[DRY-RUN] spack clean -d -f -m -p -s" @@ -1093,7 +1102,7 @@ for compiler in "${SPACK_STACK_BATCH_COMPILERS[@]}"; do # Update buildcache index spack buildcache update-index ${bootstrap_mirror_path}/bootstrap_cache # Fix permissions for the bootstrap mirror - fix_permissions ${host} ${bootstrap_mirror_path} 0 + fix_permissions ${host} ${bootstrap_mirror_path} 0 1 update_bootstrap_mirror="false" # When spack creates a bootstrap mirror, it populates the "spack" scope # with compilers and packages it finds, which can create problems later @@ -1504,13 +1513,13 @@ EOF # When creating or updating buildcaches, fix permissions for mirrors. # Mirrors do not contain executables, therefore skip looking for them. if [[ "${update_source_cache}" == "true" ]]; then - fix_permissions ${host} ${source_mirror_path} 0 + fix_permissions ${host} ${source_mirror_path} 0 1 fi if [[ "${publish_build_cache}" == "true" ]]; then - fix_permissions ${host} ${binary_mirror_path} 0 + fix_permissions ${host} ${binary_mirror_path} 0 1 fi if [[ "${update_cargo_mirror}" == "true" ]]; then - fix_permissions ${host} ${cargo_mirror_path} 0 + fix_permissions ${host} ${cargo_mirror_path} 0 1 fi # Clean up (don't remove software and configuration needed to bootstrap Spack) From a9cb5ab74a98df766c1a1fec5614425e404e6e9f Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Mon, 2 Nov 2026 10:38:17 -0500 Subject: [PATCH 6/7] batch_install.sh: restrict sispack group-write to discover hosts; document in READMEs --- configs/sites/tier1/discover/README.md | 2 ++ configs/sites/tier2/discover-gmao/README.md | 2 ++ util/gmao/README.md | 2 ++ util/gmao/batch_install.sh | 6 +++--- 4 files changed, 9 insertions(+), 3 deletions(-) diff --git a/configs/sites/tier1/discover/README.md b/configs/sites/tier1/discover/README.md index b421ae1b8a..e075d2df1c 100644 --- a/configs/sites/tier1/discover/README.md +++ b/configs/sites/tier1/discover/README.md @@ -252,6 +252,8 @@ On the very first run, use `-u` to populate the bootstrap, source, and cargo mir - `-s`: Submits the install step to SLURM. - No `-e` needed since the environment directories do not exist yet. +**Shared cache permissions (discover and discover-gmao only):** after updating the bootstrap, source, binary, and cargo mirrors, `batch_install.sh` makes them group-writable by the `sispack` group (`chgrp` plus `chmod g+rwXs`) so that any SI Team developer can update them. Install trees (environments) are not touched this way. Set `SPACK_STACK_SHARED_GROUP` to use a different group. This step is skipped on other hosts (NAS, bucy, macOS) where the group does not exist. + #### If one source fails while updating the cache The `-u` source-cache update mirrors many packages concurrently. If one reachable Git source fails to mirror, fetch that concrete spec serially on the login node instead of sending an offline compute-node job without its source: diff --git a/configs/sites/tier2/discover-gmao/README.md b/configs/sites/tier2/discover-gmao/README.md index fb87681f9c..0713276a41 100644 --- a/configs/sites/tier2/discover-gmao/README.md +++ b/configs/sites/tier2/discover-gmao/README.md @@ -253,6 +253,8 @@ On the very first run, use `-u` to populate the bootstrap, source, and cargo mir - `-s`: Submits the install step to SLURM. - No `-e` needed since the environment directories do not exist yet. +**Shared cache permissions (discover and discover-gmao only):** after updating the bootstrap, source, binary, and cargo mirrors, `batch_install.sh` makes them group-writable by the `sispack` group (`chgrp` plus `chmod g+rwXs`) so that any SI Team developer can update them. Install trees (environments) are not touched this way. Set `SPACK_STACK_SHARED_GROUP` to use a different group. This step is skipped on other hosts (NAS, bucy, macOS) where the group does not exist. + #### Step 2 — Rebuild or retry runs After the first run, the environment directories exist and the caches are populated. Drop `-u` and add `-e`: diff --git a/util/gmao/README.md b/util/gmao/README.md index 3e27fc78b9..0019042132 100644 --- a/util/gmao/README.md +++ b/util/gmao/README.md @@ -59,6 +59,8 @@ Run with `-h` for full usage: | `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to the root specs of newly created environments. Expensive, so off by default; not applied to existing environments (`-e`). Combine with `--suffix=ai` to keep the name distinct. | | `--suffix=LABEL` | Append `-LABEL` to environment names (after the date when combined with `--datestamp`), e.g. `ge-gcc-15.3.0-20261007-swelltest`. Off by default. | +**Shared cache permissions:** on `discover` and `discover-gmao` only, mirror updates are followed by a `chgrp`/`chmod g+rwXs` to group `sispack` (override with `SPACK_STACK_SHARED_GROUP`) so SI Team members can all update the shared caches. Install trees are not made group-writable. Skipped on NAS, bucy, and macOS. + ### Supported sites The script detects the current site from `$HOSTNAME` (stripping digits and domain), or you can override with `-H`: diff --git a/util/gmao/batch_install.sh b/util/gmao/batch_install.sh index 2bcb3cb8d4..26d3ac1be5 100755 --- a/util/gmao/batch_install.sh +++ b/util/gmao/batch_install.sh @@ -502,9 +502,9 @@ function fix_permissions() { exit 1 ;; esac - # Shared caches (mirrors) are also group-writable so that members of the - # shared group can update them; install trees never pass group_write. - if [[ ${group_write} -eq 1 && ${host} != "macos.gmao" ]]; then + # Shared caches (mirrors) are also group-writable on discover (the only host with the sispack group) + # so members can update them; install trees never pass group_write. + if [[ ${group_write} -eq 1 && ( ${host} == "discover" || ${host} == "discover-gmao" ) ]]; then shared_group=${SPACK_STACK_SHARED_GROUP:-sispack} nice -n 19 find ${dir} ! -group ${shared_group} -print0 | xargs --null --no-run-if-empty chgrp ${shared_group} nice -n 19 find ${dir} -type d -print0 | xargs --null chmod g+rwxs From 20df0e24b89b4a27d96de291e128f0a4d46d4605 Mon Sep 17 00:00:00 2001 From: Matthew Thompson Date: Tue, 3 Nov 2026 11:36:38 -0500 Subject: [PATCH 7/7] Restore repos/builtin submodule pointer to develop (accidental bump) --- repos/builtin | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/repos/builtin b/repos/builtin index e03613d824..7e51ad8614 160000 --- a/repos/builtin +++ b/repos/builtin @@ -1 +1 @@ -Subproject commit e03613d8246aefa7f971c2b4fc8b7006c8387c9f +Subproject commit 7e51ad86144fb59b25dab27931284ba71a70fe89