mathomp4 · Oct 21, 2026 · Oct 26, 2026 · Oct 26, 2026 · Oct 28, 2026 · Oct 28, 2026
diff --git a/configs/sites/tier1/discover/README.md b/configs/sites/tier1/discover/README.md
@@ -252,6 +252,8 @@ On the very first run, use `-u` to populate the bootstrap, source, and cargo mir
 - `-s`: Submits the install step to SLURM.
 - No `-e` needed since the environment directories do not exist yet.
 
+**Shared cache permissions (discover and discover-gmao only):** after updating the bootstrap, source, binary, and cargo mirrors, `batch_install.sh` makes them group-writable by the `sispack` group (`chgrp` plus `chmod g+rwXs`) so that any SI Team developer can update them. Install trees (environments) are not touched this way. Set `SPACK_STACK_SHARED_GROUP` to use a different group. This step is skipped on other hosts (NAS, bucy, macOS) where the group does not exist.
+
 #### If one source fails while updating the cache
 
 The `-u` source-cache update mirrors many packages concurrently. If one reachable Git source fails to mirror, fetch that concrete spec serially on the login node instead of sending an offline compute-node job without its source:
@@ -293,10 +295,10 @@ The script creates one environment per compiler under `envs/` (relative to the r
 
 | Mode | Environment names created |
 |---|---|
-| `build` | `ue-gcc-14.2.0-build`, `ue-gcc-15.2.0-build`, `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build` |
-| `install` | `ue-gcc-14.2.0`, `ue-gcc-15.2.0`, `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0` |
+| `build` | `ue-gcc-14.2.0-build`, `ue-gcc-15.2.0-build`, `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0-build`) |
+| `install` | `ue-gcc-14.2.0`, `ue-gcc-15.2.0`, `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0`) |
 
-The `ue-` prefix comes from the `unified-dev` template used by this site.
+The `ue-` prefix comes from the `unified-dev` template and `ge-` from `geos-dev`. Both templates are built by default on this site; use `-T unified-dev` or `-T geos-dev` to build only one.
 
 ---
 
@@ -477,6 +479,7 @@ The following flags are less commonly used. The descriptions below reflect our b
 | `-p` / `--partition` | Override the SLURM partition. Defaults to `preops` when account is `s1873`, otherwise unset. |
 | `-q` / `--qos` | Override the SLURM QOS. Defaults to `benchmark` when account is `s1873`, otherwise unset. |
 | `--constraint` | Override the SLURM node constraint. Defaults to `mil` (Milan nodes). No short form (`-c`/`-C` are taken). |
+| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates (`unified-dev` and `geos-dev` on `discover`). |
 | `--datestamp` | Append `-YYYYMMDD` to environment names. Off by default. |
 | `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to newly created environments. Expensive; off by default. |
 | `--suffix=LABEL` | Append `-LABEL` to environment names. Off by default. |

diff --git a/configs/sites/tier1/nas-toss5/README.md b/configs/sites/tier1/nas-toss5/README.md
@@ -12,6 +12,13 @@ This guide documents how to build **spack-stack** on NASA NAS TOSS5 systems. Use
 - [Obtain an Interactive Compute Node](#obtain-an-interactive-compute-node)
 - [Setup spack-stack](#setup-spack-stack)
 - [Using batch_install.sh (recommended)](#using-batch_installsh-recommended)
+  - [Roles and modes](#roles-and-modes)
+  - [The -H flag](#the--h-flag)
+  - [The -e flag](#the--e-flag)
+  - [The --datestamp and --suffix flags](#the---datestamp-and---suffix-flags)
+  - [Environment names](#environment-names)
+  - [Loading the stack](#loading-the-stack)
+  - [Additional options reference](#additional-options-reference)
 - [Create Environments](#create-environments)
   - [oneAPI - ifx Environment](#oneapi---ifx-environment)
   - [oneAPI - ifort Environment](#oneapi---ifort-environment)
@@ -127,6 +134,85 @@ The `-p`, `-q`, and `--constraint` options are Discover-specific and do not appl
 
 ---
 
+### Roles and modes
+
+The script requires two key flags on every invocation:
+
+- **`-r ROLE`** — either `dev` or `ops`
+- **`-m MODE`** — either `build` or `install`
+
+| Role + Mode | What it does |
+|---|---|
+| `-r dev -m build` | For SI Team developers with write access to the shared mirrors. Builds packages from source and writes results to the **default shared build cache**. |
+| `-r dev -m install` | Installs packages from the shared build cache (no compilation) and generates module files. |
+| `-r ops -m build` | For operational partners **without** write access to the shared mirrors. Builds packages and writes to a **custom build cache** specified with `-c BUILDCACHE_DIR`. Allows partners to maintain their own binary cache independently. |
+| `-r ops -m install` | Installs packages from the shared build cache and generates module files. |
+
+The `-m install` step is **required** to generate usable module files. A `-m build` run alone does not produce modules.
+
+---
+
+### The -H flag
+
+Always pass `-H nas-toss5` explicitly on NAS TOSS5. Without `-H`, the script autodetects the site from `$HOSTNAME`; explicit `-H` avoids any ambiguity.
+
+---
+
+### The -e flag
+
+The `-e` flag tells the script to continue even if the environment directories already exist. Without it, the script exits with an error if any target environment directory is present.
+
+- On the **very first run**, you do not need `-e` (the directories do not exist yet).
+- On **every subsequent run** (rebuilds, retries, installs), you must pass `-e`.
+
+---
+
+### Environment names
+
+The script creates one environment per compiler under `envs/` (relative to the repository root, unless overridden with `-d`):
+
+| Mode | Environment names created |
+|---|---|
+| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.3.0-build`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0-build`) |
+| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.3.0`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0`) |
+
+The `ue-` prefix comes from the `unified-dev` template and `ge-` from `geos-dev`; both are built by default on this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates.
+
+---
+
+### Loading the stack
+
+After the `install` step completes, point your shell to the generated module files:
+
+```bash
+module use -a /path/to/spack-stack/envs/ue-oneapi-2024.2.0/install/modulefiles/Core
+module load stack-intel-oneapi-compilers stack-intel-oneapi-mpi
+```
+
+Replace `ue-oneapi-2024.2.0` with the appropriate environment name for your compiler of choice.
+
+---
+
+### Additional options reference
+
+| Flag | Description |
+|---|---|
+| `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build`. |
+| `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). |
+| `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `oneapi@=2024.2.0,gcc@=15.3.0`). Overrides the site defaults. |
+| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates (`unified-dev` and `geos-dev`). |
+| `-a ACCOUNT` | Override the PBS account (default: `s1873`). |
+| `-o` / `--concretize-only` | Concretize environments and stop before installation. |
+| `-n` | Dry run: print what would be executed. |
+| `-L` / `--disable-locks` | Disable Spack install locks and serialize package installs for an isolated recovery run only. |
+| `-t` | Run tests for specific third-party dependencies after installation (list hardcoded in `batch_install.sh`). |
+| `-u` | Populate bootstrap, source, and cargo mirrors. Requires `-r dev -m build`. First run or cache refresh only. |
+| `--datestamp` | Append `-YYYYMMDD` to environment names. Off by default. |
+| `--suffix=LABEL` | Append `-LABEL` to environment names. Off by default. |
+| `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to newly created environments. Expensive; off by default. |
+
+---
+
 ### The --datestamp and --suffix flags
 
 By default, environments are named `<prefix>-<compiler>-<version>` (e.g. `ge-gcc-15.3.0`). To build an environment that others may be using and that you are less likely to remove by accident, tag its name:

diff --git a/configs/sites/tier1/nas/README.md b/configs/sites/tier1/nas/README.md
@@ -12,6 +12,13 @@ This guide documents how to build **spack-stack** on NASA NAS systems. Use `util
 - [Obtain an Interactive Compute Node](#obtain-an-interactive-compute-node)
 - [Setup spack-stack](#setup-spack-stack)
 - [Using batch_install.sh (recommended)](#using-batch_installsh-recommended)
+  - [Roles and modes](#roles-and-modes)
+  - [The -H flag](#the--h-flag)
+  - [The -e flag](#the--e-flag)
+  - [The --datestamp and --suffix flags](#the---datestamp-and---suffix-flags)
+  - [Environment names](#environment-names)
+  - [Loading the stack](#loading-the-stack)
+  - [Additional options reference](#additional-options-reference)
 - [Create Environments](#create-environments)
   - [oneAPI - ifx Environment](#oneapi---ifx-environment)
   - [oneAPI - ifort Environment](#oneapi---ifort-environment)
@@ -129,6 +136,85 @@ The `-p`, `-q`, and `--constraint` options are Discover-specific and do not appl
 
 ---
 
+### Roles and modes
+
+The script requires two key flags on every invocation:
+
+- **`-r ROLE`** — either `dev` or `ops`
+- **`-m MODE`** — either `build` or `install`
+
+| Role + Mode | What it does |
+|---|---|
+| `-r dev -m build` | For SI Team developers with write access to the shared mirrors. Builds packages from source and writes results to the **default shared build cache**. |
+| `-r dev -m install` | Installs packages from the shared build cache (no compilation) and generates module files. |
+| `-r ops -m build` | For operational partners **without** write access to the shared mirrors. Builds packages and writes to a **custom build cache** specified with `-c BUILDCACHE_DIR`. Allows partners to maintain their own binary cache independently. |
+| `-r ops -m install` | Installs packages from the shared build cache and generates module files. |
+
+The `-m install` step is **required** to generate usable module files. A `-m build` run alone does not produce modules.
+
+---
+
+### The -H flag
+
+Always pass `-H nas` explicitly on NAS. Without `-H`, the script autodetects the site from `$HOSTNAME`; explicit `-H` avoids any ambiguity.
+
+---
+
+### The -e flag
+
+The `-e` flag tells the script to continue even if the environment directories already exist. Without it, the script exits with an error if any target environment directory is present.
+
+- On the **very first run**, you do not need `-e` (the directories do not exist yet).
+- On **every subsequent run** (rebuilds, retries, installs), you must pass `-e`.
+
+---
+
+### Environment names
+
+The script creates one environment per compiler under `envs/` (relative to the repository root, unless overridden with `-d`):
+
+| Mode | Environment names created |
+|---|---|
+| `build` | `ue-oneapi-2024.2.0-build`, `ue-oneapi-2025.3.0-build`, `ue-gcc-15.2.0-build`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0-build`) |
+| `install` | `ue-oneapi-2024.2.0`, `ue-oneapi-2025.3.0`, `ue-gcc-15.2.0`; with the `geos-dev` template, `ge-` equivalents (e.g. `ge-oneapi-2024.2.0`) |
+
+The `ue-` prefix comes from the `unified-dev` template and `ge-` from `geos-dev`; both are built by default on this site. Use `-C` (comma-separated, e.g. `-C oneapi@=2024.2.0`) to select compilers and `-T` (comma-separated, e.g. `-T unified-dev`) to select templates.
+
+---
+
+### Loading the stack
+
+After the `install` step completes, point your shell to the generated module files:
+
+```bash
+module use -a /path/to/spack-stack/envs/ue-oneapi-2024.2.0/install/modulefiles/Core
+module load stack-intel-oneapi-compilers stack-intel-oneapi-mpi
+```
+
+Replace `ue-oneapi-2024.2.0` with the appropriate environment name for your compiler of choice.
+
+---
+
+### Additional options reference
+
+| Flag | Description |
+|---|---|
+| `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build`. |
+| `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). |
+| `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `oneapi@=2024.2.0,gcc@=15.2.0`). Overrides the site defaults. |
+| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates (`unified-dev` and `geos-dev`). |
+| `-a ACCOUNT` | Override the PBS account (default: `s1873`). |
+| `-o` / `--concretize-only` | Concretize environments and stop before installation. |
+| `-n` | Dry run: print what would be executed. |
+| `-L` / `--disable-locks` | Disable Spack install locks and serialize package installs for an isolated recovery run only. |
+| `-t` | Run tests for specific third-party dependencies after installation (list hardcoded in `batch_install.sh`). |
+| `-u` | Populate bootstrap, source, and cargo mirrors. Requires `-r dev -m build`. First run or cache refresh only. |
+| `--datestamp` | Append `-YYYYMMDD` to environment names. Off by default. |
+| `--suffix=LABEL` | Append `-LABEL` to environment names. Off by default. |
+| `--with-ai` | Add `ai-env` (py-torch + `jedi-base-env`) to newly created environments. Expensive; off by default. |
+
+---
+
 ### The --datestamp and --suffix flags
 
 By default, environments are named `<prefix>-<compiler>-<version>` (e.g. `ge-gcc-15.3.0`). To build an environment that others may be using and that you are less likely to remove by accident, tag its name:

diff --git a/configs/sites/tier2/bucy/README.md b/configs/sites/tier2/bucy/README.md
@@ -23,12 +23,75 @@ the site NAG module that sets it:
 export NAG_KUSARI_FILE=/ford1/share/gmao_SIteam/nag/nag.key
 ```
 
-For a direct local build, use:
+---
+
+## Using batch_install.sh script
+
+On Bucy the `batch_install.sh` script runs in its one-step `local` mode: it builds directly into the final environment, generates modules, and writes to the shared build cache. There is no batch scheduler on Bucy, so `-s`, `-r`, `-p`, `-q`, `--constraint`, and `-a` do not apply.
+
+### Help and dry-run
+
+```bash
+./util/gmao/batch_install.sh -h
+```
+
+Preview what the script would do without making any changes:
+
+```bash
+./util/gmao/batch_install.sh -m local -H bucy -n
+```
+
+### The -H flag
+
+Always pass `-H bucy` explicitly. Without it, the script autodetects the site from `$HOSTNAME`; explicit `-H` avoids any ambiguity.
+
+### First run
+
+On the very first run, use `-u` to populate the bootstrap, source, and cargo mirrors:
 
 ```bash
 ./util/gmao/batch_install.sh -m local -H bucy -u
 ```
 
+### The -e flag
+
+On every subsequent run (rebuilds, retries), pass `-e` so the script continues in the existing environment directories instead of exiting with an error:
+
+```bash
+./util/gmao/batch_install.sh -m local -H bucy -e
+```
+
+### Selecting compilers and templates
+
+Use `-C` to build only some compilers and `-T` to build only some templates (both comma-separated). By default all compilers in the matrix above are built with the `geos-dev` and `geos-dev-nag` templates (`geos-dev-nag` only applies to NAG):
+
+```bash
+./util/gmao/batch_install.sh -m local -H bucy -e -C gcc@=15.2.0,oneapi@=2024.2.0 -T geos-dev
+```
+
+### Other useful flags
+
+- `-d ENV_DIRS`: override the environment directory (default: `envs/` in the repo root).
+- `-c BUILDCACHE_DIR`: use a custom build cache instead of the shared one.
+- `-o` / `--concretize-only`: stop after concretization.
+- `-L` / `--disable-locks`: disable Spack install locks and serialize package installs; only for an isolated recovery run when no other process modifies the install tree.
+- `-t`: run tests for specific third-party dependencies after installation.
+- `--datestamp` / `--suffix=LABEL`: see below.
+- `--with-ai`: add `ai-env` (py-torch + `jedi-base-env`) to newly created environments; expensive, off by default.
+
+### Environment names
+
+Environments are created directly under `envs/` (not `envs/bucy`), one per compiler, named `<prefix>-<compiler>-<version>`, e.g. `ge-gcc-15.2.0`, `ge-oneapi-2024.2.0`, `ge-nag-7.2.43`. The `ge-` prefix comes from the `geos-dev` and `geos-dev-nag` templates.
+
+To keep an environment others may be using from being replaced, tag its name
+with `--datestamp` (appends `-YYYYMMDD`) and/or `--suffix=LABEL` (appends
+`-LABEL`), e.g. `./util/gmao/batch_install.sh -m local -H bucy --datestamp --suffix=swelltest`
+gives `ge-gcc-15.2.0-YYYYMMDD-swelltest`. Both are off by default. Use the same options together with `-e` to continue a tagged environment.
+
+---
+
+## Site configuration notes
+
 The shared installation, source, bootstrap, and cache locations are under
 `/ford1/share/gmao_SIteam/spack-stack`. Tcl module generation is intentionally
 not enabled in this first pass; it can be added after the Lmod hierarchy is
@@ -45,8 +108,3 @@ gmake, and so on) as externals, and forces the system `tar`, because
 `tar@1.35` fails to build against the system libacl headers.
 
 Environments are created directly under `envs/` (not `envs/bucy`).
-
-To keep an environment others may be using from being replaced, tag its name
-with `--datestamp` (appends `-YYYYMMDD`) and/or `--suffix=LABEL` (appends
-`-LABEL`), e.g. `./util/gmao/batch_install.sh -m local -H bucy --datestamp --suffix=swelltest`
-gives `ge-gcc-15.2.0-YYYYMMDD-swelltest`. Both are off by default.
diff --git a/configs/sites/tier2/discover-gmao/README.md b/configs/sites/tier2/discover-gmao/README.md
@@ -253,6 +253,8 @@ On the very first run, use `-u` to populate the bootstrap, source, and cargo mir
 - `-s`: Submits the install step to SLURM.
 - No `-e` needed since the environment directories do not exist yet.
 
+**Shared cache permissions (discover and discover-gmao only):** after updating the bootstrap, source, binary, and cargo mirrors, `batch_install.sh` makes them group-writable by the `sispack` group (`chgrp` plus `chmod g+rwXs`) so that any SI Team developer can update them. Install trees (environments) are not touched this way. Set `SPACK_STACK_SHARED_GROUP` to use a different group. This step is skipped on other hosts (NAS, bucy, macOS) where the group does not exist.
+
 #### Step 2 — Rebuild or retry runs
 
 After the first run, the environment directories exist and the caches are populated. Drop `-u` and add `-e`:
@@ -460,6 +462,7 @@ The following flags are less commonly used. The descriptions below reflect our b
 | `-c BUILDCACHE_DIR` | Use a custom build cache directory. Required with `-r ops -m build` for operational partners who do not have write access to the shared mirrors. |
 | `-d ENV_DIRS` | Override the default environment directory (default: `envs/` in the repo root). |
 | `-C COMPILERS` | Comma-separated list of compilers to build (e.g., `gcc@=15.2.0,oneapi@=2025.3.0`). Overrides the site defaults. |
+| `-T TEMPLATES` | Comma-separated list of templates to build (e.g., `geos-dev` or `unified-dev,geos-dev`). Overrides the host's default templates. |
 | `-p` / `--partition` | Override the SLURM partition. Defaults to `preops` when account is `s1873`, otherwise unset. |
 | `-q` / `--qos` | Override the SLURM QOS. Defaults to `benchmark` when account is `s1873`, otherwise unset. |
 | `--constraint` | Override the SLURM node constraint. Defaults to `mil` (Milan nodes). No short form (`-c`/`-C` are taken). |