From 188c23aae0a7888f8ca921fb194cddbaf98379ac Mon Sep 17 00:00:00 2001 From: Wes Harrell Date: Tue, 23 Dec 2025 13:35:44 +1300 Subject: [PATCH 1/8] Created new page outline --- docs/Software/Reproducible_Software.md | 52 ++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/Software/Reproducible_Software.md diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md new file mode 100644 index 000000000..df4381300 --- /dev/null +++ b/docs/Software/Reproducible_Software.md @@ -0,0 +1,52 @@ +--- +Title: Reproducible Software +tags: + - software + - reproducible + - reproducibility + - environments + - containers +--- + +## Research Software + +According to [this article in Nature](https://www.nature.com/articles/s41597-022-01710-x): + +> Research software is a fundamental and vital part of research, yet significant challenges +> to discoverability, productivity, quality, reproducibility, and sustainability exist. + +So what can we as researchers and technologists do to make our software easier to reproduce and use? +Below are some tips, tricks and best practice to help get you started. + +## Established Software Environment Tools + +### Conda/Mamba Environments + +TODO: find and point to existing conda + +### Python Virtual Environments + +TODO: find and point to venv + +## New Tools for Creating Reproducible Environments + +### UV + +TODO: I think Jen created a page for this + +### pixi + +TODO: new page + +### anything else work mentioning? + + +## Containerisation + +TODO: Point to Apptainer + + +## Compiled Software + +TODO: point to modules and maybe add something about retrieving modules and dependencies alsong with info about + the OS and other pertinent info From 87ceb71d5e6aefbc65767f5d679ffedb4b253494 Mon Sep 17 00:00:00 2001 From: Jen Reeve Date: Mon, 5 Jan 2026 14:18:27 +1300 Subject: [PATCH 2/8] adding some pointers and details --- docs/Software/Reproducible_Software.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md index df4381300..4e6f91a1d 100644 --- a/docs/Software/Reproducible_Software.md +++ b/docs/Software/Reproducible_Software.md @@ -22,17 +22,17 @@ Below are some tips, tricks and best practice to help get you started. ### Conda/Mamba Environments -TODO: find and point to existing conda +Conda environments are supported via the [Miniforge3](../Software/Available_Applications/Miniforge3.md) module. ### Python Virtual Environments TODO: find and point to venv -## New Tools for Creating Reproducible Environments +## New Tools for Reproducible Environments -### UV +### uv -TODO: I think Jen created a page for this +The [uv](../Software/Available_Applications/uv.md) package manager is available as module. ### pixi @@ -43,10 +43,8 @@ TODO: new page ## Containerisation -TODO: Point to Apptainer - +Containers are supported on Mahuika via [Apptainer](Available_Applications/Apptainer.md). ## Compiled Software -TODO: point to modules and maybe add something about retrieving modules and dependencies alsong with info about - the OS and other pertinent info +In addition to the ability to search available modules using `module spider `, all installed modules and available versions can be found in [this table](Available_Applications/index.md). From d3c89755218fc98339ec3308b339998c632c74c6 Mon Sep 17 00:00:00 2001 From: "callumnmw@gmail.com" Date: Tue, 6 Jan 2026 15:40:16 +1300 Subject: [PATCH 3/8] Added flow chart wheeee --- docs/Software/Reproducible_Software.md | 40 +++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md index df4381300..4deafee49 100644 --- a/docs/Software/Reproducible_Software.md +++ b/docs/Software/Reproducible_Software.md @@ -1,5 +1,4 @@ --- -Title: Reproducible Software tags: - software - reproducible @@ -18,8 +17,47 @@ According to [this article in Nature](https://www.nature.com/articles/s41597-022 So what can we as researchers and technologists do to make our software easier to reproduce and use? Below are some tips, tricks and best practice to help get you started. +```mermaid +flowchart TD + A{"Do you require a specific operating system?"} + A -- "Yes" ---> CON + A -- "No" ---> B + + B{"What software are you using?"} + B -- "Python" ---> D + B -- "Other" ---> C + + C{"Do you need to match environments with another computer?"} + C -- "Yes" ---> E + C -- "No" ---> ENVM + + D{"How many packages are you using?"} + D -- "Few" ---> ENVM + D -- "Dozens" ---> VENV + D -- "Hundreds" ---> CONDA + + E{"Can you build it yourself?"} + E -- "Yes" ---> BLD + E -- "No" ---> CON + + BLD["Build the software from source"] + ENVM["Use the Environment Module"] + CON["Use a Apptainer container"] + CONDA["Use Conda / Mamba"] + VENV["Use Python Virtual Environment"] +``` + + ## Established Software Environment Tools +### Environment Modules + +The most simple way to access software on the cluster is using [Environment Modules](). + +This will get you access to a specific version of your software, built for the clusters hardware. + +If you want to exactly replicate your local environment this may not be suitable. + ### Conda/Mamba Environments TODO: find and point to existing conda From 8b3118210fcbf681b56660a5c49bddcea3598184 Mon Sep 17 00:00:00 2001 From: CallumWalley Date: Tue, 3 Nov 2026 17:03:32 +1300 Subject: [PATCH 4/8] update mermaid --- docs/Software/.pages.yml | 4 + .../Available_Applications/Miniforge3.md | 1 + .../Software/Available_Applications/Python.md | 2 + docs/Software/Available_Applications/pixi.md | 71 ++++++ docs/Software/Available_Applications/uv.md | 1 + .../Installing_Applications_Yourself.md | 2 + docs/Software/Reproducible_Software.md | 235 ++++++++++++++---- docs/Software/Software_Version_Management.md | 20 +- mkdocs.yml | 2 +- 9 files changed, 273 insertions(+), 65 deletions(-) create mode 100644 docs/Software/.pages.yml create mode 100644 docs/Software/Available_Applications/pixi.md diff --git a/docs/Software/.pages.yml b/docs/Software/.pages.yml new file mode 100644 index 000000000..4875952f4 --- /dev/null +++ b/docs/Software/.pages.yml @@ -0,0 +1,4 @@ +--- +nav: + - Available_Applications + - "*" diff --git a/docs/Software/Available_Applications/Miniforge3.md b/docs/Software/Available_Applications/Miniforge3.md index 7622640aa..308f12d7c 100644 --- a/docs/Software/Available_Applications/Miniforge3.md +++ b/docs/Software/Available_Applications/Miniforge3.md @@ -130,4 +130,5 @@ conda env create -f environment.yml -p /nesi/project//my_conda_env - If you only need access to Python and standard numerical libraries (numpy, scipy, matplotlib, etc.), you can use our [Python environment module](Python.md). + - To compare Conda with the other ways of managing software, see [Reproducible Software](../Reproducible_Software.md). diff --git a/docs/Software/Available_Applications/Python.md b/docs/Software/Available_Applications/Python.md index 2736a7e4b..c2ec852cb 100644 --- a/docs/Software/Available_Applications/Python.md +++ b/docs/Software/Available_Applications/Python.md @@ -237,6 +237,8 @@ must be installed separately. Each Python environment module comes with its own particular suite of packages, and the system Python has its own installed packages. +To compare virtual environments with uv, Conda and containers, see [Reproducible Software](../Reproducible_Software.md). + The provided packages can be listed using ``` bash diff --git a/docs/Software/Available_Applications/pixi.md b/docs/Software/Available_Applications/pixi.md new file mode 100644 index 000000000..cd0aa2cec --- /dev/null +++ b/docs/Software/Available_Applications/pixi.md @@ -0,0 +1,71 @@ +--- +created_at: 2026-11-03 +description: Using pixi on Mahuika to install conda and PyPI packages into a project environment, run it in a batch job, and share it with a lock file +title: pixi +tags: + - python + - software +--- + +[//]: <> (APPS PAGE BOILERPLATE START) +{% set app_name = page.title | trim %} +{% set app = applications[app_name] %} +{% include "partials/app_header.html" %} +[//]: <> (APPS PAGE BOILERPLATE END) + +pixi gives each project its own environment, with packages from conda-forge and PyPI. +The packages you ask for are listed in `pixi.toml`, and the exact version of every package is pinned in `pixi.lock`. +To compare pixi with Conda, uv and containers, see [Reproducible Software](../Reproducible_Software.md). + +## Set the cache directory + +By default, pixi keeps its package cache in your home directory. Move it to your project's nobackup directory: + +```sh +export PIXI_CACHE_DIR=/nesi/nobackup/nesi99991/pixi_cache +``` + +Replace `nesi99991` with your project code. Add this line to your `~/.bashrc` to keep the setting. + +## Create a project + +Make the project in your project directory: its environment, in the `.pixi` directory, can contain many files. + +```sh +module purge +module load pixi/{{ app.default }} +cd /nesi/project/nesi99991/my_project +pixi init +pixi add python numpy +pixi run python my_script.py +``` + +Replace `nesi99991` with your project code. Add PyPI packages with `pixi add --pypi PACKAGE_NAME`. + +To run the project in a batch job: + +```sl +#!/bin/bash -e + +#SBATCH --job-name pixi-job +#SBATCH --account nesi99991 +#SBATCH --time 01:00:00 + +module purge +module load pixi/{{ app.default }} + +cd /nesi/project/nesi99991/my_project +pixi run python my_script.py +``` + +## Share an environment + +Keep `pixi.toml` and `pixi.lock` with your code, for example in Git. Anyone with both files can recreate the environment with `pixi install`. + +A project made on Mahuika only lists the `linux-64` platform. To use it on another computer, such as a Mac, add that platform: + +```sh +pixi workspace platform add osx-arm64 +``` + +The [pixi documentation](https://pixi.sh/latest/) has more on projects, tasks and multiple environments. diff --git a/docs/Software/Available_Applications/uv.md b/docs/Software/Available_Applications/uv.md index 39fc7e072..ae5dcc9ea 100644 --- a/docs/Software/Available_Applications/uv.md +++ b/docs/Software/Available_Applications/uv.md @@ -15,6 +15,7 @@ tags: uv will manage Python packages including determining dependencies to create virtual environments. There are two primary approaches to using uv: [for individual scripts](#using-uv-for-scripts) and [for projects or groups of scripts](#using-uv-for-projects). +To compare uv with other ways of managing Python packages, see [Reproducible Software](../Reproducible_Software.md). ## Using uv for scripts diff --git a/docs/Software/Installing_Applications_Yourself.md b/docs/Software/Installing_Applications_Yourself.md index 62392c636..f62f37248 100644 --- a/docs/Software/Installing_Applications_Yourself.md +++ b/docs/Software/Installing_Applications_Yourself.md @@ -18,6 +18,8 @@ In any case, if you have issues, do not hesitate to See [Software Installation Request](Software_Installation_Request.md) for guidelines on what information you should include. +To compare the ways of setting up software yourself, such as virtual environments, Conda and containers, see [Reproducible Software](Reproducible_Software.md). + ## Packages for Existing Module How to add package to an existing module will vary based on the language in question. diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md index 65b0b1334..cc217bb47 100644 --- a/docs/Software/Reproducible_Software.md +++ b/docs/Software/Reproducible_Software.md @@ -1,88 +1,215 @@ --- +created_at: 2025-12-23 +description: Choose between environment modules, Python virtual environments, uv, Conda, pixi, Apptainer containers and building from source on Mahuika, and record your environment so it can be recreated. tags: - software - - reproducible - - reproducibility - - environments + - python - containers --- -## Research Software +!!! prerequisite + If you are new to modules, start with the [Environment & Modules](../Tutorials/Introduction_To_HPC/Environment_And_Modules.md) lesson. -According to [this article in Nature](https://www.nature.com/articles/s41597-022-01710-x): - -> Research software is a fundamental and vital part of research, yet significant challenges -> to discoverability, productivity, quality, reproducibility, and sustainability exist. - -So what can we as researchers and technologists do to make our software easier to reproduce and use? -Below are some tips, tricks and best practice to help get you started. +There are several ways to set up the software for your research on Mahuika. +They differ in how much control you have, how well the software is optimised for the cluster, +and how easily you, or someone else, can recreate the same environment later. +This page helps you choose one, and record it so your results can be reproduced. ```mermaid flowchart TD - A{"Do you require a specific operating system?"} - A -- "Yes" ---> CON - A -- "No" ---> B - - B{"What software are you using?"} - B -- "Python" ---> D - B -- "Other" ---> C - - C{"Do you need to match environments with another computer?"} - C -- "Yes" ---> E - C -- "No" ---> ENVM - - D{"How many packages are you using?"} - D -- "Few" ---> ENVM - D -- "Dozens" ---> VENV - D -- "Hundreds" ---> CONDA - - E{"Can you build it yourself?"} - E -- "Yes" ---> BLD - E -- "No" ---> CON - - BLD["Build the software from source"] - ENVM["Use the Environment Module"] - CON["Use a Apptainer container"] - CONDA["Use Conda / Mamba"] - VENV["Use Python Virtual Environment"] + A{"Do you need a specific operating system?"} + A -- "Yes" --> CON + A -- "No" --> B + + B{"What software are you using?"} + B -- "Python" --> D + B -- "R or Julia" --> RJ + B -- "Other" --> C + + C{"Do you need to match an environment on another computer?"} + C -- "Yes" --> CON + C -- "No" --> M + + M{"Is it installed as a module?"} + M -- "Yes" --> ENVM + M -- "No" --> K + + K{"Is it available as a conda package?"} + K -- "Yes" --> CONDA + K -- "No" --> E + + E{"Can you build it yourself?"} + E -- "Yes" --> BLD + E -- "No" --> REQ + + D{"How many packages are you using?"} + D -- "Few" --> ENVM + D -- "Dozens" --> VENV + D -- "Hundreds" --> CONDA + + ENVM["Use an environment module"] + VENV["Use a Python virtual environment or uv"] + CONDA["Use Conda or pixi"] + RJ["Add packages to the R or Julia module"] + CON["Use an Apptainer container"] + BLD["Build it from source"] + REQ["Ask us to install it"] + + classDef outcome stroke-width:3px + class ENVM,CON,VENV,CONDA,RJ,BLD outcome + + click ENVM "#environment-modules" + click CON "#apptainer-containers" + click VENV "#python-packages" + click CONDA "#conda-environments" + click RJ "#r-and-julia-packages" + click BLD "#build-from-source" ``` +## Comparison -## Established Software Environment Tools +| Method | Good for | On Mahuika | Portable | +| - | - | - | - | +| [Environment module](#environment-modules) | Software REANNZ already installs | Yes | No | +| [Python virtual environment](#virtual-environments) | Adding Python packages to the Python module | Partly, with the Python module | Mostly, with `requirements.txt` | +| [uv](#uv) | New Python projects, and sharing them | Partly, with the Python module | Yes, with `uv.lock` | +| [Conda or Mamba](#conda-and-mamba) | Conda packages, and software with non-Python dependencies | No | Mostly, on the same operating system | +| [pixi](#pixi) | Conda packages, with a lock file | No | Yes, with `pixi.lock` | +| [Apptainer container](#apptainer-containers) | Another operating system, Docker images, and exact copies | Usually not | Yes, copy the image | +| [Build from source](#build-from-source) | Software not available any other way | Yes | No, build it again | -### Environment Modules +## Environment modules -The most simple way to access software on the cluster is using [Environment Modules](). +REANNZ installs and maintains software on the cluster as modules. +They are built and optimised for the cluster's hardware, and need no setup. +Search for a module with `module spider`, or browse [Supported Applications](Available_Applications/index.md). + +```sh +module spider Python +``` + +Load modules with their full version, not just their name. +The default version changes when a newer one is installed, so `module load Python` may give you a different version later. + +```sh +module load Python/{{ applications.Python.default }} +``` -This will get you access to a specific version of your software, built for the clusters hardware. +See [Software Version Management](Software_Version_Management.md) for how versions and compiler toolchains work. -If you want to exactly replicate your local environment this may not be suitable. +A module can't be copied to another computer. To get the same environment elsewhere, use one of the other methods on this page. +If the software or version you need isn't installed, we can install it: see [Software Installation Request](Software_Installation_Request.md). -### Conda/Mamba Environments +## Python packages -Conda environments are supported via the [Miniforge3](../Software/Available_Applications/Miniforge3.md) module. +The Python module includes many common packages for computational work, such as NumPy, SciPy and Matplotlib. +If it has everything you need, use it directly. To add packages, use a virtual environment or uv, described below. -### Python Virtual Environments +Only use `pip install --user` if you are not using Python as part of a reproducible workflow. It installs packages in your home directory, where every project shares them and they can conflict. -TODO: find and point to venv +### Virtual environments -## New Tools for Reproducible Environments +A Python virtual environment is a directory holding the packages for one project, on top of a Python module. +You create one with the `venv` module that comes with Python, and install packages into it with `pip`. +Create it in your project directory, so it doesn't fill your home directory and your collaborators can use it, +and load the same Python module whenever you use it. +See [Installing packages in a Python virtual environment](Available_Applications/Python.md#installing-packages-in-a-python-virtual-environment). ### uv -The [uv](../Software/Available_Applications/uv.md) package manager is available as module. +[uv](Available_Applications/uv.md) is a newer tool that does the same job as `venv` and `pip`, much faster. +It also writes a lock file, `uv.lock`, listing the exact version of every package, +so `uv sync` can recreate the same environment on Mahuika or any other computer. +This makes uv a good choice for a new Python project, or one you share with others. +The [uv](Available_Applications/uv.md#caching) page explains how to keep its cache out of your home directory. + +## Conda environments + +Conda installs packages from channels such as conda-forge. +Unlike `pip`, it can install compiled libraries and software written in other languages, so many research tools are distributed as conda packages. +Use Conda when your software is a conda package, or your Python project needs many packages with complex dependencies. +You can manage conda packages with Conda and Mamba, or with pixi. + +Conda packages have some drawbacks on the cluster: + +- They are generic builds, not optimised for the cluster's hardware, and they aren't curated by REANNZ. +- An environment can contain a large number of files, which count towards your project's + [quota on the number of files](../Storage/Filesystems_and_Quotas.md#nesiproject). + +### Conda and Mamba + +`conda` and `mamba` are provided by the [Miniforge3](Available_Applications/Miniforge3.md) module. +`mamba` works like `conda`, but resolves dependencies faster. +Follow the setup on the Miniforge3 page. In particular, don't run `conda init`, +and create environments in your project directory with `--prefix`, so they don't fill your home directory. ### pixi -TODO: new page +A newer tool, [pixi](Available_Applications/pixi.md), installs the same conda packages, as well as packages from PyPI. +Like uv, it keeps a lock file, `pixi.lock`, next to the project's `pixi.toml`, +so `pixi install` recreates the exact environment on another computer. +Each project gets its own environment, in a `.pixi` directory inside the project. +The [pixi](Available_Applications/pixi.md#set-the-cache-directory) page explains how to keep its cache out of your home directory. + +## R and Julia packages + +The R and Julia modules include many packages, and you can install more into a library in your project directory, +where your collaborators can use the same versions. +See [installing R packages](Available_Applications/R.md#dealing-with-packages) and +[installing Julia packages](Available_Applications/Julia.md#installing-julia-packages). + +To record exact package versions, use the [`renv`](https://rstudio.github.io/renv/) package for R, +or a Julia [project environment](https://pkgdocs.julialang.org/v1/environments/), whose `Manifest.toml` lists the version of every package. + +For other languages, see [Packages for Existing Module](Installing_Applications_Yourself.md#packages-for-existing-module). + +## Apptainer containers + +A container holds software together with its operating system and every library it needs, in a single image file. +Mahuika runs containers with [Apptainer](Containers/Apptainer.md), which can also pull and run Docker images. + +Use a container when: + +- the software needs a different operating system, or newer system libraries, than the cluster has, +- the developers distribute it as a Docker or Apptainer image, +- you need exactly the same environment on Mahuika and another computer. + +An image is a single file, so it is easy to copy, archive and share, and it doesn't add thousands of files to your quota. +Building your own image takes more work than the other methods, and MPI and GPU software need extra setup: +see [MPI in Apptainer](Containers/Apptainer.md#mpi) and [NVIDIA GPU Containers](Containers/NVIDIA_GPU_Containers.md). + +## Build from source + +If none of these methods has your software, you can compile it yourself with the cluster's compilers and libraries. +See [Installing Applications Yourself](Installing_Applications_Yourself.md) for the usual steps. + +Software you build yourself is yours to maintain. Install it in your project directory, +and keep a note of the modules you built it with, as you will need the same modules loaded to run it. -### anything else work mentioning? +We can also install software for you, or help you install it. +See [Software Installation Request](Software_Installation_Request.md) for what to tell us. +## Record your environment -## Containerisation +Whichever method you use, save the files that describe your environment with your code, +ideally in version control such as [Git](../Getting_Started/Cheat_Sheets/Git-Reference_Sheet.md). +With them, you, your collaborators or a reviewer can recreate the environment later. +This is part of making research software reusable, as described in +[Introducing the FAIR Principles for research software](https://www.nature.com/articles/s41597-022-01710-x). -Containers are supported on Mahuika via [Apptainer](Available_Applications/Apptainer.md). +| Method | What to save | How to recreate it | +| - | - | - | +| Environment modules | The `module load` lines, with full versions, in your batch script | Load the same modules | +| Python virtual environment | `requirements.txt`, from `pip freeze > requirements.txt` | `pip install -r requirements.txt` in a new virtual environment | +| uv | `pyproject.toml` and `uv.lock` | `uv sync` | +| Conda | `environment.yml`, from `conda env export > environment.yml` | `conda env create --file environment.yml --prefix /nesi/project//my_conda_env` | +| pixi | `pixi.toml` and `pixi.lock` | `pixi install` | +| Apptainer | The definition file (`.def`), and the image (`.sif`) for an exact copy | `apptainer build`, or copy the image | +| Built from source | The source version, the modules you loaded, and your build commands | Repeat the build | -## Compiled Software +## Environments in JupyterLab -In addition to the ability to search available modules using `module spider `, all installed modules and available versions can be found in [this table](Available_Applications/index.md). +You can use a module, Python virtual environment, Conda environment or container in JupyterLab on Mahuika OnDemand +by adding it as a kernel. The `nesi-add-kernel` tool does most of the setup. +See [Python and R Kernels in JupyterLab](../Interactive_Computing/OnDemand/Apps/JupyterLab/python_and_r_kernels_in_JupyterLab.md) +and [Containers as Kernels in JupyterLab](../Interactive_Computing/OnDemand/Apps/JupyterLab/containers_as_kernels_in_JupyterLab.md). diff --git a/docs/Software/Software_Version_Management.md b/docs/Software/Software_Version_Management.md index 409fc6c00..9df19cb43 100644 --- a/docs/Software/Software_Version_Management.md +++ b/docs/Software/Software_Version_Management.md @@ -33,14 +33,14 @@ this may change. module load ANSYS/18.1 ``` -Will always load that version specifically. +Will always load `18.1` specifically. ## Toolchains and Environment Management ### Compiler Toolchains -Many of the software modules on the cluster are built with a specific compiler toolchain (often referred to as simply a _toolchain_). A toolchain -is the compiler and development libraries that are used to build software on the cluster. +Many of the software modules on the cluster are built with a specific compiler toolchain (often referred to as simply a _toolchain_). +A toolchain is the compiler and development libraries that are used to build software on the cluster. You can see this in the module version, such as the following: @@ -49,13 +49,13 @@ Python/3.11.6-foss-2023a PnetCDF/1.9.0-intel-2020a ``` -The software version and toolchain are separated by the first hyphen, for instance the -version of Python is `3.11.6` and the toolchain it was built with is `foss-2023a`. Whareas -PnetCDF was built with `intel-2020a` These are incompatible toolchains since they use different -compilers and have conflicting dependencies. Therefore loading these modules in the -same environment with either conflict and fail, or will swap module dependencies which will -cause some of your software to crash or fail with a library error. For example if I try to -load the software above I receive the following notification: +The software version and toolchain are separated by the first hyphen, +for instance the version of Python is `3.11.6` and the toolchain it was built with is `foss-2023a`. +Whereas PnetCDF was built with `intel-2020a`, +These are incompatible toolchains since they use different compilers and have conflicting dependencies. +Loading these modules in the same environment with either conflict and fail, +or will swap module dependencies which will cause some of your software to crash or fail with a library error. +For example if I try to load the software above I receive the following notification: ```out The following have been reloaded with a version change: diff --git a/mkdocs.yml b/mkdocs.yml index 6b3e73bd5..179e7a6ba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -139,7 +139,7 @@ plugins: include_dir: overrides - llmstxt: markdown_description: >- - NeSI Mahuika HPC support documentation: getting access, running batch jobs + Mahuika HPC support documentation: getting access, running batch jobs with Slurm, available software and modules, storage and data transfer, interactive computing (Jupyter, OnDemand), and platform policies. # These are fnmatch patterns, not pathlib globs - `*` already crosses `/`, so From d1a056e118b91d49df062919f17a5d0b3d800536 Mon Sep 17 00:00:00 2001 From: Cal <35017184+CallumWalley@users.noreply.github.com> Date: Wed, 4 Nov 2026 13:58:06 +1300 Subject: [PATCH 5/8] Update docs/Software/Available_Applications/Python.md Co-authored-by: Jen Reeve <31419037+jen-reeve@users.noreply.github.com> Signed-off-by: Cal <35017184+CallumWalley@users.noreply.github.com> --- docs/Software/Available_Applications/Python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Software/Available_Applications/Python.md b/docs/Software/Available_Applications/Python.md index c2ec852cb..8e9cbfd66 100644 --- a/docs/Software/Available_Applications/Python.md +++ b/docs/Software/Available_Applications/Python.md @@ -63,7 +63,7 @@ Our most recent Python environment modules have: #SBATCH --account nesi99999 #SBATCH --ntasks=2 # Number of MPI tasks #SBATCH --time=00:30:00 - #SBATCH --mem-per-cpu=512MB # Memory per logical CPU + #SBATCH --mem-per-cpu 512MB # Memory per logical CPU module load Python srun python PythonMPI.py # Executes ntasks copies of the script From 2bdcbd28de8663866d3a324fb6c0396f892ead22 Mon Sep 17 00:00:00 2001 From: Cal <35017184+CallumWalley@users.noreply.github.com> Date: Wed, 4 Nov 2026 13:59:02 +1300 Subject: [PATCH 6/8] Apply batched suggestions from code review Co-authored-by: Jen Reeve <31419037+jen-reeve@users.noreply.github.com> Signed-off-by: Cal <35017184+CallumWalley@users.noreply.github.com> --- docs/Software/Available_Applications/Miniforge3.md | 11 +++++------ docs/Software/Available_Applications/Python.md | 8 ++++---- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/Software/Available_Applications/Miniforge3.md b/docs/Software/Available_Applications/Miniforge3.md index 308f12d7c..f5a759d4a 100644 --- a/docs/Software/Available_Applications/Miniforge3.md +++ b/docs/Software/Available_Applications/Miniforge3.md @@ -16,7 +16,7 @@ environment managers. Miniforge is a replacement for Miniconda and the Anaconda `conda` and `mamba` provide the tools that allow you to install packages and their dependencies in a dedicated environment. This gives you more freedom to -install software yourself. Note this may come at the expense of less optimized +install software yourself. Note this may come at the expense of less optimised packages and no curation by the REANNZ team. ## Module loading and conda environments isolation @@ -42,19 +42,18 @@ Here are the explanations for each line of this snippet: - `source $(conda info --base)/etc/profile.d/conda.sh` ensures that you can use the `conda activate` command. - `export PYTHONNOUSERSITE=1` makes sure that local packages installed - in your home folder `~/.local/lib/pythonX.Y/site-packages/` (where + in your home directory `~/.local/lib/pythonX.Y/site-packages/` (where `X.Y` is the Python version, e.g. 3.8) by `pip install --user` are excluded from your conda environments. !!! warning "conda init" - Please *do not* run `conda init`. It inserts a + Please _do not_ run `conda init`. It inserts a snippet in your `~/.bashrc` file that will freeze the version of conda used, bypassing the environment module system. It is also known to cause other issues with your account. !!! warning "Defaults Channel" The `defaults` channel is blocked due to Anaconda's licensing requirements. If this channel is listed in either the `environment.yml` or your conda configuration you will receive an error: - ``` out Failed to create Conda environment The channel is not accessible or is invalid. @@ -75,7 +74,7 @@ Here are the explanations for each line of this snippet: Conda environments and the conda packages cache can take a lot of storage space. By default, Conda use [/home](../../Storage/Filesystems_and_Quotas.md), -which is restricted to 20GB on Mahuika. Here are some techniques to avoid +which is restricted to 20 GB on Mahuika. Here are some techniques to avoid running out of space when using Conda. First, we recommend that you move the cache folder used for downloaded @@ -96,7 +95,7 @@ setting is saved in your `~/.condarc` configuration file. Next, we recommend using the `-p` or `--prefix` options when creating new conda environments, instead of `-n` or `--name` options. Using `-p` or `--prefix`, you can specify the conda environment folder location, -ideally in your project folder. For example: +ideally in your project directory. For example: ``` sh conda create --prefix /nesi/project//my_conda_env python=3.8 diff --git a/docs/Software/Available_Applications/Python.md b/docs/Software/Available_Applications/Python.md index 8e9cbfd66..d025b7f87 100644 --- a/docs/Software/Available_Applications/Python.md +++ b/docs/Software/Available_Applications/Python.md @@ -12,7 +12,7 @@ description: Using Python on the cluster and choosing between system Python and {% include "partials/app_header.html" %} [//]: <> (APPS PAGE BOILERPLATE END) -All versions of Python available on Mahuika platforms are owned and +All versions of Python available on Mahuika are owned and licensed by the Python Software Foundation. Each version is released under a specific open-source licence. The licences are available on [the Python documentation server](https://docs.python.org). @@ -59,10 +59,10 @@ Our most recent Python environment modules have: ```sl #!/bin/bash -e - #SBATCH --job-name=PythonMPI + #SBATCH --job-name PythonMPI #SBATCH --account nesi99999 - #SBATCH --ntasks=2 # Number of MPI tasks - #SBATCH --time=00:30:00 + #SBATCH --ntasks 2 # Number of MPI tasks + #SBATCH --time 00:30:00 #SBATCH --mem-per-cpu 512MB # Memory per logical CPU module load Python From 4480bc2fbc1fbbef0b63349911bece3ce4ba53f4 Mon Sep 17 00:00:00 2001 From: CallumWalley Date: Wed, 4 Nov 2026 15:13:26 +1300 Subject: [PATCH 7/8] more readable --- docs/Software/Reproducible_Software.md | 42 +++++++++++++++----------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md index cc217bb47..b2c01131e 100644 --- a/docs/Software/Reproducible_Software.md +++ b/docs/Software/Reproducible_Software.md @@ -17,53 +17,59 @@ This page helps you choose one, and record it so your results can be reproduced. ```mermaid flowchart TD - A{"Do you need a specific operating system?"} - A -- "Yes" --> CON + %% Questions, in the order a reader meets them + A{{"Do you need a specific
operating system?"}} + B{{"What software
are you using?"}} + D{{"How many packages
are you using?"}} + C{{"Do you need to match
an environment on
another computer?"}} + M{{"Is it installed
as a module?"}} + K{{"Is it a
conda package?"}} + E{{"Can you build
it yourself?"}} + + %% Outcomes, named as in the comparison table + ENVM(["Environment module"]) + VENV(["Python virtual
environment or uv"]) + CONDA(["Conda or pixi"]) + RJ(["R or Julia packages"]) + CON(["Apptainer container"]) + BLD(["Build from source"]) + REQ(["Ask us to install it"]) + + %% Edge order sets the layout: keep "Other" first, Python second A -- "No" --> B - B{"What software are you using?"} + B -- "Other" --> C B -- "Python" --> D B -- "R or Julia" --> RJ - B -- "Other" --> C - C{"Do you need to match an environment on another computer?"} C -- "Yes" --> CON C -- "No" --> M - M{"Is it installed as a module?"} M -- "Yes" --> ENVM M -- "No" --> K - K{"Is it available as a conda package?"} K -- "Yes" --> CONDA K -- "No" --> E - E{"Can you build it yourself?"} E -- "Yes" --> BLD E -- "No" --> REQ - D{"How many packages are you using?"} D -- "Few" --> ENVM D -- "Dozens" --> VENV D -- "Hundreds" --> CONDA - ENVM["Use an environment module"] - VENV["Use a Python virtual environment or uv"] - CONDA["Use Conda or pixi"] - RJ["Add packages to the R or Julia module"] - CON["Use an Apptainer container"] - BLD["Build it from source"] - REQ["Ask us to install it"] + A -- "Yes" --> CON classDef outcome stroke-width:3px - class ENVM,CON,VENV,CONDA,RJ,BLD outcome + class ENVM,VENV,CONDA,RJ,CON,BLD,REQ outcome click ENVM "#environment-modules" - click CON "#apptainer-containers" click VENV "#python-packages" click CONDA "#conda-environments" click RJ "#r-and-julia-packages" + click CON "#apptainer-containers" click BLD "#build-from-source" + click REQ "#build-from-source" ``` ## Comparison From 21c5a0bcbf2a7792c801f5ef778e76a13959c1e4 Mon Sep 17 00:00:00 2001 From: CallumWalley Date: Wed, 4 Nov 2026 15:33:10 +1300 Subject: [PATCH 8/8] better layout --- docs/Software/Reproducible_Software.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md index b2c01131e..96fba2b76 100644 --- a/docs/Software/Reproducible_Software.md +++ b/docs/Software/Reproducible_Software.md @@ -15,15 +15,16 @@ They differ in how much control you have, how well the software is optimised for and how easily you, or someone else, can recreate the same environment later. This page helps you choose one, and record it so your results can be reproduced. +{% raw %} + ```mermaid flowchart TD %% Questions, in the order a reader meets them A{{"Do you need a specific
operating system?"}} B{{"What software
are you using?"}} - D{{"How many packages
are you using?"}} C{{"Do you need to match
an environment on
another computer?"}} + D{{"How many packages
are you using?"}} M{{"Is it installed
as a module?"}} - K{{"Is it a
conda package?"}} E{{"Can you build
it yourself?"}} %% Outcomes, named as in the comparison table @@ -46,10 +47,7 @@ flowchart TD C -- "No" --> M M -- "Yes" --> ENVM - M -- "No" --> K - - K -- "Yes" --> CONDA - K -- "No" --> E + M -- "No" --> E E -- "Yes" --> BLD E -- "No" --> REQ @@ -72,6 +70,8 @@ flowchart TD click REQ "#build-from-source" ``` +{% endraw %} + ## Comparison | Method | Good for | On Mahuika | Portable |