diff --git a/docs/Software/.pages.yml b/docs/Software/.pages.yml new file mode 100644 index 000000000..4875952f4 --- /dev/null +++ b/docs/Software/.pages.yml @@ -0,0 +1,4 @@ +--- +nav: + - Available_Applications + - "*" diff --git a/docs/Software/Available_Applications/Miniforge3.md b/docs/Software/Available_Applications/Miniforge3.md index 7622640aa..f5a759d4a 100644 --- a/docs/Software/Available_Applications/Miniforge3.md +++ b/docs/Software/Available_Applications/Miniforge3.md @@ -16,7 +16,7 @@ environment managers. Miniforge is a replacement for Miniconda and the Anaconda `conda` and `mamba` provide the tools that allow you to install packages and their dependencies in a dedicated environment. This gives you more freedom to -install software yourself. Note this may come at the expense of less optimized +install software yourself. Note this may come at the expense of less optimised packages and no curation by the REANNZ team. ## Module loading and conda environments isolation @@ -42,19 +42,18 @@ Here are the explanations for each line of this snippet: - `source $(conda info --base)/etc/profile.d/conda.sh` ensures that you can use the `conda activate` command. - `export PYTHONNOUSERSITE=1` makes sure that local packages installed - in your home folder `~/.local/lib/pythonX.Y/site-packages/` (where + in your home directory `~/.local/lib/pythonX.Y/site-packages/` (where `X.Y` is the Python version, e.g. 3.8) by `pip install --user` are excluded from your conda environments. !!! warning "conda init" - Please *do not* run `conda init`. It inserts a + Please _do not_ run `conda init`. It inserts a snippet in your `~/.bashrc` file that will freeze the version of conda used, bypassing the environment module system. It is also known to cause other issues with your account. !!! warning "Defaults Channel" The `defaults` channel is blocked due to Anaconda's licensing requirements. If this channel is listed in either the `environment.yml` or your conda configuration you will receive an error: - ``` out Failed to create Conda environment The channel is not accessible or is invalid. @@ -75,7 +74,7 @@ Here are the explanations for each line of this snippet: Conda environments and the conda packages cache can take a lot of storage space. By default, Conda use [/home](../../Storage/Filesystems_and_Quotas.md), -which is restricted to 20GB on Mahuika. Here are some techniques to avoid +which is restricted to 20 GB on Mahuika. Here are some techniques to avoid running out of space when using Conda. First, we recommend that you move the cache folder used for downloaded @@ -96,7 +95,7 @@ setting is saved in your `~/.condarc` configuration file. Next, we recommend using the `-p` or `--prefix` options when creating new conda environments, instead of `-n` or `--name` options. Using `-p` or `--prefix`, you can specify the conda environment folder location, -ideally in your project folder. For example: +ideally in your project directory. For example: ``` sh conda create --prefix /nesi/project//my_conda_env python=3.8 @@ -130,4 +129,5 @@ conda env create -f environment.yml -p /nesi/project//my_conda_env - If you only need access to Python and standard numerical libraries (numpy, scipy, matplotlib, etc.), you can use our [Python environment module](Python.md). + - To compare Conda with the other ways of managing software, see [Reproducible Software](../Reproducible_Software.md). diff --git a/docs/Software/Available_Applications/Python.md b/docs/Software/Available_Applications/Python.md index 2736a7e4b..d025b7f87 100644 --- a/docs/Software/Available_Applications/Python.md +++ b/docs/Software/Available_Applications/Python.md @@ -12,7 +12,7 @@ description: Using Python on the cluster and choosing between system Python and {% include "partials/app_header.html" %} [//]: <> (APPS PAGE BOILERPLATE END) -All versions of Python available on Mahuika platforms are owned and +All versions of Python available on Mahuika are owned and licensed by the Python Software Foundation. Each version is released under a specific open-source licence. The licences are available on [the Python documentation server](https://docs.python.org). @@ -59,11 +59,11 @@ Our most recent Python environment modules have: ```sl #!/bin/bash -e - #SBATCH --job-name=PythonMPI + #SBATCH --job-name PythonMPI #SBATCH --account nesi99999 - #SBATCH --ntasks=2 # Number of MPI tasks - #SBATCH --time=00:30:00 - #SBATCH --mem-per-cpu=512MB # Memory per logical CPU + #SBATCH --ntasks 2 # Number of MPI tasks + #SBATCH --time 00:30:00 + #SBATCH --mem-per-cpu 512MB # Memory per logical CPU module load Python srun python PythonMPI.py # Executes ntasks copies of the script @@ -237,6 +237,8 @@ must be installed separately. Each Python environment module comes with its own particular suite of packages, and the system Python has its own installed packages. +To compare virtual environments with uv, Conda and containers, see [Reproducible Software](../Reproducible_Software.md). + The provided packages can be listed using ``` bash diff --git a/docs/Software/Available_Applications/pixi.md b/docs/Software/Available_Applications/pixi.md new file mode 100644 index 000000000..cd0aa2cec --- /dev/null +++ b/docs/Software/Available_Applications/pixi.md @@ -0,0 +1,71 @@ +--- +created_at: 2026-11-03 +description: Using pixi on Mahuika to install conda and PyPI packages into a project environment, run it in a batch job, and share it with a lock file +title: pixi +tags: + - python + - software +--- + +[//]: <> (APPS PAGE BOILERPLATE START) +{% set app_name = page.title | trim %} +{% set app = applications[app_name] %} +{% include "partials/app_header.html" %} +[//]: <> (APPS PAGE BOILERPLATE END) + +pixi gives each project its own environment, with packages from conda-forge and PyPI. +The packages you ask for are listed in `pixi.toml`, and the exact version of every package is pinned in `pixi.lock`. +To compare pixi with Conda, uv and containers, see [Reproducible Software](../Reproducible_Software.md). + +## Set the cache directory + +By default, pixi keeps its package cache in your home directory. Move it to your project's nobackup directory: + +```sh +export PIXI_CACHE_DIR=/nesi/nobackup/nesi99991/pixi_cache +``` + +Replace `nesi99991` with your project code. Add this line to your `~/.bashrc` to keep the setting. + +## Create a project + +Make the project in your project directory: its environment, in the `.pixi` directory, can contain many files. + +```sh +module purge +module load pixi/{{ app.default }} +cd /nesi/project/nesi99991/my_project +pixi init +pixi add python numpy +pixi run python my_script.py +``` + +Replace `nesi99991` with your project code. Add PyPI packages with `pixi add --pypi PACKAGE_NAME`. + +To run the project in a batch job: + +```sl +#!/bin/bash -e + +#SBATCH --job-name pixi-job +#SBATCH --account nesi99991 +#SBATCH --time 01:00:00 + +module purge +module load pixi/{{ app.default }} + +cd /nesi/project/nesi99991/my_project +pixi run python my_script.py +``` + +## Share an environment + +Keep `pixi.toml` and `pixi.lock` with your code, for example in Git. Anyone with both files can recreate the environment with `pixi install`. + +A project made on Mahuika only lists the `linux-64` platform. To use it on another computer, such as a Mac, add that platform: + +```sh +pixi workspace platform add osx-arm64 +``` + +The [pixi documentation](https://pixi.sh/latest/) has more on projects, tasks and multiple environments. diff --git a/docs/Software/Available_Applications/uv.md b/docs/Software/Available_Applications/uv.md index 39fc7e072..ae5dcc9ea 100644 --- a/docs/Software/Available_Applications/uv.md +++ b/docs/Software/Available_Applications/uv.md @@ -15,6 +15,7 @@ tags: uv will manage Python packages including determining dependencies to create virtual environments. There are two primary approaches to using uv: [for individual scripts](#using-uv-for-scripts) and [for projects or groups of scripts](#using-uv-for-projects). +To compare uv with other ways of managing Python packages, see [Reproducible Software](../Reproducible_Software.md). ## Using uv for scripts diff --git a/docs/Software/Installing_Applications_Yourself.md b/docs/Software/Installing_Applications_Yourself.md index 62392c636..f62f37248 100644 --- a/docs/Software/Installing_Applications_Yourself.md +++ b/docs/Software/Installing_Applications_Yourself.md @@ -18,6 +18,8 @@ In any case, if you have issues, do not hesitate to See [Software Installation Request](Software_Installation_Request.md) for guidelines on what information you should include. +To compare the ways of setting up software yourself, such as virtual environments, Conda and containers, see [Reproducible Software](Reproducible_Software.md). + ## Packages for Existing Module How to add package to an existing module will vary based on the language in question. diff --git a/docs/Software/Reproducible_Software.md b/docs/Software/Reproducible_Software.md new file mode 100644 index 000000000..96fba2b76 --- /dev/null +++ b/docs/Software/Reproducible_Software.md @@ -0,0 +1,221 @@ +--- +created_at: 2025-12-23 +description: Choose between environment modules, Python virtual environments, uv, Conda, pixi, Apptainer containers and building from source on Mahuika, and record your environment so it can be recreated. +tags: + - software + - python + - containers +--- + +!!! prerequisite + If you are new to modules, start with the [Environment & Modules](../Tutorials/Introduction_To_HPC/Environment_And_Modules.md) lesson. + +There are several ways to set up the software for your research on Mahuika. +They differ in how much control you have, how well the software is optimised for the cluster, +and how easily you, or someone else, can recreate the same environment later. +This page helps you choose one, and record it so your results can be reproduced. + +{% raw %} + +```mermaid +flowchart TD + %% Questions, in the order a reader meets them + A{{"Do you need a specific
operating system?"}} + B{{"What software
are you using?"}} + C{{"Do you need to match
an environment on
another computer?"}} + D{{"How many packages
are you using?"}} + M{{"Is it installed
as a module?"}} + E{{"Can you build
it yourself?"}} + + %% Outcomes, named as in the comparison table + ENVM(["Environment module"]) + VENV(["Python virtual
environment or uv"]) + CONDA(["Conda or pixi"]) + RJ(["R or Julia packages"]) + CON(["Apptainer container"]) + BLD(["Build from source"]) + REQ(["Ask us to install it"]) + + %% Edge order sets the layout: keep "Other" first, Python second + A -- "No" --> B + + B -- "Other" --> C + B -- "Python" --> D + B -- "R or Julia" --> RJ + + C -- "Yes" --> CON + C -- "No" --> M + + M -- "Yes" --> ENVM + M -- "No" --> E + + E -- "Yes" --> BLD + E -- "No" --> REQ + + D -- "Few" --> ENVM + D -- "Dozens" --> VENV + D -- "Hundreds" --> CONDA + + A -- "Yes" --> CON + + classDef outcome stroke-width:3px + class ENVM,VENV,CONDA,RJ,CON,BLD,REQ outcome + + click ENVM "#environment-modules" + click VENV "#python-packages" + click CONDA "#conda-environments" + click RJ "#r-and-julia-packages" + click CON "#apptainer-containers" + click BLD "#build-from-source" + click REQ "#build-from-source" +``` + +{% endraw %} + +## Comparison + +| Method | Good for | On Mahuika | Portable | +| - | - | - | - | +| [Environment module](#environment-modules) | Software REANNZ already installs | Yes | No | +| [Python virtual environment](#virtual-environments) | Adding Python packages to the Python module | Partly, with the Python module | Mostly, with `requirements.txt` | +| [uv](#uv) | New Python projects, and sharing them | Partly, with the Python module | Yes, with `uv.lock` | +| [Conda or Mamba](#conda-and-mamba) | Conda packages, and software with non-Python dependencies | No | Mostly, on the same operating system | +| [pixi](#pixi) | Conda packages, with a lock file | No | Yes, with `pixi.lock` | +| [Apptainer container](#apptainer-containers) | Another operating system, Docker images, and exact copies | Usually not | Yes, copy the image | +| [Build from source](#build-from-source) | Software not available any other way | Yes | No, build it again | + +## Environment modules + +REANNZ installs and maintains software on the cluster as modules. +They are built and optimised for the cluster's hardware, and need no setup. +Search for a module with `module spider`, or browse [Supported Applications](Available_Applications/index.md). + +```sh +module spider Python +``` + +Load modules with their full version, not just their name. +The default version changes when a newer one is installed, so `module load Python` may give you a different version later. + +```sh +module load Python/{{ applications.Python.default }} +``` + +See [Software Version Management](Software_Version_Management.md) for how versions and compiler toolchains work. + +A module can't be copied to another computer. To get the same environment elsewhere, use one of the other methods on this page. +If the software or version you need isn't installed, we can install it: see [Software Installation Request](Software_Installation_Request.md). + +## Python packages + +The Python module includes many common packages for computational work, such as NumPy, SciPy and Matplotlib. +If it has everything you need, use it directly. To add packages, use a virtual environment or uv, described below. + +Only use `pip install --user` if you are not using Python as part of a reproducible workflow. It installs packages in your home directory, where every project shares them and they can conflict. + +### Virtual environments + +A Python virtual environment is a directory holding the packages for one project, on top of a Python module. +You create one with the `venv` module that comes with Python, and install packages into it with `pip`. +Create it in your project directory, so it doesn't fill your home directory and your collaborators can use it, +and load the same Python module whenever you use it. +See [Installing packages in a Python virtual environment](Available_Applications/Python.md#installing-packages-in-a-python-virtual-environment). + +### uv + +[uv](Available_Applications/uv.md) is a newer tool that does the same job as `venv` and `pip`, much faster. +It also writes a lock file, `uv.lock`, listing the exact version of every package, +so `uv sync` can recreate the same environment on Mahuika or any other computer. +This makes uv a good choice for a new Python project, or one you share with others. +The [uv](Available_Applications/uv.md#caching) page explains how to keep its cache out of your home directory. + +## Conda environments + +Conda installs packages from channels such as conda-forge. +Unlike `pip`, it can install compiled libraries and software written in other languages, so many research tools are distributed as conda packages. +Use Conda when your software is a conda package, or your Python project needs many packages with complex dependencies. +You can manage conda packages with Conda and Mamba, or with pixi. + +Conda packages have some drawbacks on the cluster: + +- They are generic builds, not optimised for the cluster's hardware, and they aren't curated by REANNZ. +- An environment can contain a large number of files, which count towards your project's + [quota on the number of files](../Storage/Filesystems_and_Quotas.md#nesiproject). + +### Conda and Mamba + +`conda` and `mamba` are provided by the [Miniforge3](Available_Applications/Miniforge3.md) module. +`mamba` works like `conda`, but resolves dependencies faster. +Follow the setup on the Miniforge3 page. In particular, don't run `conda init`, +and create environments in your project directory with `--prefix`, so they don't fill your home directory. + +### pixi + +A newer tool, [pixi](Available_Applications/pixi.md), installs the same conda packages, as well as packages from PyPI. +Like uv, it keeps a lock file, `pixi.lock`, next to the project's `pixi.toml`, +so `pixi install` recreates the exact environment on another computer. +Each project gets its own environment, in a `.pixi` directory inside the project. +The [pixi](Available_Applications/pixi.md#set-the-cache-directory) page explains how to keep its cache out of your home directory. + +## R and Julia packages + +The R and Julia modules include many packages, and you can install more into a library in your project directory, +where your collaborators can use the same versions. +See [installing R packages](Available_Applications/R.md#dealing-with-packages) and +[installing Julia packages](Available_Applications/Julia.md#installing-julia-packages). + +To record exact package versions, use the [`renv`](https://rstudio.github.io/renv/) package for R, +or a Julia [project environment](https://pkgdocs.julialang.org/v1/environments/), whose `Manifest.toml` lists the version of every package. + +For other languages, see [Packages for Existing Module](Installing_Applications_Yourself.md#packages-for-existing-module). + +## Apptainer containers + +A container holds software together with its operating system and every library it needs, in a single image file. +Mahuika runs containers with [Apptainer](Containers/Apptainer.md), which can also pull and run Docker images. + +Use a container when: + +- the software needs a different operating system, or newer system libraries, than the cluster has, +- the developers distribute it as a Docker or Apptainer image, +- you need exactly the same environment on Mahuika and another computer. + +An image is a single file, so it is easy to copy, archive and share, and it doesn't add thousands of files to your quota. +Building your own image takes more work than the other methods, and MPI and GPU software need extra setup: +see [MPI in Apptainer](Containers/Apptainer.md#mpi) and [NVIDIA GPU Containers](Containers/NVIDIA_GPU_Containers.md). + +## Build from source + +If none of these methods has your software, you can compile it yourself with the cluster's compilers and libraries. +See [Installing Applications Yourself](Installing_Applications_Yourself.md) for the usual steps. + +Software you build yourself is yours to maintain. Install it in your project directory, +and keep a note of the modules you built it with, as you will need the same modules loaded to run it. + +We can also install software for you, or help you install it. +See [Software Installation Request](Software_Installation_Request.md) for what to tell us. + +## Record your environment + +Whichever method you use, save the files that describe your environment with your code, +ideally in version control such as [Git](../Getting_Started/Cheat_Sheets/Git-Reference_Sheet.md). +With them, you, your collaborators or a reviewer can recreate the environment later. +This is part of making research software reusable, as described in +[Introducing the FAIR Principles for research software](https://www.nature.com/articles/s41597-022-01710-x). + +| Method | What to save | How to recreate it | +| - | - | - | +| Environment modules | The `module load` lines, with full versions, in your batch script | Load the same modules | +| Python virtual environment | `requirements.txt`, from `pip freeze > requirements.txt` | `pip install -r requirements.txt` in a new virtual environment | +| uv | `pyproject.toml` and `uv.lock` | `uv sync` | +| Conda | `environment.yml`, from `conda env export > environment.yml` | `conda env create --file environment.yml --prefix /nesi/project//my_conda_env` | +| pixi | `pixi.toml` and `pixi.lock` | `pixi install` | +| Apptainer | The definition file (`.def`), and the image (`.sif`) for an exact copy | `apptainer build`, or copy the image | +| Built from source | The source version, the modules you loaded, and your build commands | Repeat the build | + +## Environments in JupyterLab + +You can use a module, Python virtual environment, Conda environment or container in JupyterLab on Mahuika OnDemand +by adding it as a kernel. The `nesi-add-kernel` tool does most of the setup. +See [Python and R Kernels in JupyterLab](../Interactive_Computing/OnDemand/Apps/JupyterLab/python_and_r_kernels_in_JupyterLab.md) +and [Containers as Kernels in JupyterLab](../Interactive_Computing/OnDemand/Apps/JupyterLab/containers_as_kernels_in_JupyterLab.md). diff --git a/docs/Software/Software_Version_Management.md b/docs/Software/Software_Version_Management.md index 409fc6c00..9df19cb43 100644 --- a/docs/Software/Software_Version_Management.md +++ b/docs/Software/Software_Version_Management.md @@ -33,14 +33,14 @@ this may change. module load ANSYS/18.1 ``` -Will always load that version specifically. +Will always load `18.1` specifically. ## Toolchains and Environment Management ### Compiler Toolchains -Many of the software modules on the cluster are built with a specific compiler toolchain (often referred to as simply a _toolchain_). A toolchain -is the compiler and development libraries that are used to build software on the cluster. +Many of the software modules on the cluster are built with a specific compiler toolchain (often referred to as simply a _toolchain_). +A toolchain is the compiler and development libraries that are used to build software on the cluster. You can see this in the module version, such as the following: @@ -49,13 +49,13 @@ Python/3.11.6-foss-2023a PnetCDF/1.9.0-intel-2020a ``` -The software version and toolchain are separated by the first hyphen, for instance the -version of Python is `3.11.6` and the toolchain it was built with is `foss-2023a`. Whareas -PnetCDF was built with `intel-2020a` These are incompatible toolchains since they use different -compilers and have conflicting dependencies. Therefore loading these modules in the -same environment with either conflict and fail, or will swap module dependencies which will -cause some of your software to crash or fail with a library error. For example if I try to -load the software above I receive the following notification: +The software version and toolchain are separated by the first hyphen, +for instance the version of Python is `3.11.6` and the toolchain it was built with is `foss-2023a`. +Whereas PnetCDF was built with `intel-2020a`, +These are incompatible toolchains since they use different compilers and have conflicting dependencies. +Loading these modules in the same environment with either conflict and fail, +or will swap module dependencies which will cause some of your software to crash or fail with a library error. +For example if I try to load the software above I receive the following notification: ```out The following have been reloaded with a version change: diff --git a/mkdocs.yml b/mkdocs.yml index 75bac073a..e4e049794 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -139,7 +139,7 @@ plugins: include_dir: overrides - llmstxt: markdown_description: >- - NeSI Mahuika HPC support documentation: getting access, running batch jobs + Mahuika HPC support documentation: getting access, running batch jobs with Slurm, available software and modules, storage and data transfer, interactive computing (Jupyter, OnDemand), and platform policies. # These are fnmatch patterns, not pathlib globs - `*` already crosses `/`, so