Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/Software/.pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
nav:
- Available_Applications
- "*"
12 changes: 6 additions & 6 deletions docs/Software/Available_Applications/Miniforge3.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@

The `Miniforge3` environment module provides the
[Conda](https://docs.conda.io/projects/conda/en/latest/) and [Mamba](https://mamba.readthedocs.io/en/latest/) package and
environment managers. Miniforge is a replacement for Miniconda and the Anaconda _defaults_ channel which is now [licence encumbered](https://www.anaconda.com/legal).

Check warning on line 15 in docs/Software/Available_Applications/Miniforge3.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'Miniforge' is misspelled.

`conda` and `mamba` provide the tools that allow you to install packages and their
dependencies in a dedicated environment. This gives you more freedom to
install software yourself. Note this may come at the expense of less optimized
install software yourself. Note this may come at the expense of less optimised
packages and no curation by the REANNZ team.

## Module loading and conda environments isolation
Expand All @@ -42,23 +42,22 @@
- `source $(conda info --base)/etc/profile.d/conda.sh` ensures that
you can use the `conda activate` command.
- `export PYTHONNOUSERSITE=1` makes sure that local packages installed
in your home folder `~/.local/lib/pythonX.Y/site-packages/` (where
in your home directory `~/.local/lib/pythonX.Y/site-packages/` (where
`X.Y` is the Python version, e.g. 3.8) by `pip install --user` are
excluded from your conda environments.

!!! warning "conda init"

Check warning on line 49 in docs/Software/Available_Applications/Miniforge3.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'init' is misspelled.
Please *do not* run `conda init`. It inserts a
Please _do not_ run `conda init`. It inserts a
snippet in your `~/.bashrc` file that will freeze the version of conda
used, bypassing the environment module system. It is also known to cause other issues with your account.

!!! warning "Defaults Channel"
The `defaults` channel is blocked due to Anaconda's licensing requirements.
If this channel is listed in either the `environment.yml` or your conda configuration you will receive an error:

``` out
Failed to create Conda environment
The channel is not accessible or is invalid.
```

Check notice on line 60 in docs/Software/Available_Applications/Miniforge3.md

View workflow job for this annotation

GitHub Actions / Check Markdown

Trailing spaces

Expected: 0 or 2; Actual: 1

If your conda config includes the defaults channel you can remove it with:

Expand All @@ -75,7 +74,7 @@
Conda environments and the conda packages cache can take a lot of
storage space. By default, Conda use
[/home](../../Storage/Filesystems_and_Quotas.md),
which is restricted to 20GB on Mahuika. Here are some techniques to avoid
which is restricted to 20 GB on Mahuika. Here are some techniques to avoid
running out of space when using Conda.

First, we recommend that you move the cache folder used for downloaded
Expand All @@ -96,7 +95,7 @@
Next, we recommend using the `-p` or `--prefix` options when creating
new conda environments, instead of `-n` or `--name` options. Using `-p`
or `--prefix`, you can specify the conda environment folder location,
ideally in your project folder. For example:
ideally in your project directory. For example:

``` sh
conda create --prefix /nesi/project/<project_code>/my_conda_env python=3.8
Expand Down Expand Up @@ -128,6 +127,7 @@
- If you want a more reproducible and isolated environment, we
recommend using [Apptainer containers](Apptainer.md).
- If you only need access to Python and standard numerical libraries
(numpy, scipy, matplotlib, etc.), you can use our

Check notice on line 130 in docs/Software/Available_Applications/Miniforge3.md

View workflow job for this annotation

GitHub Actions / Check Markdown

Trailing spaces

Expected: 0 or 2; Actual: 1
[Python environment module](Python.md).
- To compare Conda with the other ways of managing software, see [Reproducible Software](../Reproducible_Software.md).

12 changes: 7 additions & 5 deletions docs/Software/Available_Applications/Python.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
{% include "partials/app_header.html" %}
[//]: <> (APPS PAGE BOILERPLATE END)

All versions of Python available on Mahuika platforms are owned and
All versions of Python available on Mahuika are owned and
licensed by the Python Software Foundation. Each version is released
under a specific open-source licence. The licences are available on
[the Python documentation server](https://docs.python.org).
Expand Down Expand Up @@ -59,11 +59,11 @@
```sl
#!/bin/bash -e

#SBATCH --job-name=PythonMPI
#SBATCH --job-name PythonMPI
#SBATCH --account nesi99999
#SBATCH --ntasks=2 # Number of MPI tasks
#SBATCH --time=00:30:00
#SBATCH --mem-per-cpu=512MB # Memory per logical CPU
#SBATCH --ntasks 2 # Number of MPI tasks
#SBATCH --time 00:30:00
#SBATCH --mem-per-cpu 512MB # Memory per logical CPU

module load Python
srun python PythonMPI.py # Executes ntasks copies of the script
Expand Down Expand Up @@ -102,13 +102,13 @@
=== "Shared Memory Example"

``` sl
#!/bin/bash -e

Check warning on line 105 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check slurm scripts

slurm_shebang

Your shebang was ' #!/bin/bash -e', should use '^#!\/bin\/bash -e\s*$'

#SBATCH --job-name=PytonMultiprocessing

Check warning on line 107 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check slurm scripts

inconsistant_delimiter

Header uses both whitespace and '=' delimiters.

Check notice on line 107 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check slurm scripts

malformed_delimiter

Whitespace is preferred SLURM header delimiter.
#SBATCH --account nesi99999
#SBATCH --cpus-per-task=2 # Number of logical CPUs

Check notice on line 109 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check slurm scripts

malformed_delimiter

Whitespace is preferred SLURM header delimiter.
#SBATCH --time=00:10:00

Check notice on line 110 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check slurm scripts

malformed_delimiter

Whitespace is preferred SLURM header delimiter.
#SBATCH --mem-per-cpu=512MB # Memory per logical CPU

Check notice on line 111 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check slurm scripts

malformed_delimiter

Whitespace is preferred SLURM header delimiter.

module load Python
python PythonMultiprocessing.py
Expand Down Expand Up @@ -161,7 +161,7 @@
from within the script or using a command line argument. In the
following both options are presented:

The job scripts calling both examples:

Check warning on line 164 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.batch_script

'Write 'batch script', not 'job scripts'.'

``` sl
#!/bin/bash -e
Expand Down Expand Up @@ -237,6 +237,8 @@
its own particular suite of packages, and the system Python has its own
installed packages.

To compare virtual environments with uv, Conda and containers, see [Reproducible Software](../Reproducible_Software.md).

The provided packages can be listed using

``` bash
Expand All @@ -263,7 +265,7 @@
dependencies between projects, avoid filling your home space and being
able to share installation with collaborators

### Installing packages in a Python virtual environment

Check warning on line 268 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

lexical_illusions

'There's a lexical illusion in 'a Python virtual environment

A Python virtual environment is lightweight system to create an
environment which contains specific packages for a project, without
Expand All @@ -277,7 +279,7 @@
python3 -m venv /nesi/project/PROJECT_ID/my_venv
```

where `PROJECT_ID` is your NeSI project ID.

Check warning on line 282 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.project_code

'Write 'project code', not 'project ID'.'

Check warning on line 282 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.nesi

'Call the organisation REANNZ and the platform Mahuika. Only write NeSI where a name still carries it (see 'Names and terms' in FORMAT.md).'

Note that you need to *activate* the virtual environment before using it
(to run a script or install packages)
Expand All @@ -293,7 +295,7 @@
pip install prodXY
```

Then a Slurm job submission script running your Python script would look

Check warning on line 298 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.batch_script

'Write 'batch script', not 'submission script'.'
like

``` sl
Expand Down Expand Up @@ -328,7 +330,7 @@
A downside of this is that now your virtual environment also finds
packages from your `$HOME` folder. To avoid this behaviour, make sure
to use `export PYTHONNOUSERSITE=1` before calling pip or running a
Python script. For example, in a Slurm job submission script

Check warning on line 333 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.batch_script

'Write 'batch script', not 'submission script'.'

``` sl
#!/bin/bash -e
Expand All @@ -347,14 +349,14 @@

## Further notes

### iPython

Check warning on line 352 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Prose

lexical_illusions

'There's a lexical illusion in 'iPython

Check warning on line 352 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.

iPython (*i*nteractive *Python*) is an enhanced tool for accessing a

Check warning on line 354 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'nteractive' is misspelled.

Check warning on line 354 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.
Python command line. It is available in many Mahuika Python modules.

#### Starting iPython

Check warning on line 357 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.

To open an iPython console, simply run the `ipython` command:

Check warning on line 359 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.

``` sh
[jblo123@build-wm ~]$ module load Python/3.6.3-gimkl-2017a
Expand All @@ -363,7 +365,7 @@

#### Listing available functions

You can use iPython to list the functions available that start with a

Check warning on line 368 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.
given string. Please note that if the string denotes a module (i.e., it
has a full stop somewhere in it), that module (or the function you want
from it) must first be imported, using either an "import X" statement or
Expand Down Expand Up @@ -393,7 +395,7 @@

#### Getting information about an object

In iPython, you can query any object by typing the object name followed

Check warning on line 398 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.
by a question mark (?), then hitting Enter. For instance:

``` py
Expand Down Expand Up @@ -423,6 +425,6 @@
defined yet, Python will tell you that the object in question couldn't
be found.

#### Quitting iPython

Check warning on line 428 in docs/Software/Available_Applications/Python.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'iPython' is misspelled.

Just enter the `quit` command at the iPython prompt.
71 changes: 71 additions & 0 deletions docs/Software/Available_Applications/pixi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
created_at: 2026-11-03
description: Using pixi on Mahuika to install conda and PyPI packages into a project environment, run it in a batch job, and share it with a lock file
title: pixi
tags:
- python
- software
---

[//]: <> (APPS PAGE BOILERPLATE START)
{% set app_name = page.title | trim %}
{% set app = applications[app_name] %}
{% include "partials/app_header.html" %}
[//]: <> (APPS PAGE BOILERPLATE END)

pixi gives each project its own environment, with packages from conda-forge and PyPI.
The packages you ask for are listed in `pixi.toml`, and the exact version of every package is pinned in `pixi.lock`.
To compare pixi with Conda, uv and containers, see [Reproducible Software](../Reproducible_Software.md).

## Set the cache directory

By default, pixi keeps its package cache in your home directory. Move it to your project's nobackup directory:

```sh
export PIXI_CACHE_DIR=/nesi/nobackup/nesi99991/pixi_cache
```

Replace `nesi99991` with your project code. Add this line to your `~/.bashrc` to keep the setting.

## Create a project

Make the project in your project directory: its environment, in the `.pixi` directory, can contain many files.

```sh
module purge
module load pixi/{{ app.default }}
cd /nesi/project/nesi99991/my_project
pixi init
pixi add python numpy
pixi run python my_script.py
```

Replace `nesi99991` with your project code. Add PyPI packages with `pixi add --pypi PACKAGE_NAME`.

To run the project in a batch job:

```sl
#!/bin/bash -e

#SBATCH --job-name pixi-job
#SBATCH --account nesi99991
#SBATCH --time 01:00:00

module purge
module load pixi/{{ app.default }}

cd /nesi/project/nesi99991/my_project
pixi run python my_script.py
```

## Share an environment

Keep `pixi.toml` and `pixi.lock` with your code, for example in Git. Anyone with both files can recreate the environment with `pixi install`.

A project made on Mahuika only lists the `linux-64` platform. To use it on another computer, such as a Mac, add that platform:

```sh
pixi workspace platform add osx-arm64
```

The [pixi documentation](https://pixi.sh/latest/) has more on projects, tasks and multiple environments.
1 change: 1 addition & 0 deletions docs/Software/Available_Applications/uv.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ tags:

uv will manage Python packages including determining dependencies to create virtual environments.
There are two primary approaches to using uv: [for individual scripts](#using-uv-for-scripts) and [for projects or groups of scripts](#using-uv-for-projects).
To compare uv with other ways of managing Python packages, see [Reproducible Software](../Reproducible_Software.md).

## Using uv for scripts

Expand Down
2 changes: 2 additions & 0 deletions docs/Software/Installing_Applications_Yourself.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

See [Software Installation Request](Software_Installation_Request.md) for guidelines on what information you should include.

To compare the ways of setting up software yourself, such as virtual environments, Conda and containers, see [Reproducible Software](Reproducible_Software.md).

## Packages for Existing Module

How to add package to an existing module will vary based on the language in question.
Expand Down Expand Up @@ -104,7 +106,7 @@
- The location of the compiled library or libraries, using
`-L/path/to/lib/`
- The name of each library, typically without prefixes and suffixes.
For example, if the full library file name is `libfoo.so.1.2.3`

Check warning on line 109 in docs/Software/Installing_Applications_Yourself.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.filename

'Write it as one word ('filename', 'hostname'), not 'file name'. Fix: 'filename'.'
(with aliases `libfoo.so.1` and `libfoo.so`), the expected entry on
the link line is `-lfoo`.

Expand Down Expand Up @@ -170,7 +172,7 @@
built with OpenMP support, which requires you to set an OpenMP flag
(`-fopenmp` for GNU compilers, `-qopenmp` for Intel) in your linker
command.
- Do you see a very long list of complex-looking function names, and

Check warning on line 175 in docs/Software/Installing_Applications_Yourself.md

View workflow job for this annotation

GitHub Actions / Check Prose

weasel_words.very

'Substitute 'damn' every time you're inclined to write 'very'; your editor will delete it and the writing will be just as it should be.'
does your source code or external library dependency include C++
code? You may need to explicitly link against the C++ standard
library (`-lstdc++` for GNU compilers, `-cxxlib` for Intel
Expand Down
Loading
Loading