MANA (MPI-Agnostic, Network-Agnostic MPI)

MANA is an implementation of transparent checkpointing for MPI. It is built as a plugin on top of DMTCP.

For details of installing and using MANA, please see:

• MANA Manual

Older documentation:

• MANA documentation (https://mana-doc.readthedocs.io/en/latest/)
• the MANA 'man' page (or man ./mana.1 on a local copy)

As seen in the MANA documentation, for install and a quick start, do:

git clone https://github.com/mpickpt/mana
cd mana
./configure
make -j8
# Testing on a single host (but see MANA docs for running on a cluster):
PATH_TO_MANA/bin/mana_coordinator
PATH_TO_MANA/bin/mana_launch [mana options] [user_program and args]

If dmtcp/ is empty, ./configure checks out the DMTCP commit that MANA uses (the dmtcp submodule). A later git pull can change that commit without changing dmtcp/; make then warns, and git submodule update checks out the new commit.

make puts MANA in the source tree's bin and lib/dmtcp. To install it, with DMTCP, into another directory, give that directory to ./configure:

./configure --prefix=$HOME/mana-install
make -j8
make install        # 'make uninstall' removes it

MANA's scripts find its other files from their own directory, so keep the default --bindir and --libdir (PREFIX/bin and PREFIX/lib).

On Debian and Ubuntu, also install the package libc6-dbg: MANA needs the symbols of the dynamic loader, which these distributions strip.

./configure uses the MPI library whose mpicc comes first in PATH: MPICH or a library based on it (such as Cray MPICH), or Open MPI 4.1 or 5.0. Build the program with the same MPI library. To switch MPI libraries, run make clean first. For example, on Fedora, with Open MPI:

PATH=/usr/lib64/openmpi/bin:$PATH ./configure
PATH=/usr/lib64/openmpi/bin:$PATH make -j8

With the Cray Programming Environment, as on Perlmutter, ./configure uses the compiler wrappers cc, CC and ftn instead. To choose the MPI compilers yourself, set MPICC, MPICXX and MPIFC. For example:

./configure MPICC=cc MPICXX=CC MPIFC=ftn

In a container that runs as root, Open MPI's mpirun needs OMPI_ALLOW_RUN_AS_ROOT=1 and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1.

In a container, MANA needs personality(ADDR_NO_RANDOMIZE), which the default seccomp profile of Docker and Podman forbids: run the container with --security-opt seccomp=unconfined (or a profile that allows it), as MANA's CI does.

To run MANA's tests, which checkpoint and restart each test program:

make tests                               # builds them; 'make' does not
test/autotest.py         # or: make check
test/autotest.py --help  # e.g. --launcher 'srun -n {n}'

MANA is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License, version 3 or (at your option) any later version. This is the same license as DMTCP. See COPYING.LESSER for the LGPL and COPYING for the GNU GPL, whose terms the LGPL incorporates. The DMTCP submodule and the autotools scripts carry their own license notices.

---

To cite this project in a publicatoin, please cite:

• "MANA for MPI: MPI-Agnostic Network-Agnostic Transparent Checkpointing", Rohan Garg, Gregory Price, and Gene Cooperman, HPDC'19. (original academic prototype)

For other MANA publications with updates to this project, see:

• "Enabling Practical Transparent Checkpointing for MPI: A Topological Sort Approach", Yao Xu and Gene Cooperman, IEEE International Conference on Cluster Computing (Cluster'24)

• "Implementation-Oblivious Transparent Checkpoint-Restart for MPI", Yao Xu, Leonid Belyaev, Twinkle Jain, Derek Schafer, Anthony Skjellum, Gene Cooperman, SuperCheck-SC23 Workshop at SC'23. (near production version of MANA)