diff --git a/README.md b/README.md
index a7d05ce0f..a63100369 100644
--- a/README.md
+++ b/README.md
@@ -1,32 +1,58 @@
-## MANA (MPI-Agnostic, Network-Agnostic MPI)
+## MANA (MPI-Agnostic, Network-Agnostic Checkpointing)
MANA is an implementation of transparent checkpointing for MPI. It is
-built as a plugin on top of [DMTCP](https://github.com/dmtcp/dmtcp).
+built as a plugin on top of [DMTCP](https://github.com/dmtcp/dmtcp). An MPI
+program runs under MANA without changes and without relinking.
-For details of installing and using MANA, please see:
-- [MANA Manual](https://github.com/mpickpt/mana/wiki)
+### Documentation
-Older documentation:
-- [MANA documentation (https://mana-doc.readthedocs.io/en/latest/) ](https://mana-doc.readthedocs.io/en/latest/)
-- [the MANA 'man' page](manpages/mana.1.md) (or `man ./mana.1` on a local copy)
+- [MANA's website](https://mpickpt.github.io/mana/): a quick start, a user
+ guide, a developer guide, and the release notes.
+- [The MANA man page](manpages/mana.1.md): how to checkpoint and restart, and
+ the options of each command. In a clone, read it with
+ `man -l manpages/mana.1`.
-As seen in the MANA documentation, for install and a quick start, do:
+MANA's commands (`mana_coordinator`, `mana_launch`, `mana_status` and
+`mana_restart`) are wrappers of DMTCP's commands, and they accept DMTCP's
+options too. See [DMTCP's documentation](https://dmtcp.github.io/).
+
+### Building and installing
+
+MANA needs:
+
+- an MPI library: MPICH or a library based on it (such as Cray MPICH), or
+ Open MPI 4.1 or 5.0;
+- a C++20 compiler, and Python 3 for `mana_launch` and `mana_restart`;
+- on Debian and Ubuntu, the package libc6-dbg: MANA needs the symbols of
+ the dynamic loader, which these distributions strip.
+
+To build MANA:
git clone https://github.com/mpickpt/mana
cd mana
- ./configure
+ ./configure # also checks out DMTCP, MANA's dmtcp submodule
make -j8
- # Testing on a single host (but see MANA docs for running on a cluster):
- PATH_TO_MANA/bin/mana_coordinator
- PATH_TO_MANA/bin/mana_launch [mana options] [user_program and args]
-If `dmtcp/` is empty, `./configure` checks out the DMTCP commit that MANA
-uses (the `dmtcp` submodule). A later `git pull` can change that commit
-without changing `dmtcp/`; `make` then warns, and `git submodule update`
-checks out the new commit.
+`make` builds only MANA, and puts it in the source tree's `bin` and
+`lib/dmtcp`.
+
+`./configure` uses the MPI compilers that come first in `PATH`: `mpicc`,
+`mpic++` (or `mpicxx`) and `mpifort` (or `mpif90`), or `cc`, `CC` and `ftn`
+with the Cray Programming Environment. To choose others, set `MPICC`,
+`MPICXX` and `MPIFC`. For example:
+
+ ./configure MPICC=$MPI_HOME/bin/mpicc MPICXX=$MPI_HOME/bin/mpicxx \
+ MPIFC=$MPI_HOME/bin/mpifort
+
+Build the program with the same MPI library. To switch MPI libraries, run
+`make clean` first.
+
+A later `git pull` can change the DMTCP commit that MANA uses without
+changing `dmtcp/`; `make` then warns, and `git submodule update` checks out
+the new commit.
-`make` puts MANA in the source tree's `bin` and `lib/dmtcp`. To install it,
-with DMTCP, into another directory, give that directory to `./configure`:
+To install MANA, with DMTCP, into another directory, give that directory to
+`./configure`:
./configure --prefix=$HOME/mana-install
make -j8
@@ -35,23 +61,6 @@ with DMTCP, into another directory, give that directory to `./configure`:
MANA's scripts find its other files from their own directory, so keep the
default `--bindir` and `--libdir` (`PREFIX/bin` and `PREFIX/lib`).
-On Debian and Ubuntu, also install the package libc6-dbg: MANA needs the
-symbols of the dynamic loader, which these distributions strip.
-
-`./configure` uses the MPI library whose `mpicc` comes first in `PATH`: MPICH
-or a library based on it (such as Cray MPICH), or Open MPI 4.1 or 5.0. Build
-the program with the same MPI library. To switch MPI libraries, run `make
-clean` first. For example, on Fedora, with Open MPI:
-
- PATH=/usr/lib64/openmpi/bin:$PATH ./configure
- PATH=/usr/lib64/openmpi/bin:$PATH make -j8
-
-With the Cray Programming Environment, as on Perlmutter, `./configure` uses
-the compiler wrappers `cc`, `CC` and `ftn` instead. To choose the MPI
-compilers yourself, set `MPICC`, `MPICXX` and `MPIFC`. For example:
-
- ./configure MPICC=cc MPICXX=CC MPIFC=ftn
-
In a container that runs as root, Open MPI's `mpirun` needs
`OMPI_ALLOW_RUN_AS_ROOT=1` and `OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1`.
@@ -60,12 +69,71 @@ default seccomp profile of Docker and Podman forbids: run the container with
`--security-opt seccomp=unconfined` (or a profile that allows it), as MANA's
CI does.
+### Quick start
+
+After building MANA, build its example program, `examples/ring`. Then start
+a coordinator, and run the program under MANA on one host:
+
+ make -C examples # builds examples/ring
+ bin/mana_coordinator
+ mpirun -np 4 bin/mana_launch examples/ring # 60 steps, 1 s each
+
+While it runs, checkpoint it from another shell, in the same directory:
+
+ bin/mana_status --bcheckpoint # each rank N writes ckpt_rank_N/
+
+The command returns when the checkpoint is complete. Then stop the job
+(Ctrl-C), and restart it from the checkpoint, with a new coordinator:
+
+ bin/mana_coordinator
+ mpirun -np 4 bin/mana_restart # goes on from the checkpoint
+
+The restarted job goes on from the step of the checkpoint, not from step 1.
+On a cluster, run the same commands in an allocation, with `srun` in place of
+`mpirun`. The man page has the other ways to checkpoint, such as
+`mana_status --kcheckpoint` and periodic checkpoints.
+
+### Make targets
+
+| Target | What it does |
+|---|---|
+| `make` | Builds DMTCP and MANA, and puts them in `bin/` and `lib/dmtcp/`. It first prints the build environment and, in a git clone, links MANA's git hook. |
+| `make mana` | The same build, without printing the environment or linking the hook. |
+| `make tests` | Builds the end-to-end tests (`test/`), the example (`examples/ring`) and the benchmarks (`bench/`). `make` does not build them. |
+| `make check` | Builds MANA and the end-to-end tests, and runs the tests with `test/autotest.py`. |
+| `make install` | Installs MANA and DMTCP into the `--prefix` of `./configure`. `make uninstall` removes them. |
+| `make tidy` | Removes checkpoint files from the top directory: `ckpt_rank_*`, `ckpt_*.dmtcp` and `dmtcp_restart_script*`. |
+| `make clean` | Does `make tidy`, and removes MANA's build outputs in `src/`. It keeps DMTCP's build and the tests. |
+| `make distclean` | Does `make clean` and DMTCP's `make distclean`, and removes `bin/`, `lib/` and the files that `./configure` made. Run `./configure` again after it. |
+
+To build only one set of programs, run `make` in its directory:
+`make -C test` (the end-to-end tests), `make -C examples` (`ring`), or
+`make -C bench` (the benchmarks).
+
+### Tests and benchmarks
+
To run MANA's tests, which checkpoint and restart each test program:
- make tests # builds them; 'make' does not
+ make tests # builds the tests, examples and benchmarks
test/autotest.py # or: make check
test/autotest.py --help # e.g. --launcher 'srun -n {n}'
+The benchmarks in `bench/` measure MANA's overhead. They are built with
+`-O2`, and `make check` does not run them.
+
+- `virtid_bench` measures the cost that MANA adds to each wrapped MPI call.
+ Run it natively and under MANA, and subtract.
+- `synthetic_app` has profiles inspired by the communication of applications
+ such as VASP and GROMACS. A run under MANA must print the same checksum
+ as a native run. See
+ [bench/README-synthetic_app.md](bench/README-synthetic_app.md).
+
+`synthetic_app` has less computation and more communication than the
+applications that inspired its profiles, so it is closer to a
+microbenchmark: MANA's overhead on a real program can be different.
+
+### License
+
MANA is free software: you can redistribute it and/or modify it under the
terms of the GNU Lesser General Public License, version 3 or (at your option)
any later version. This is the same license as DMTCP. See
@@ -75,7 +143,7 @@ autotools scripts carry their own license notices.
---
-To cite this project in a publicatoin, please cite:
+To cite this project in a publication, please cite:
* "MANA for MPI: MPI-Agnostic Network-Agnostic Transparent Checkpointing",
Rohan Garg, Gregory Price, and Gene Cooperman, HPDC'19.
diff --git a/manpages/mana.1 b/manpages/mana.1
index 8e5820908..2975993cd 100644
--- a/manpages/mana.1
+++ b/manpages/mana.1
@@ -1,28 +1,30 @@
.\" Automatically generated by Pandoc 2.9.2.1
.\"
-.TH "MANA" "1" "31 March 2022" "" ""
+.TH "MANA" "1" "5 November 2026" "" ""
.hy
.SH NAME
.PP
\f[B]mana\f[R] \[en] MANA family of commands for checkpointing MPI jobs
.SH SYNOPSIS
.PP
-\f[B]mana_launch\f[R] [--help] [--verbose] [--timing] [DMTCP_OPTIONS]
-[--ckptdir \f[I]DIR\f[R]] COMMAND [\f[I]ARGS\&...\f[R]]
+\f[B]mana_coordinator\f[R] [--help] [--verbose]
+[\f[I]DMTCP_OPTIONS\f[R]]
.PD 0
.P
.PD
-\f[B]mana_coordinator\f[R] [--help] [--verbose]
-[\f[I]DMTCP_OPTIONS\f[R]]
+\f[B]mana_launch\f[R] [--help] [--verbose] [--timing] [--gdb] [--ckptdir
+\f[I]DIR\f[R]] [\f[I]DMTCP_OPTIONS\f[R]] \f[I]PROGRAM\f[R]
+[\f[I]ARGS\&...\f[R]]
.PD 0
.P
.PD
-\f[B]mana_status\f[R] [--help] [--verbose] [\f[I]DMTCP_OPTIONS\f[R]]
+\f[B]mana_status\f[R] [--help] [--verbose] [\f[I]COMMAND\f[R]]
.PD 0
.P
.PD
-\f[B]mana_restart\f[R] [--help] [--verbose] [\f[I]DMTCP_OPTIONS\f[R]]
-[--restartdir \f[I]DIR\f[R]]
+\f[B]mana_restart\f[R] [--help] [--verbose] [--timing] [--gdb]
+[--restartdir \f[I]DIR\f[R]] [--ckptdir \f[I]DIR\f[R]]
+[\f[I]DMTCP_OPTIONS\f[R]]
.SH DESCRIPTION
.PP
\f[B]MANA\f[R] is a package that enables checkpoint-restart for MPI
@@ -32,62 +34,228 @@ Checkpointing\[rq].
It is designed to be compatible with most MPI implementations and most
underlying networks.
MANA is built on top of the DMTCP checkpointing package.
-.SH COMMAND LINE OPTIONS
+.PP
+A MANA job uses four commands.
+\f[C]mana_coordinator\f[R] starts the coordinator that the ranks connect
+to.
+\f[C]mana_launch\f[R] runs each rank of an MPI program under MANA.
+\f[C]mana_status\f[R] lists the ranks and asks the coordinator for a
+checkpoint.
+\f[C]mana_restart\f[R] restarts the ranks from their checkpoint.
+The MPI program needs no change and no relinking.
+The four commands are wrappers of DMTCP\[cq]s commands, and they accept
+DMTCP\[cq]s options too (see COMMANDS AND OPTIONS).
+.PP
+MANA\[cq]s website, , has the rest of
+MANA\[cq]s documentation: a user guide, a developer guide and the
+release notes.
+.SH CHECKPOINT AND RESTART
+.PP
+For example, with the example program \f[C]examples/ring\f[R] (see
+BUILDING) on one host, from MANA\[cq]s directory:
+.IP
+.nf
+\f[C]
+bin/mana_coordinator # 1. Start a coordinator.
+mpirun -np 4 bin/mana_launch examples/ring # 2. Run the program under MANA.
+bin/mana_status --bcheckpoint # 3. In another shell: checkpoint.
+\f[R]
+.fi
+.PP
+At step 3, each rank N writes its checkpoint into the directory
+\f[C]ckpt_rank_N\f[R] in the current directory.
+The command returns when the checkpoint is complete, and the program
+goes on.
+To stop the job and restart it from the checkpoint:
+.IP
+.nf
+\f[C]
+# 4. Stop the job, for example with Ctrl-C.
+bin/mana_coordinator # 5. Start a new coordinator.
+mpirun -np 4 bin/mana_restart # 6. Restart from ckpt_rank_*.
+\f[R]
+.fi
+.PP
+The restarted program goes on from where the checkpoint was taken.
+Restart with the same number of ranks.
+A coordinator exits when its last rank exits, so each run needs a new
+one.
+.PP
+Other ways to checkpoint:
+.IP \[bu] 2
+\f[C]bin/mana_status --kcheckpoint\f[R]: checkpoint, then stop the job
+(steps 3 and 4 in one command).
+.IP \[bu] 2
+\f[C]bin/mana_status --checkpoint\f[R]: checkpoint, and return at once,
+before the checkpoint is done.
+.IP \[bu] 2
+\f[C]bin/mana_coordinator -i SECONDS\f[R]: checkpoint every
+\f[I]SECONDS\f[R] seconds.
+.IP \[bu] 2
+From the program, with DMTCP\[cq]s API (see the directory
+\f[C]dmtcp/test/plugin/applic-initiated-ckpt\f[R]).
+.PP
+A new checkpoint replaces the previous one.
+\f[C]mana_launch\f[R] stops if the checkpoint directory already has
+\f[C]ckpt_rank_*\f[R] directories: move or delete an old checkpoint
+before a new run.
+.PP
+With Slurm, run the same commands in an allocation, with \f[C]srun\f[R]:
+.IP
+.nf
+\f[C]
+salloc -N 2 -t 01:00:00 # or your cluster\[aq]s command for an allocation
+bin/mana_coordinator -i 60
+srun -N 2 bin/mana_launch examples/ring
+bin/mana_coordinator -i 60
+srun -N 2 bin/mana_restart
+\f[R]
+.fi
+.SH COMMANDS AND OPTIONS
+.PP
+Each MANA command is a wrapper of a DMTCP command:
+\f[C]mana_coordinator\f[R] runs \f[C]dmtcp_coordinator\f[R],
+\f[C]mana_launch\f[R] runs \f[C]dmtcp_launch\f[R], \f[C]mana_status\f[R]
+runs \f[C]dmtcp_command\f[R], and \f[C]mana_restart\f[R] runs the
+restart of \f[C]dmtcp_restart\f[R] in MANA\[cq]s lower half.
+A MANA command handles the options below, and passes the others to its
+DMTCP command, so DMTCP\[cq]s options work too.
+MANA sets some of them itself: \f[C]--exit-on-last\f[R] and
+\f[C]--daemon\f[R] for \f[C]dmtcp_coordinator\f[R], and
+\f[C]--join-coordinator\f[R] and \f[C]--no-gzip\f[R] for
+\f[C]dmtcp_launch\f[R].
+For DMTCP\[cq]s options, see each DMTCP command\[cq]s \f[C]--help\f[R],
+DMTCP\[cq]s man pages (see SEE ALSO), and DMTCP\[cq]s documentation at
+.
+.SS mana_coordinator
+.PP
+Starts a DMTCP coordinator in the background, and writes its host and
+port to a file where the other commands find it (see FILES).
.TP
-\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R]
-Show additional DMTCP_OPTIONS for a MANA command.
+\f[B]\f[R]\f[C]-i SECONDS\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]--interval SECONDS\f[R]\f[B]\f[R]
+Checkpoint every \f[I]SECONDS\f[R] seconds (default: 0, no periodic
+checkpoints).
.TP
\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R]
-Display the underlying DMTCP command and DMTCP_OPTIONS used, and other
-info.
+Print the \f[C]dmtcp_coordinator\f[R] command that
+\f[C]mana_coordinator\f[R] runs.
+.TP
+\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R]
+Print the options of \f[C]dmtcp_coordinator\f[R].
+.SS mana_launch
+.PP
+Runs \f[I]PROGRAM\f[R] with \f[I]ARGS\f[R] under MANA.
+Run \f[C]mana_launch\f[R] under the MPI launcher, such as \f[C]srun\f[R]
+or \f[C]mpirun -np N\f[R], in place of \f[I]PROGRAM\f[R].
+A coordinator must be running.
+.TP
+\f[B]\f[R]\f[C]--ckptdir DIR\f[R]\f[B]\f[R]
+Write the checkpoints into \f[I]DIR\f[R], which must exist (default: the
+current directory).
.TP
\f[B]\f[R]\f[C]--timing\f[R]\f[B]\f[R]
-Print times to stderr for INIT, EXIT, and ckkpt-restart events.
-(stays active during both mana_launch and mana_restart)
-.SH MANA PROGRAM EXECUTION
+Rank 0 prints to stderr the time since the start at each step of a
+checkpoint, and at exit.
+.TP
+\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R]
+Print the \f[C]dmtcp_launch\f[R] command that \f[C]mana_launch\f[R]
+runs, and DMTCP\[cq]s notes and warnings, which are otherwise not
+printed.
+.TP
+\f[B]\f[R]\f[C]--gdb\f[R]\f[B]\f[R]
+Run the \f[C]dmtcp_launch\f[R] command under \f[C]gdb --args\f[R], to
+debug MANA itself.
+.TP
+\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R]
+Print a usage statement.
.PP
-Execute a MANA command with \f[C]--help\f[R] for a usage statement.
+\f[C]mana_launch\f[R] loads the plugins of DMTCP\[cq]s
+\f[C]--with-plugin\f[R] after MANA\[cq]s own.
+.SS mana_status
.PP
-Checkpoints may be invoked:
+Without \f[I]COMMAND\f[R], lists the ranks that are connected to the
+coordinator.
+A \f[I]COMMAND\f[R] asks the coordinator for an action:
+.TP
+\f[B]\f[R]\f[C]--checkpoint\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]-c\f[R]\f[B]\f[R]
+Checkpoint the job.
+The job goes on.
+.TP
+\f[B]\f[R]\f[C]--bcheckpoint\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]-bc\f[R]\f[B]\f[R]
+Checkpoint the job, and return when the checkpoint is done.
.TP
-\f[B]\f[R]\f[C]periodically\f[R]\f[B]\f[R]
-via the \f[C]--interval\f[R] or \f[C]-i\f[R] flag (units in seconds)
+\f[B]\f[R]\f[C]--kcheckpoint\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]-kc\f[R]\f[B]\f[R]
+Checkpoint the job, then stop it.
.TP
-\f[B]\f[R]\f[C]under program control\f[R]\f[B]\f[R]
-(see \f[C]dmtcp/test/plugin/applic-initiated-ckpt\f[R] directory)
+\f[B]\f[R]\f[C]-i SECONDS\f[R]\f[B]\f[R]
+Checkpoint every \f[I]SECONDS\f[R] seconds from now on (0: no periodic
+checkpoints).
+.TP
+\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R]
+Print the \f[C]dmtcp_command\f[R] command that \f[C]mana_status\f[R]
+runs.
.TP
-\f[B]\f[R]\f[C]externally\f[R]\f[B]\f[R]
-via \f[C]mana_status --checkpoint\f[R]
+\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R]
+Print the usage of \f[C]dmtcp_command\f[R], which lists the other
+commands.
+.SS mana_restart
.PP
-A typical workflow for using MANA after untar\[aq]ing is:
+Restarts a job from its checkpoint.
+Run \f[C]mana_restart\f[R] under the MPI launcher, with the same number
+of ranks as the job, after a new \f[C]mana_coordinator\f[R].
+Each rank N restarts from \f[C]ckpt_rank_N\f[R], and keeps the
+\f[C]MANA_P2P_WAIT\f[R] mode of the job.
+.TP
+\f[B]\f[R]\f[C]--restartdir DIR\f[R]\f[B]\f[R]
+Restart from the \f[C]ckpt_rank_*\f[R] directories in \f[I]DIR\f[R]
+(default: the current directory).
+\f[C]mana_restart\f[R] stops if a checkpoint there is incomplete (a
+\f[C].temp\f[R] file).
+.TP
+\f[B]\f[R]\f[C]--ckptdir DIR\f[R]\f[B]\f[R]
+Write later checkpoints into \f[I]DIR\f[R], which must exist.
+.TP
+\f[B]\f[R]\f[C]--timing\f[R]\f[B]\f[R]
+Rank 0 prints to stderr the time since the restart at the restart, at
+each step of a checkpoint, and at exit.
+.TP
+\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R]
+Print the command that \f[C]mana_restart\f[R] runs, and DMTCP\[cq]s
+notes and warnings, which are otherwise not printed.
+.TP
+\f[B]\f[R]\f[C]--gdb\f[R]\f[B]\f[R]
+Run the restart under \f[C]gdb --args\f[R], to debug MANA itself.
+.TP
+\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R]
+Print a usage statement.
+.SH BUILDING
.IP
.nf
\f[C]
-cd dmtcp-mana
-\&./configure --enable-debug
-make -j mana tests
-# Build the program with mpicc as usual; no relinking for MANA is needed.
-# 'make tests' also builds an example: examples/ring
-salloc -N 2 -q interactive -C haswell -t 01:00:00
-bin/mana_coordinator -i10
-srun -N 2 bin/mana_launch examples/ring
-bin/mana_coordinator -i10
-srun -N 2 bin/mana_restart
+git clone https://github.com/mpickpt/mana
+cd mana
+\&./configure
+make -j
+make tests # the tests, and the example examples/ring
\f[R]
.fi
.PP
-MANA supports most features of DMTCP, including:
-.IP \[bu] 2
-plugins to extend the functionality of DMTCP
-.IP \[bu] 2
-virtualization of pathname prefixes (plugin/pathvirt)
-.IP \[bu] 2
-modification of environment variables at restart (plugin/modify-env)
-.IP \[bu] 2
-distinct checkpoint filenames
-(\f[C]dmtcp_launch --enable-unique-checkpoint-filenames\f[R]).
-The default is to overwrite the last checkpoint.
-.SH ENVIRONMENT VARIABLES AND DEBUGGING
+\f[C]make\f[R] puts MANA\[cq]s commands in \f[C]bin/\f[R].
+To install MANA into another directory, run
+\f[C]./configure --prefix=DIR\f[R], \f[C]make\f[R] and
+\f[C]make install\f[R].
+The commands are then in \f[C]DIR/bin\f[R].
+.SH NOTES
+.PP
+MANA supports most features of DMTCP, including plugins
+(\f[C]mana_launch --with-plugin\f[R]) and the modification of
+environment variables at restart (\f[C]dmtcp/plugin/modify-env\f[R]).
+.PP
+MANA does not support DMTCP\[cq]s distinct checkpoint filenames
+(\f[C]--enable-unique-checkpoint-filenames\f[R]): \f[C]mana_restart\f[R]
+takes one image from each \f[C]ckpt_rank_N\f[R] directory, and not
+always the newest.
+.SH ENVIRONMENT
.TP
\f[B]\f[R]\f[C]MANA_P2P_WAIT\f[R]\f[B]\f[R]
How \f[C]MPI_Send\f[R] and \f[C]MPI_Recv\f[R] wait for their message.
@@ -106,36 +274,26 @@ In \f[C]blocking\f[R] mode, MANA initializes the MPI library with
MANA reads the variable at \f[C]MPI_Init\f[R] and keeps the mode after
restart.
.TP
-\f[B]\f[R]\f[C]MANA_DEBUG\f[R]\f[B]\f[R]
-MANA will print to stderr extra information to help developers debug
-MANA.
-.TP
\f[B]\f[R]\f[C]MANA_PRELOAD\f[R]\f[B]\f[R]
A colon-separated list of libraries that \f[C]mana_launch\f[R] loads
ahead of MANA, so that they intercept MPI calls before MANA does (e.g.,
a PMPI profiler).
A library in \f[C]LD_PRELOAD\f[R] loads after them.
.TP
-\f[B]\f[R]\f[C]DMTCP_MANA_PAUSE\f[R]\f[B] or \f[R]\f[C]DMTCP_LAUNCH_PAUSE\f[R]\f[B]\f[R]
-DMTCP/MANA will pause during launch to allow \f[C]gdb attach\f[R] (GDB
-must be on same node.)
-.TP
-\f[B]\f[R]\f[C]DMTCP_MANA_PAUSE\f[R]\f[B]\f[R]
-DMTCP/MANA will pause very early in restart, inside
-\f[C]mtcp/mtcp_restart.c\f[R] shortly before calling
-\f[C]splitProcess()\f[R] (intended for developers)
+\f[B]\f[R]\f[C]DMTCP_LOG_LEVEL\f[R]\f[B]\f[R]
+How much DMTCP prints to stderr: \f[C]error\f[R], \f[C]warn\f[R],
+\f[C]note\f[R] or \f[C]trace\f[R].
.TP
\f[B]\f[R]\f[C]DMTCP_RESTART_PAUSE\f[R]\f[B]\f[R]
-DMTCP/MANA will pause during restart, before resuming execution, to
-allow \f[C]gdb attach\f[R] (GDB must be on same node.)
-.PP
-\f[B]\f[CB]INSPECTING MANA for DEBUGGING\f[B]\f[R]
+DMTCP/MANA will pause during restart, at the stage given by the value (1
+to 7), to allow \f[C]gdb attach\f[R] (GDB must be on same node.)
+.SH DEBUGGING
.PP
To see status of ranks (especially during checkpoint), try:
.IP
.nf
\f[C]
-bin/mana_status --list
+bin/mana_status
\f[R]
.fi
.PP
@@ -143,16 +301,16 @@ To inspect a checkpoint image from, for example, Rank \f[C]0\f[R], try:
.IP
.nf
\f[C]
-dmtcp/util/readdmtcp.sh ckpt_rank0/ckpt_*.dmtcp
+dmtcp/util/readdmtcp.sh ckpt_rank_0/ckpt_*.dmtcp
\f[R]
.fi
.PP
-To debug during restart at stage `N', replace `N' Below, and execute:
+To debug during restart at stage `N', replace `N' below, and execute:
.IP
.nf
\f[C]
-srun ... env DMTCP\[rs]_RESTART\[rs]_PAUSE=N mana\[rs]_restart ... APPLICATION &
-gdb -p APPLICATION PID # from a different terminal
+srun ... env DMTCP_RESTART_PAUSE=N bin/mana_restart &
+gdb -p PID # PID of the paused rank, from another terminal
\f[R]
.fi
.PP
@@ -162,7 +320,7 @@ following:
.nf
\f[C]
(gdb) source dmtcp/util/gdb-dmtcp-utils.py
-(gdb) load-symbol-library ADDRESS-OR-LIBRARY-SUBSTRING
+(gdb) load-symbols-library ADDRESS-OR-LIBRARY-SUBSTRING
(gdb) dmtcp # to see other available commands
\f[R]
.fi
@@ -171,18 +329,37 @@ If you are debugging the lower half internals of MANA, you may need:
.IP
.nf
\f[C]
-(gdb) file bin/lh_proxy
+(gdb) file bin/lower-half
\f[R]
.fi
.PP
If debugging the low-level restart process, please read
-restart/plugin/README.
+src/lower-half/README.
+.SH FILES
+.TP
+\f[B]\f[R]\f[C]\[ti]/.mana.rc\f[R]\f[B]\f[R], or \f[B]\f[R]\f[C]\[ti]/.mana-slurm-JOBID.rc\f[R]\f[B]\f[R] in a Slurm job
+The coordinator\[cq]s host and port, written by
+\f[C]mana_coordinator\f[R] and read by the other commands.
+\f[C]mana_launch\f[R] and \f[C]mana_restart\f[R] delete these files when
+they are more than a week old.
+.TP
+\f[B]\f[R]\f[C]ckpt_rank_N/\f[R]\f[B]\f[R]
+The checkpoint of rank N, in the checkpoint directory: its image
+\f[C]ckpt_*.dmtcp\f[R] and the file \f[C]header.mana\f[R].
.SH BUGS
.PP
Report bugs in MANA to: https://github.com/mpickpt/mana
.SH SEE ALSO
.PP
-\f[B]MANA home page:\f[R] ,
+\f[B]MANA website:\f[R] ,
+.PD 0
+.P
+.PD
+\f[B]MANA source code:\f[R] ,
+.PD 0
+.P
+.PD
+\f[B]DMTCP home page:\f[R] ,
.PD 0
.P
.PD
diff --git a/manpages/mana.1.md b/manpages/mana.1.md
index b405946ce..f2d1f781c 100644
--- a/manpages/mana.1.md
+++ b/manpages/mana.1.md
@@ -1,5 +1,5 @@
---
-date: 31 March 2022
+date: 5 November 2026
section: 1
title: MANA
description: MPI-Agnostic Network-Agnostic Checkpointing
@@ -11,10 +11,10 @@ description: MPI-Agnostic Network-Agnostic Checkpointing
# SYNOPSIS
-**mana_launch** [\--help] [\--verbose] [\--timing] [DMTCP_OPTIONS] [\--ckptdir *DIR*] COMMAND [*ARGS...*]\
**mana_coordinator** [\--help] [\--verbose] [*DMTCP_OPTIONS*]\
-**mana_status** [\--help] [\--verbose] [*DMTCP_OPTIONS*]\
-**mana_restart** [\--help] [\--verbose] [*DMTCP_OPTIONS*] [\--restartdir *DIR*]
+**mana_launch** [\--help] [\--verbose] [\--timing] [\--gdb] [\--ckptdir *DIR*] [*DMTCP_OPTIONS*] *PROGRAM* [*ARGS...*]\
+**mana_status** [\--help] [\--verbose] [*COMMAND*]\
+**mana_restart** [\--help] [\--verbose] [\--timing] [\--gdb] [\--restartdir *DIR*] [\--ckptdir *DIR*] [*DMTCP_OPTIONS*]
# DESCRIPTION
@@ -24,56 +24,197 @@ Checkpointing". It is designed to be compatible with most MPI
implementations and most underlying networks. MANA is built on top of
the DMTCP checkpointing package.
-# COMMAND LINE OPTIONS
+A MANA job uses four commands. `mana_coordinator` starts the coordinator
+that the ranks connect to. `mana_launch` runs each rank of an MPI program
+under MANA. `mana_status` lists the ranks and asks the coordinator for a
+checkpoint. `mana_restart` restarts the ranks from their checkpoint. The MPI
+program needs no change and no relinking. The four commands are wrappers of
+DMTCP's commands, and they accept DMTCP's options too (see COMMANDS AND
+OPTIONS).
-**`--help`**
-: Show additional DMTCP_OPTIONS for a MANA command.
+MANA's website, \, has the rest of MANA's
+documentation: a user guide, a developer guide and the release notes.
-**`--verbose`**
-: Display the underlying DMTCP command and DMTCP_OPTIONS used, and other info.
+# CHECKPOINT AND RESTART
-**`--timing`**
-: Print times to stderr for INIT, EXIT, and ckkpt-restart events.
- (stays active during both mana_launch and mana_restart)
+For example, with the example program `examples/ring` (see BUILDING) on one
+host, from MANA's directory:
+
+```bash
+bin/mana_coordinator # 1. Start a coordinator.
+mpirun -np 4 bin/mana_launch examples/ring # 2. Run the program under MANA.
+bin/mana_status --bcheckpoint # 3. In another shell: checkpoint.
+```
-# MANA PROGRAM EXECUTION
+At step 3, each rank N writes its checkpoint into the directory
+`ckpt_rank_N` in the current directory. The command returns when the
+checkpoint is complete, and the program goes on. To stop the job and
+restart it from the checkpoint:
-Execute a MANA command with `--help` for a usage statement.
+```bash
+# 4. Stop the job, for example with Ctrl-C.
+bin/mana_coordinator # 5. Start a new coordinator.
+mpirun -np 4 bin/mana_restart # 6. Restart from ckpt_rank_*.
+```
-Checkpoints may be invoked:
+The restarted program goes on from where the checkpoint was taken. Restart
+with the same number of ranks. A coordinator exits when its last rank
+exits, so each run needs a new one.
-**`periodically`**
-: via the `--interval` or `-i` flag (units in seconds)
+Other ways to checkpoint:
-**`under program control`**
-: (see `dmtcp/test/plugin/applic-initiated-ckpt` directory)
+* `bin/mana_status --kcheckpoint`: checkpoint, then stop the job (steps 3
+ and 4 in one command).
+* `bin/mana_status --checkpoint`: checkpoint, and return at once, before
+ the checkpoint is done.
+* `bin/mana_coordinator -i SECONDS`: checkpoint every *SECONDS* seconds.
+* From the program, with DMTCP's API (see the directory
+ `dmtcp/test/plugin/applic-initiated-ckpt`).
-**`externally`**
-: via `mana_status --checkpoint`
+A new checkpoint replaces the previous one. `mana_launch` stops if the
+checkpoint directory already has `ckpt_rank_*` directories: move or delete
+an old checkpoint before a new run.
-A typical workflow for using MANA after untar\'ing is:
+With Slurm, run the same commands in an allocation, with `srun`:
```bash
-cd dmtcp-mana
-./configure --enable-debug
-make -j mana tests
-# Build the program with mpicc as usual; no relinking for MANA is needed.
-# 'make tests' also builds an example: examples/ring
-salloc -N 2 -q interactive -C haswell -t 01:00:00
-bin/mana_coordinator -i10
+salloc -N 2 -t 01:00:00 # or your cluster's command for an allocation
+bin/mana_coordinator -i 60
srun -N 2 bin/mana_launch examples/ring
-bin/mana_coordinator -i10
+bin/mana_coordinator -i 60
srun -N 2 bin/mana_restart
```
-MANA supports most features of DMTCP, including:
+# COMMANDS AND OPTIONS
+
+Each MANA command is a wrapper of a DMTCP command: `mana_coordinator` runs
+`dmtcp_coordinator`, `mana_launch` runs `dmtcp_launch`, `mana_status` runs
+`dmtcp_command`, and `mana_restart` runs the restart of `dmtcp_restart` in
+MANA's lower half. A MANA command handles the options below, and passes the
+others to its DMTCP command, so DMTCP's options work too. MANA sets some of
+them itself: `--exit-on-last` and `--daemon` for `dmtcp_coordinator`, and
+`--join-coordinator` and `--no-gzip` for `dmtcp_launch`. For DMTCP's
+options, see each DMTCP command's `--help`, DMTCP's man pages (see SEE
+ALSO), and DMTCP's documentation at \.
+
+## mana_coordinator
+
+Starts a DMTCP coordinator in the background, and writes its host and port
+to a file where the other commands find it (see FILES).
+
+**`-i SECONDS`**, **`--interval SECONDS`**
+: Checkpoint every *SECONDS* seconds (default: 0, no periodic checkpoints).
+
+**`--verbose`**
+: Print the `dmtcp_coordinator` command that `mana_coordinator` runs.
+
+**`--help`**
+: Print the options of `dmtcp_coordinator`.
+
+## mana_launch
+
+Runs *PROGRAM* with *ARGS* under MANA. Run `mana_launch` under the MPI
+launcher, such as `srun` or `mpirun -np N`, in place of *PROGRAM*. A
+coordinator must be running.
+
+**`--ckptdir DIR`**
+: Write the checkpoints into *DIR*, which must exist (default: the current
+ directory).
+
+**`--timing`**
+: Rank 0 prints to stderr the time since the start at each step of a
+ checkpoint, and at exit.
+
+**`--verbose`**
+: Print the `dmtcp_launch` command that `mana_launch` runs, and DMTCP's notes
+ and warnings, which are otherwise not printed.
+
+**`--gdb`**
+: Run the `dmtcp_launch` command under `gdb --args`, to debug MANA itself.
+
+**`--help`**
+: Print a usage statement.
+
+`mana_launch` loads the plugins of DMTCP's `--with-plugin` after MANA's own.
+
+## mana_status
+
+Without *COMMAND*, lists the ranks that are connected to the coordinator. A
+*COMMAND* asks the coordinator for an action:
+
+**`--checkpoint`**, **`-c`**
+: Checkpoint the job. The job goes on.
+
+**`--bcheckpoint`**, **`-bc`**
+: Checkpoint the job, and return when the checkpoint is done.
+
+**`--kcheckpoint`**, **`-kc`**
+: Checkpoint the job, then stop it.
+
+**`-i SECONDS`**
+: Checkpoint every *SECONDS* seconds from now on (0: no periodic
+ checkpoints).
+
+**`--verbose`**
+: Print the `dmtcp_command` command that `mana_status` runs.
+
+**`--help`**
+: Print the usage of `dmtcp_command`, which lists the other commands.
+
+## mana_restart
+
+Restarts a job from its checkpoint. Run `mana_restart` under the MPI
+launcher, with the same number of ranks as the job, after a new
+`mana_coordinator`. Each rank N restarts from `ckpt_rank_N`, and keeps the
+`MANA_P2P_WAIT` mode of the job.
+
+**`--restartdir DIR`**
+: Restart from the `ckpt_rank_*` directories in *DIR* (default: the current
+ directory). `mana_restart` stops if a checkpoint there is incomplete (a
+ `.temp` file).
+
+**`--ckptdir DIR`**
+: Write later checkpoints into *DIR*, which must exist.
+
+**`--timing`**
+: Rank 0 prints to stderr the time since the restart at the restart, at
+ each step of a checkpoint, and at exit.
+
+**`--verbose`**
+: Print the command that `mana_restart` runs, and DMTCP's notes and
+ warnings, which are otherwise not printed.
+
+**`--gdb`**
+: Run the restart under `gdb --args`, to debug MANA itself.
+
+**`--help`**
+: Print a usage statement.
+
+# BUILDING
+
+```bash
+git clone https://github.com/mpickpt/mana
+cd mana
+./configure
+make -j
+make tests # the tests, and the example examples/ring
+```
+
+`make` puts MANA's commands in `bin/`. To install MANA into another
+directory, run `./configure --prefix=DIR`, `make` and `make install`. The
+commands are then in `DIR/bin`.
+
+# NOTES
-* plugins to extend the functionality of DMTCP
-* virtualization of pathname prefixes (plugin/pathvirt)
-* modification of environment variables at restart (plugin/modify-env)
-* distinct checkpoint filenames (`dmtcp_launch --enable-unique-checkpoint-filenames`). The default is to overwrite the last checkpoint.
+MANA supports most features of DMTCP, including plugins
+(`mana_launch --with-plugin`) and the modification of environment variables
+at restart (`dmtcp/plugin/modify-env`).
-# ENVIRONMENT VARIABLES AND DEBUGGING
+MANA does not support DMTCP's distinct checkpoint filenames
+(`--enable-unique-checkpoint-filenames`): `mana_restart` takes one image from
+each `ckpt_rank_N` directory, and not always the newest.
+
+# ENVIRONMENT
**`MANA_P2P_WAIT`**
@@ -88,58 +229,61 @@ MANA supports most features of DMTCP, including:
can fail at a checkpoint (e.g., over UCX). MANA reads the variable at
`MPI_Init` and keeps the mode after restart.
-**`MANA_DEBUG`**
-
-: MANA will print to stderr extra information to help developers debug MANA.
-
**`MANA_PRELOAD`**
: A colon-separated list of libraries that `mana_launch` loads ahead of MANA,
so that they intercept MPI calls before MANA does (e.g., a PMPI profiler).
A library in `LD_PRELOAD` loads after them.
-**`DMTCP_MANA_PAUSE` or `DMTCP_LAUNCH_PAUSE`**
-
-: DMTCP/MANA will pause during launch to allow `gdb attach` (GDB must be on same node.)
+**`DMTCP_LOG_LEVEL`**
-**`DMTCP_MANA_PAUSE`**
-
-: DMTCP/MANA will pause very early in restart, inside
- `mtcp/mtcp_restart.c` shortly before calling `splitProcess()` (intended
- for developers)
+: How much DMTCP prints to stderr: `error`, `warn`, `note` or `trace`.
**`DMTCP_RESTART_PAUSE`**
-: DMTCP/MANA will pause during restart, before resuming execution, to
- allow `gdb attach` (GDB must be on same node.)
+: DMTCP/MANA will pause during restart, at the stage given by the value (1
+ to 7), to allow `gdb attach` (GDB must be on same node.)
-**`INSPECTING MANA for DEBUGGING`**
+# DEBUGGING
To see status of ranks (especially during checkpoint), try:
- bin/mana_status --list
+ bin/mana_status
To inspect a checkpoint image from, for example, Rank `0`, try:
- dmtcp/util/readdmtcp.sh ckpt_rank0/ckpt_*.dmtcp
+ dmtcp/util/readdmtcp.sh ckpt_rank_0/ckpt_*.dmtcp
-To debug during restart at stage 'N', replace 'N' Below, and execute:
+To debug during restart at stage 'N', replace 'N' below, and execute:
- srun ... env DMTCP\_RESTART\_PAUSE=N mana\_restart ... APPLICATION &
- gdb -p APPLICATION PID # from a different terminal
+ srun ... env DMTCP_RESTART_PAUSE=N bin/mana_restart &
+ gdb -p PID # PID of the paused rank, from another terminal
To see the stack, you then may or may not need to try some of the
following:
(gdb) source dmtcp/util/gdb-dmtcp-utils.py
- (gdb) load-symbol-library ADDRESS-OR-LIBRARY-SUBSTRING
+ (gdb) load-symbols-library ADDRESS-OR-LIBRARY-SUBSTRING
(gdb) dmtcp # to see other available commands
If you are debugging the lower half internals of MANA, you may need:
- (gdb) file bin/lh_proxy
+ (gdb) file bin/lower-half
+
+If debugging the low-level restart process, please read src/lower-half/README.
+
+# FILES
+
+**`~/.mana.rc`**, or **`~/.mana-slurm-JOBID.rc`** in a Slurm job
+
+: The coordinator's host and port, written by `mana_coordinator` and read by
+ the other commands. `mana_launch` and `mana_restart` delete these files
+ when they are more than a week old.
+
+**`ckpt_rank_N/`**
-If debugging the low-level restart process, please read restart/plugin/README.
+: The checkpoint of rank N, in the checkpoint directory: its image
+ `ckpt_*.dmtcp` and the file `header.mana`.
# BUGS
@@ -147,6 +291,8 @@ Report bugs in MANA to: https://github.com/mpickpt/mana
# SEE ALSO
-**MANA home page:** \, \
+**MANA website:** \, \
+**MANA source code:** \, \
+**DMTCP home page:** \, \
**dmtcp**(1), **dmtcp_coordinator**(1), **dmtcp_launch**(1), \
**dmtcp_restart**(1), **dmtcp_command**(1)