From 6e126553e7aa8ce23a612c72a7bbbe98d9433789 Mon Sep 17 00:00:00 2001 From: Yao Xu Date: Thu, 5 Nov 2026 15:42:53 -0500 Subject: [PATCH 1/2] Update the man page for the current MANA The man page still described MANA as of 2022, and it did not say how to take a checkpoint or which command takes which option. Rewrite it for the current commands and build: - A new section, CHECKPOINT AND RESTART, shows step by step how to checkpoint a job and restart it, with examples/ring on one host and with Slurm, and lists the other ways to checkpoint. - A new section, COMMANDS AND OPTIONS, describes each command and the options that it takes, as the scripts handle them: mana_status's checkpoint commands, --gdb, and the checks of --ckptdir and --restartdir. It says that each command is a wrapper of a DMTCP command and passes it the other options, so DMTCP's options work too, and points to DMTCP's documentation. A FILES section names the coordinator's .rc file and the ckpt_rank_N directories. - BUILDING clones MANA's repository instead of untarring dmtcp-mana, runs ./configure without --enable-debug (which builds with -O0), builds the tests with 'make tests', and says how to install MANA. The salloc line no longer asks for Cori's haswell nodes. - MANA_DEBUG, DMTCP_MANA_PAUSE and DMTCP_LAUNCH_PAUSE are removed: nothing in MANA or DMTCP reads them. DMTCP_LOG_LEVEL is added. - The pathvirt plugin is removed: DMTCP no longer builds it. Unique checkpoint filenames are listed as not supported: mana_restart takes one image of each rank, and not always the newest. - The lower half is bin/lower-half, not bin/lh_proxy. A rank's directory is ckpt_rank_0, not ckpt_rank0. The gdb command is load-symbols-library, and the README on the restart is src/lower-half/README. - DESCRIPTION and SEE ALSO point to MANA's website, https://mpickpt.github.io/mana/, for the rest of MANA's documentation. SEE ALSO calls the GitHub repository "MANA source code". Tested: the steps of CHECKPOINT AND RESTART with examples/ring on 4 ranks (the restart goes on from the checkpoint's step and ends), and mana_status --kcheckpoint, -bc and -i. mana.1 is regenerated from mana.1.md with pandoc 2.9.2.1, the version that made the tracked file. --- manpages/mana.1 | 321 +++++++++++++++++++++++++++++++++++---------- manpages/mana.1.md | 266 ++++++++++++++++++++++++++++--------- 2 files changed, 455 insertions(+), 132 deletions(-) diff --git a/manpages/mana.1 b/manpages/mana.1 index 8e5820908..2975993cd 100644 --- a/manpages/mana.1 +++ b/manpages/mana.1 @@ -1,28 +1,30 @@ .\" Automatically generated by Pandoc 2.9.2.1 .\" -.TH "MANA" "1" "31 March 2022" "" "" +.TH "MANA" "1" "5 November 2026" "" "" .hy .SH NAME .PP \f[B]mana\f[R] \[en] MANA family of commands for checkpointing MPI jobs .SH SYNOPSIS .PP -\f[B]mana_launch\f[R] [--help] [--verbose] [--timing] [DMTCP_OPTIONS] -[--ckptdir \f[I]DIR\f[R]] COMMAND [\f[I]ARGS\&...\f[R]] +\f[B]mana_coordinator\f[R] [--help] [--verbose] +[\f[I]DMTCP_OPTIONS\f[R]] .PD 0 .P .PD -\f[B]mana_coordinator\f[R] [--help] [--verbose] -[\f[I]DMTCP_OPTIONS\f[R]] +\f[B]mana_launch\f[R] [--help] [--verbose] [--timing] [--gdb] [--ckptdir +\f[I]DIR\f[R]] [\f[I]DMTCP_OPTIONS\f[R]] \f[I]PROGRAM\f[R] +[\f[I]ARGS\&...\f[R]] .PD 0 .P .PD -\f[B]mana_status\f[R] [--help] [--verbose] [\f[I]DMTCP_OPTIONS\f[R]] +\f[B]mana_status\f[R] [--help] [--verbose] [\f[I]COMMAND\f[R]] .PD 0 .P .PD -\f[B]mana_restart\f[R] [--help] [--verbose] [\f[I]DMTCP_OPTIONS\f[R]] -[--restartdir \f[I]DIR\f[R]] +\f[B]mana_restart\f[R] [--help] [--verbose] [--timing] [--gdb] +[--restartdir \f[I]DIR\f[R]] [--ckptdir \f[I]DIR\f[R]] +[\f[I]DMTCP_OPTIONS\f[R]] .SH DESCRIPTION .PP \f[B]MANA\f[R] is a package that enables checkpoint-restart for MPI @@ -32,62 +34,228 @@ Checkpointing\[rq]. It is designed to be compatible with most MPI implementations and most underlying networks. MANA is built on top of the DMTCP checkpointing package. -.SH COMMAND LINE OPTIONS +.PP +A MANA job uses four commands. +\f[C]mana_coordinator\f[R] starts the coordinator that the ranks connect +to. +\f[C]mana_launch\f[R] runs each rank of an MPI program under MANA. +\f[C]mana_status\f[R] lists the ranks and asks the coordinator for a +checkpoint. +\f[C]mana_restart\f[R] restarts the ranks from their checkpoint. +The MPI program needs no change and no relinking. +The four commands are wrappers of DMTCP\[cq]s commands, and they accept +DMTCP\[cq]s options too (see COMMANDS AND OPTIONS). +.PP +MANA\[cq]s website, , has the rest of +MANA\[cq]s documentation: a user guide, a developer guide and the +release notes. +.SH CHECKPOINT AND RESTART +.PP +For example, with the example program \f[C]examples/ring\f[R] (see +BUILDING) on one host, from MANA\[cq]s directory: +.IP +.nf +\f[C] +bin/mana_coordinator # 1. Start a coordinator. +mpirun -np 4 bin/mana_launch examples/ring # 2. Run the program under MANA. +bin/mana_status --bcheckpoint # 3. In another shell: checkpoint. +\f[R] +.fi +.PP +At step 3, each rank N writes its checkpoint into the directory +\f[C]ckpt_rank_N\f[R] in the current directory. +The command returns when the checkpoint is complete, and the program +goes on. +To stop the job and restart it from the checkpoint: +.IP +.nf +\f[C] +# 4. Stop the job, for example with Ctrl-C. +bin/mana_coordinator # 5. Start a new coordinator. +mpirun -np 4 bin/mana_restart # 6. Restart from ckpt_rank_*. +\f[R] +.fi +.PP +The restarted program goes on from where the checkpoint was taken. +Restart with the same number of ranks. +A coordinator exits when its last rank exits, so each run needs a new +one. +.PP +Other ways to checkpoint: +.IP \[bu] 2 +\f[C]bin/mana_status --kcheckpoint\f[R]: checkpoint, then stop the job +(steps 3 and 4 in one command). +.IP \[bu] 2 +\f[C]bin/mana_status --checkpoint\f[R]: checkpoint, and return at once, +before the checkpoint is done. +.IP \[bu] 2 +\f[C]bin/mana_coordinator -i SECONDS\f[R]: checkpoint every +\f[I]SECONDS\f[R] seconds. +.IP \[bu] 2 +From the program, with DMTCP\[cq]s API (see the directory +\f[C]dmtcp/test/plugin/applic-initiated-ckpt\f[R]). +.PP +A new checkpoint replaces the previous one. +\f[C]mana_launch\f[R] stops if the checkpoint directory already has +\f[C]ckpt_rank_*\f[R] directories: move or delete an old checkpoint +before a new run. +.PP +With Slurm, run the same commands in an allocation, with \f[C]srun\f[R]: +.IP +.nf +\f[C] +salloc -N 2 -t 01:00:00 # or your cluster\[aq]s command for an allocation +bin/mana_coordinator -i 60 +srun -N 2 bin/mana_launch examples/ring +bin/mana_coordinator -i 60 +srun -N 2 bin/mana_restart +\f[R] +.fi +.SH COMMANDS AND OPTIONS +.PP +Each MANA command is a wrapper of a DMTCP command: +\f[C]mana_coordinator\f[R] runs \f[C]dmtcp_coordinator\f[R], +\f[C]mana_launch\f[R] runs \f[C]dmtcp_launch\f[R], \f[C]mana_status\f[R] +runs \f[C]dmtcp_command\f[R], and \f[C]mana_restart\f[R] runs the +restart of \f[C]dmtcp_restart\f[R] in MANA\[cq]s lower half. +A MANA command handles the options below, and passes the others to its +DMTCP command, so DMTCP\[cq]s options work too. +MANA sets some of them itself: \f[C]--exit-on-last\f[R] and +\f[C]--daemon\f[R] for \f[C]dmtcp_coordinator\f[R], and +\f[C]--join-coordinator\f[R] and \f[C]--no-gzip\f[R] for +\f[C]dmtcp_launch\f[R]. +For DMTCP\[cq]s options, see each DMTCP command\[cq]s \f[C]--help\f[R], +DMTCP\[cq]s man pages (see SEE ALSO), and DMTCP\[cq]s documentation at +. +.SS mana_coordinator +.PP +Starts a DMTCP coordinator in the background, and writes its host and +port to a file where the other commands find it (see FILES). .TP -\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R] -Show additional DMTCP_OPTIONS for a MANA command. +\f[B]\f[R]\f[C]-i SECONDS\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]--interval SECONDS\f[R]\f[B]\f[R] +Checkpoint every \f[I]SECONDS\f[R] seconds (default: 0, no periodic +checkpoints). .TP \f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R] -Display the underlying DMTCP command and DMTCP_OPTIONS used, and other -info. +Print the \f[C]dmtcp_coordinator\f[R] command that +\f[C]mana_coordinator\f[R] runs. +.TP +\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R] +Print the options of \f[C]dmtcp_coordinator\f[R]. +.SS mana_launch +.PP +Runs \f[I]PROGRAM\f[R] with \f[I]ARGS\f[R] under MANA. +Run \f[C]mana_launch\f[R] under the MPI launcher, such as \f[C]srun\f[R] +or \f[C]mpirun -np N\f[R], in place of \f[I]PROGRAM\f[R]. +A coordinator must be running. +.TP +\f[B]\f[R]\f[C]--ckptdir DIR\f[R]\f[B]\f[R] +Write the checkpoints into \f[I]DIR\f[R], which must exist (default: the +current directory). .TP \f[B]\f[R]\f[C]--timing\f[R]\f[B]\f[R] -Print times to stderr for INIT, EXIT, and ckkpt-restart events. -(stays active during both mana_launch and mana_restart) -.SH MANA PROGRAM EXECUTION +Rank 0 prints to stderr the time since the start at each step of a +checkpoint, and at exit. +.TP +\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R] +Print the \f[C]dmtcp_launch\f[R] command that \f[C]mana_launch\f[R] +runs, and DMTCP\[cq]s notes and warnings, which are otherwise not +printed. +.TP +\f[B]\f[R]\f[C]--gdb\f[R]\f[B]\f[R] +Run the \f[C]dmtcp_launch\f[R] command under \f[C]gdb --args\f[R], to +debug MANA itself. +.TP +\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R] +Print a usage statement. .PP -Execute a MANA command with \f[C]--help\f[R] for a usage statement. +\f[C]mana_launch\f[R] loads the plugins of DMTCP\[cq]s +\f[C]--with-plugin\f[R] after MANA\[cq]s own. +.SS mana_status .PP -Checkpoints may be invoked: +Without \f[I]COMMAND\f[R], lists the ranks that are connected to the +coordinator. +A \f[I]COMMAND\f[R] asks the coordinator for an action: +.TP +\f[B]\f[R]\f[C]--checkpoint\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]-c\f[R]\f[B]\f[R] +Checkpoint the job. +The job goes on. +.TP +\f[B]\f[R]\f[C]--bcheckpoint\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]-bc\f[R]\f[B]\f[R] +Checkpoint the job, and return when the checkpoint is done. .TP -\f[B]\f[R]\f[C]periodically\f[R]\f[B]\f[R] -via the \f[C]--interval\f[R] or \f[C]-i\f[R] flag (units in seconds) +\f[B]\f[R]\f[C]--kcheckpoint\f[R]\f[B]\f[R], \f[B]\f[R]\f[C]-kc\f[R]\f[B]\f[R] +Checkpoint the job, then stop it. .TP -\f[B]\f[R]\f[C]under program control\f[R]\f[B]\f[R] -(see \f[C]dmtcp/test/plugin/applic-initiated-ckpt\f[R] directory) +\f[B]\f[R]\f[C]-i SECONDS\f[R]\f[B]\f[R] +Checkpoint every \f[I]SECONDS\f[R] seconds from now on (0: no periodic +checkpoints). +.TP +\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R] +Print the \f[C]dmtcp_command\f[R] command that \f[C]mana_status\f[R] +runs. .TP -\f[B]\f[R]\f[C]externally\f[R]\f[B]\f[R] -via \f[C]mana_status --checkpoint\f[R] +\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R] +Print the usage of \f[C]dmtcp_command\f[R], which lists the other +commands. +.SS mana_restart .PP -A typical workflow for using MANA after untar\[aq]ing is: +Restarts a job from its checkpoint. +Run \f[C]mana_restart\f[R] under the MPI launcher, with the same number +of ranks as the job, after a new \f[C]mana_coordinator\f[R]. +Each rank N restarts from \f[C]ckpt_rank_N\f[R], and keeps the +\f[C]MANA_P2P_WAIT\f[R] mode of the job. +.TP +\f[B]\f[R]\f[C]--restartdir DIR\f[R]\f[B]\f[R] +Restart from the \f[C]ckpt_rank_*\f[R] directories in \f[I]DIR\f[R] +(default: the current directory). +\f[C]mana_restart\f[R] stops if a checkpoint there is incomplete (a +\f[C].temp\f[R] file). +.TP +\f[B]\f[R]\f[C]--ckptdir DIR\f[R]\f[B]\f[R] +Write later checkpoints into \f[I]DIR\f[R], which must exist. +.TP +\f[B]\f[R]\f[C]--timing\f[R]\f[B]\f[R] +Rank 0 prints to stderr the time since the restart at the restart, at +each step of a checkpoint, and at exit. +.TP +\f[B]\f[R]\f[C]--verbose\f[R]\f[B]\f[R] +Print the command that \f[C]mana_restart\f[R] runs, and DMTCP\[cq]s +notes and warnings, which are otherwise not printed. +.TP +\f[B]\f[R]\f[C]--gdb\f[R]\f[B]\f[R] +Run the restart under \f[C]gdb --args\f[R], to debug MANA itself. +.TP +\f[B]\f[R]\f[C]--help\f[R]\f[B]\f[R] +Print a usage statement. +.SH BUILDING .IP .nf \f[C] -cd dmtcp-mana -\&./configure --enable-debug -make -j mana tests -# Build the program with mpicc as usual; no relinking for MANA is needed. -# 'make tests' also builds an example: examples/ring -salloc -N 2 -q interactive -C haswell -t 01:00:00 -bin/mana_coordinator -i10 -srun -N 2 bin/mana_launch examples/ring -bin/mana_coordinator -i10 -srun -N 2 bin/mana_restart +git clone https://github.com/mpickpt/mana +cd mana +\&./configure +make -j +make tests # the tests, and the example examples/ring \f[R] .fi .PP -MANA supports most features of DMTCP, including: -.IP \[bu] 2 -plugins to extend the functionality of DMTCP -.IP \[bu] 2 -virtualization of pathname prefixes (plugin/pathvirt) -.IP \[bu] 2 -modification of environment variables at restart (plugin/modify-env) -.IP \[bu] 2 -distinct checkpoint filenames -(\f[C]dmtcp_launch --enable-unique-checkpoint-filenames\f[R]). -The default is to overwrite the last checkpoint. -.SH ENVIRONMENT VARIABLES AND DEBUGGING +\f[C]make\f[R] puts MANA\[cq]s commands in \f[C]bin/\f[R]. +To install MANA into another directory, run +\f[C]./configure --prefix=DIR\f[R], \f[C]make\f[R] and +\f[C]make install\f[R]. +The commands are then in \f[C]DIR/bin\f[R]. +.SH NOTES +.PP +MANA supports most features of DMTCP, including plugins +(\f[C]mana_launch --with-plugin\f[R]) and the modification of +environment variables at restart (\f[C]dmtcp/plugin/modify-env\f[R]). +.PP +MANA does not support DMTCP\[cq]s distinct checkpoint filenames +(\f[C]--enable-unique-checkpoint-filenames\f[R]): \f[C]mana_restart\f[R] +takes one image from each \f[C]ckpt_rank_N\f[R] directory, and not +always the newest. +.SH ENVIRONMENT .TP \f[B]\f[R]\f[C]MANA_P2P_WAIT\f[R]\f[B]\f[R] How \f[C]MPI_Send\f[R] and \f[C]MPI_Recv\f[R] wait for their message. @@ -106,36 +274,26 @@ In \f[C]blocking\f[R] mode, MANA initializes the MPI library with MANA reads the variable at \f[C]MPI_Init\f[R] and keeps the mode after restart. .TP -\f[B]\f[R]\f[C]MANA_DEBUG\f[R]\f[B]\f[R] -MANA will print to stderr extra information to help developers debug -MANA. -.TP \f[B]\f[R]\f[C]MANA_PRELOAD\f[R]\f[B]\f[R] A colon-separated list of libraries that \f[C]mana_launch\f[R] loads ahead of MANA, so that they intercept MPI calls before MANA does (e.g., a PMPI profiler). A library in \f[C]LD_PRELOAD\f[R] loads after them. .TP -\f[B]\f[R]\f[C]DMTCP_MANA_PAUSE\f[R]\f[B] or \f[R]\f[C]DMTCP_LAUNCH_PAUSE\f[R]\f[B]\f[R] -DMTCP/MANA will pause during launch to allow \f[C]gdb attach\f[R] (GDB -must be on same node.) -.TP -\f[B]\f[R]\f[C]DMTCP_MANA_PAUSE\f[R]\f[B]\f[R] -DMTCP/MANA will pause very early in restart, inside -\f[C]mtcp/mtcp_restart.c\f[R] shortly before calling -\f[C]splitProcess()\f[R] (intended for developers) +\f[B]\f[R]\f[C]DMTCP_LOG_LEVEL\f[R]\f[B]\f[R] +How much DMTCP prints to stderr: \f[C]error\f[R], \f[C]warn\f[R], +\f[C]note\f[R] or \f[C]trace\f[R]. .TP \f[B]\f[R]\f[C]DMTCP_RESTART_PAUSE\f[R]\f[B]\f[R] -DMTCP/MANA will pause during restart, before resuming execution, to -allow \f[C]gdb attach\f[R] (GDB must be on same node.) -.PP -\f[B]\f[CB]INSPECTING MANA for DEBUGGING\f[B]\f[R] +DMTCP/MANA will pause during restart, at the stage given by the value (1 +to 7), to allow \f[C]gdb attach\f[R] (GDB must be on same node.) +.SH DEBUGGING .PP To see status of ranks (especially during checkpoint), try: .IP .nf \f[C] -bin/mana_status --list +bin/mana_status \f[R] .fi .PP @@ -143,16 +301,16 @@ To inspect a checkpoint image from, for example, Rank \f[C]0\f[R], try: .IP .nf \f[C] -dmtcp/util/readdmtcp.sh ckpt_rank0/ckpt_*.dmtcp +dmtcp/util/readdmtcp.sh ckpt_rank_0/ckpt_*.dmtcp \f[R] .fi .PP -To debug during restart at stage `N', replace `N' Below, and execute: +To debug during restart at stage `N', replace `N' below, and execute: .IP .nf \f[C] -srun ... env DMTCP\[rs]_RESTART\[rs]_PAUSE=N mana\[rs]_restart ... APPLICATION & -gdb -p APPLICATION PID # from a different terminal +srun ... env DMTCP_RESTART_PAUSE=N bin/mana_restart & +gdb -p PID # PID of the paused rank, from another terminal \f[R] .fi .PP @@ -162,7 +320,7 @@ following: .nf \f[C] (gdb) source dmtcp/util/gdb-dmtcp-utils.py -(gdb) load-symbol-library ADDRESS-OR-LIBRARY-SUBSTRING +(gdb) load-symbols-library ADDRESS-OR-LIBRARY-SUBSTRING (gdb) dmtcp # to see other available commands \f[R] .fi @@ -171,18 +329,37 @@ If you are debugging the lower half internals of MANA, you may need: .IP .nf \f[C] -(gdb) file bin/lh_proxy +(gdb) file bin/lower-half \f[R] .fi .PP If debugging the low-level restart process, please read -restart/plugin/README. +src/lower-half/README. +.SH FILES +.TP +\f[B]\f[R]\f[C]\[ti]/.mana.rc\f[R]\f[B]\f[R], or \f[B]\f[R]\f[C]\[ti]/.mana-slurm-JOBID.rc\f[R]\f[B]\f[R] in a Slurm job +The coordinator\[cq]s host and port, written by +\f[C]mana_coordinator\f[R] and read by the other commands. +\f[C]mana_launch\f[R] and \f[C]mana_restart\f[R] delete these files when +they are more than a week old. +.TP +\f[B]\f[R]\f[C]ckpt_rank_N/\f[R]\f[B]\f[R] +The checkpoint of rank N, in the checkpoint directory: its image +\f[C]ckpt_*.dmtcp\f[R] and the file \f[C]header.mana\f[R]. .SH BUGS .PP Report bugs in MANA to: https://github.com/mpickpt/mana .SH SEE ALSO .PP -\f[B]MANA home page:\f[R] , +\f[B]MANA website:\f[R] , +.PD 0 +.P +.PD +\f[B]MANA source code:\f[R] , +.PD 0 +.P +.PD +\f[B]DMTCP home page:\f[R] , .PD 0 .P .PD diff --git a/manpages/mana.1.md b/manpages/mana.1.md index b405946ce..f2d1f781c 100644 --- a/manpages/mana.1.md +++ b/manpages/mana.1.md @@ -1,5 +1,5 @@ --- -date: 31 March 2022 +date: 5 November 2026 section: 1 title: MANA description: MPI-Agnostic Network-Agnostic Checkpointing @@ -11,10 +11,10 @@ description: MPI-Agnostic Network-Agnostic Checkpointing # SYNOPSIS -**mana_launch** [\--help] [\--verbose] [\--timing] [DMTCP_OPTIONS] [\--ckptdir *DIR*] COMMAND [*ARGS...*]\ **mana_coordinator** [\--help] [\--verbose] [*DMTCP_OPTIONS*]\ -**mana_status** [\--help] [\--verbose] [*DMTCP_OPTIONS*]\ -**mana_restart** [\--help] [\--verbose] [*DMTCP_OPTIONS*] [\--restartdir *DIR*] +**mana_launch** [\--help] [\--verbose] [\--timing] [\--gdb] [\--ckptdir *DIR*] [*DMTCP_OPTIONS*] *PROGRAM* [*ARGS...*]\ +**mana_status** [\--help] [\--verbose] [*COMMAND*]\ +**mana_restart** [\--help] [\--verbose] [\--timing] [\--gdb] [\--restartdir *DIR*] [\--ckptdir *DIR*] [*DMTCP_OPTIONS*] # DESCRIPTION @@ -24,56 +24,197 @@ Checkpointing". It is designed to be compatible with most MPI implementations and most underlying networks. MANA is built on top of the DMTCP checkpointing package. -# COMMAND LINE OPTIONS +A MANA job uses four commands. `mana_coordinator` starts the coordinator +that the ranks connect to. `mana_launch` runs each rank of an MPI program +under MANA. `mana_status` lists the ranks and asks the coordinator for a +checkpoint. `mana_restart` restarts the ranks from their checkpoint. The MPI +program needs no change and no relinking. The four commands are wrappers of +DMTCP's commands, and they accept DMTCP's options too (see COMMANDS AND +OPTIONS). -**`--help`** -: Show additional DMTCP_OPTIONS for a MANA command. +MANA's website, \, has the rest of MANA's +documentation: a user guide, a developer guide and the release notes. -**`--verbose`** -: Display the underlying DMTCP command and DMTCP_OPTIONS used, and other info. +# CHECKPOINT AND RESTART -**`--timing`** -: Print times to stderr for INIT, EXIT, and ckkpt-restart events. - (stays active during both mana_launch and mana_restart) +For example, with the example program `examples/ring` (see BUILDING) on one +host, from MANA's directory: + +```bash +bin/mana_coordinator # 1. Start a coordinator. +mpirun -np 4 bin/mana_launch examples/ring # 2. Run the program under MANA. +bin/mana_status --bcheckpoint # 3. In another shell: checkpoint. +``` -# MANA PROGRAM EXECUTION +At step 3, each rank N writes its checkpoint into the directory +`ckpt_rank_N` in the current directory. The command returns when the +checkpoint is complete, and the program goes on. To stop the job and +restart it from the checkpoint: -Execute a MANA command with `--help` for a usage statement. +```bash +# 4. Stop the job, for example with Ctrl-C. +bin/mana_coordinator # 5. Start a new coordinator. +mpirun -np 4 bin/mana_restart # 6. Restart from ckpt_rank_*. +``` -Checkpoints may be invoked: +The restarted program goes on from where the checkpoint was taken. Restart +with the same number of ranks. A coordinator exits when its last rank +exits, so each run needs a new one. -**`periodically`** -: via the `--interval` or `-i` flag (units in seconds) +Other ways to checkpoint: -**`under program control`** -: (see `dmtcp/test/plugin/applic-initiated-ckpt` directory) +* `bin/mana_status --kcheckpoint`: checkpoint, then stop the job (steps 3 + and 4 in one command). +* `bin/mana_status --checkpoint`: checkpoint, and return at once, before + the checkpoint is done. +* `bin/mana_coordinator -i SECONDS`: checkpoint every *SECONDS* seconds. +* From the program, with DMTCP's API (see the directory + `dmtcp/test/plugin/applic-initiated-ckpt`). -**`externally`** -: via `mana_status --checkpoint` +A new checkpoint replaces the previous one. `mana_launch` stops if the +checkpoint directory already has `ckpt_rank_*` directories: move or delete +an old checkpoint before a new run. -A typical workflow for using MANA after untar\'ing is: +With Slurm, run the same commands in an allocation, with `srun`: ```bash -cd dmtcp-mana -./configure --enable-debug -make -j mana tests -# Build the program with mpicc as usual; no relinking for MANA is needed. -# 'make tests' also builds an example: examples/ring -salloc -N 2 -q interactive -C haswell -t 01:00:00 -bin/mana_coordinator -i10 +salloc -N 2 -t 01:00:00 # or your cluster's command for an allocation +bin/mana_coordinator -i 60 srun -N 2 bin/mana_launch examples/ring -bin/mana_coordinator -i10 +bin/mana_coordinator -i 60 srun -N 2 bin/mana_restart ``` -MANA supports most features of DMTCP, including: +# COMMANDS AND OPTIONS + +Each MANA command is a wrapper of a DMTCP command: `mana_coordinator` runs +`dmtcp_coordinator`, `mana_launch` runs `dmtcp_launch`, `mana_status` runs +`dmtcp_command`, and `mana_restart` runs the restart of `dmtcp_restart` in +MANA's lower half. A MANA command handles the options below, and passes the +others to its DMTCP command, so DMTCP's options work too. MANA sets some of +them itself: `--exit-on-last` and `--daemon` for `dmtcp_coordinator`, and +`--join-coordinator` and `--no-gzip` for `dmtcp_launch`. For DMTCP's +options, see each DMTCP command's `--help`, DMTCP's man pages (see SEE +ALSO), and DMTCP's documentation at \. + +## mana_coordinator + +Starts a DMTCP coordinator in the background, and writes its host and port +to a file where the other commands find it (see FILES). + +**`-i SECONDS`**, **`--interval SECONDS`** +: Checkpoint every *SECONDS* seconds (default: 0, no periodic checkpoints). + +**`--verbose`** +: Print the `dmtcp_coordinator` command that `mana_coordinator` runs. + +**`--help`** +: Print the options of `dmtcp_coordinator`. + +## mana_launch + +Runs *PROGRAM* with *ARGS* under MANA. Run `mana_launch` under the MPI +launcher, such as `srun` or `mpirun -np N`, in place of *PROGRAM*. A +coordinator must be running. + +**`--ckptdir DIR`** +: Write the checkpoints into *DIR*, which must exist (default: the current + directory). + +**`--timing`** +: Rank 0 prints to stderr the time since the start at each step of a + checkpoint, and at exit. + +**`--verbose`** +: Print the `dmtcp_launch` command that `mana_launch` runs, and DMTCP's notes + and warnings, which are otherwise not printed. + +**`--gdb`** +: Run the `dmtcp_launch` command under `gdb --args`, to debug MANA itself. + +**`--help`** +: Print a usage statement. + +`mana_launch` loads the plugins of DMTCP's `--with-plugin` after MANA's own. + +## mana_status + +Without *COMMAND*, lists the ranks that are connected to the coordinator. A +*COMMAND* asks the coordinator for an action: + +**`--checkpoint`**, **`-c`** +: Checkpoint the job. The job goes on. + +**`--bcheckpoint`**, **`-bc`** +: Checkpoint the job, and return when the checkpoint is done. + +**`--kcheckpoint`**, **`-kc`** +: Checkpoint the job, then stop it. + +**`-i SECONDS`** +: Checkpoint every *SECONDS* seconds from now on (0: no periodic + checkpoints). + +**`--verbose`** +: Print the `dmtcp_command` command that `mana_status` runs. + +**`--help`** +: Print the usage of `dmtcp_command`, which lists the other commands. + +## mana_restart + +Restarts a job from its checkpoint. Run `mana_restart` under the MPI +launcher, with the same number of ranks as the job, after a new +`mana_coordinator`. Each rank N restarts from `ckpt_rank_N`, and keeps the +`MANA_P2P_WAIT` mode of the job. + +**`--restartdir DIR`** +: Restart from the `ckpt_rank_*` directories in *DIR* (default: the current + directory). `mana_restart` stops if a checkpoint there is incomplete (a + `.temp` file). + +**`--ckptdir DIR`** +: Write later checkpoints into *DIR*, which must exist. + +**`--timing`** +: Rank 0 prints to stderr the time since the restart at the restart, at + each step of a checkpoint, and at exit. + +**`--verbose`** +: Print the command that `mana_restart` runs, and DMTCP's notes and + warnings, which are otherwise not printed. + +**`--gdb`** +: Run the restart under `gdb --args`, to debug MANA itself. + +**`--help`** +: Print a usage statement. + +# BUILDING + +```bash +git clone https://github.com/mpickpt/mana +cd mana +./configure +make -j +make tests # the tests, and the example examples/ring +``` + +`make` puts MANA's commands in `bin/`. To install MANA into another +directory, run `./configure --prefix=DIR`, `make` and `make install`. The +commands are then in `DIR/bin`. + +# NOTES -* plugins to extend the functionality of DMTCP -* virtualization of pathname prefixes (plugin/pathvirt) -* modification of environment variables at restart (plugin/modify-env) -* distinct checkpoint filenames (`dmtcp_launch --enable-unique-checkpoint-filenames`). The default is to overwrite the last checkpoint. +MANA supports most features of DMTCP, including plugins +(`mana_launch --with-plugin`) and the modification of environment variables +at restart (`dmtcp/plugin/modify-env`). -# ENVIRONMENT VARIABLES AND DEBUGGING +MANA does not support DMTCP's distinct checkpoint filenames +(`--enable-unique-checkpoint-filenames`): `mana_restart` takes one image from +each `ckpt_rank_N` directory, and not always the newest. + +# ENVIRONMENT **`MANA_P2P_WAIT`** @@ -88,58 +229,61 @@ MANA supports most features of DMTCP, including: can fail at a checkpoint (e.g., over UCX). MANA reads the variable at `MPI_Init` and keeps the mode after restart. -**`MANA_DEBUG`** - -: MANA will print to stderr extra information to help developers debug MANA. - **`MANA_PRELOAD`** : A colon-separated list of libraries that `mana_launch` loads ahead of MANA, so that they intercept MPI calls before MANA does (e.g., a PMPI profiler). A library in `LD_PRELOAD` loads after them. -**`DMTCP_MANA_PAUSE` or `DMTCP_LAUNCH_PAUSE`** - -: DMTCP/MANA will pause during launch to allow `gdb attach` (GDB must be on same node.) +**`DMTCP_LOG_LEVEL`** -**`DMTCP_MANA_PAUSE`** - -: DMTCP/MANA will pause very early in restart, inside - `mtcp/mtcp_restart.c` shortly before calling `splitProcess()` (intended - for developers) +: How much DMTCP prints to stderr: `error`, `warn`, `note` or `trace`. **`DMTCP_RESTART_PAUSE`** -: DMTCP/MANA will pause during restart, before resuming execution, to - allow `gdb attach` (GDB must be on same node.) +: DMTCP/MANA will pause during restart, at the stage given by the value (1 + to 7), to allow `gdb attach` (GDB must be on same node.) -**`INSPECTING MANA for DEBUGGING`** +# DEBUGGING To see status of ranks (especially during checkpoint), try: - bin/mana_status --list + bin/mana_status To inspect a checkpoint image from, for example, Rank `0`, try: - dmtcp/util/readdmtcp.sh ckpt_rank0/ckpt_*.dmtcp + dmtcp/util/readdmtcp.sh ckpt_rank_0/ckpt_*.dmtcp -To debug during restart at stage 'N', replace 'N' Below, and execute: +To debug during restart at stage 'N', replace 'N' below, and execute: - srun ... env DMTCP\_RESTART\_PAUSE=N mana\_restart ... APPLICATION & - gdb -p APPLICATION PID # from a different terminal + srun ... env DMTCP_RESTART_PAUSE=N bin/mana_restart & + gdb -p PID # PID of the paused rank, from another terminal To see the stack, you then may or may not need to try some of the following: (gdb) source dmtcp/util/gdb-dmtcp-utils.py - (gdb) load-symbol-library ADDRESS-OR-LIBRARY-SUBSTRING + (gdb) load-symbols-library ADDRESS-OR-LIBRARY-SUBSTRING (gdb) dmtcp # to see other available commands If you are debugging the lower half internals of MANA, you may need: - (gdb) file bin/lh_proxy + (gdb) file bin/lower-half + +If debugging the low-level restart process, please read src/lower-half/README. + +# FILES + +**`~/.mana.rc`**, or **`~/.mana-slurm-JOBID.rc`** in a Slurm job + +: The coordinator's host and port, written by `mana_coordinator` and read by + the other commands. `mana_launch` and `mana_restart` delete these files + when they are more than a week old. + +**`ckpt_rank_N/`** -If debugging the low-level restart process, please read restart/plugin/README. +: The checkpoint of rank N, in the checkpoint directory: its image + `ckpt_*.dmtcp` and the file `header.mana`. # BUGS @@ -147,6 +291,8 @@ Report bugs in MANA to: https://github.com/mpickpt/mana # SEE ALSO -**MANA home page:** \, \ +**MANA website:** \, \ +**MANA source code:** \, \ +**DMTCP home page:** \, \ **dmtcp**(1), **dmtcp_coordinator**(1), **dmtcp_launch**(1), \ **dmtcp_restart**(1), **dmtcp_command**(1) From 38763a1dc54bedf4f5fc44b1fc454a0eb55cc926 Mon Sep 17 00:00:00 2001 From: Yao Xu Date: Thu, 5 Nov 2026 19:59:52 -0500 Subject: [PATCH 2/2] Update the README for the current MANA The README's quick start ran mana_launch without an MPI launcher, and it pointed to the wiki and readthedocs, which describe MANA's old source layout. - Building and installing comes first: what MANA needs (the MPI libraries, a C++20 compiler, Python 3, and libc6-dbg on Debian and Ubuntu), how to build it (./configure also checks out the DMTCP submodule), and the notes on choosing the MPI compilers, installing and containers. It drops the Fedora example that set PATH on each command: a correct MPI installation has mpicc in PATH, and MPICC, MPICXX and MPIFC choose others. - A quick start then builds examples/ring, runs it under MANA with mpirun, checkpoints it with mana_status --bcheckpoint, and restarts it, as the man page does. --bcheckpoint returns when the checkpoint is complete, so the job can be stopped right after it. - A table says what each make target does (make, mana, tests, check, install, tidy, clean and distclean), and how to build only the tests, the example or the benchmarks. - Tests and benchmarks says how to run the tests, and what the two benchmarks measure. synthetic_app has less computation and more communication than the applications that inspired its profiles, so it is closer to a microbenchmark, and MANA's overhead on a real program can be different. - Documentation points to MANA's website, https://mpickpt.github.io/mana/, and to the man page, in place of the wiki and readthedocs. The README says that MANA's commands are wrappers of DMTCP's, and links DMTCP's documentation. - The name is "MPI-Agnostic, Network-Agnostic Checkpointing", as in the man page, and the typo "publicatoin" is fixed. Tested: the quick start as written, from the top of the tree: the restart goes on from the checkpoint's step, and the job ends. --- README.md | 142 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 105 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index a7d05ce0f..a63100369 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,58 @@ -## MANA (MPI-Agnostic, Network-Agnostic MPI) +## MANA (MPI-Agnostic, Network-Agnostic Checkpointing) MANA is an implementation of transparent checkpointing for MPI. It is -built as a plugin on top of [DMTCP](https://github.com/dmtcp/dmtcp). +built as a plugin on top of [DMTCP](https://github.com/dmtcp/dmtcp). An MPI +program runs under MANA without changes and without relinking. -For details of installing and using MANA, please see: -- [MANA Manual](https://github.com/mpickpt/mana/wiki) +### Documentation -Older documentation: -- [MANA documentation (https://mana-doc.readthedocs.io/en/latest/) ](https://mana-doc.readthedocs.io/en/latest/) -- [the MANA 'man' page](manpages/mana.1.md) (or `man ./mana.1` on a local copy) +- [MANA's website](https://mpickpt.github.io/mana/): a quick start, a user + guide, a developer guide, and the release notes. +- [The MANA man page](manpages/mana.1.md): how to checkpoint and restart, and + the options of each command. In a clone, read it with + `man -l manpages/mana.1`. -As seen in the MANA documentation, for install and a quick start, do: +MANA's commands (`mana_coordinator`, `mana_launch`, `mana_status` and +`mana_restart`) are wrappers of DMTCP's commands, and they accept DMTCP's +options too. See [DMTCP's documentation](https://dmtcp.github.io/). + +### Building and installing + +MANA needs: + +- an MPI library: MPICH or a library based on it (such as Cray MPICH), or + Open MPI 4.1 or 5.0; +- a C++20 compiler, and Python 3 for `mana_launch` and `mana_restart`; +- on Debian and Ubuntu, the package libc6-dbg: MANA needs the symbols of + the dynamic loader, which these distributions strip. + +To build MANA: git clone https://github.com/mpickpt/mana cd mana - ./configure + ./configure # also checks out DMTCP, MANA's dmtcp submodule make -j8 - # Testing on a single host (but see MANA docs for running on a cluster): - PATH_TO_MANA/bin/mana_coordinator - PATH_TO_MANA/bin/mana_launch [mana options] [user_program and args] -If `dmtcp/` is empty, `./configure` checks out the DMTCP commit that MANA -uses (the `dmtcp` submodule). A later `git pull` can change that commit -without changing `dmtcp/`; `make` then warns, and `git submodule update` -checks out the new commit. +`make` builds only MANA, and puts it in the source tree's `bin` and +`lib/dmtcp`. + +`./configure` uses the MPI compilers that come first in `PATH`: `mpicc`, +`mpic++` (or `mpicxx`) and `mpifort` (or `mpif90`), or `cc`, `CC` and `ftn` +with the Cray Programming Environment. To choose others, set `MPICC`, +`MPICXX` and `MPIFC`. For example: + + ./configure MPICC=$MPI_HOME/bin/mpicc MPICXX=$MPI_HOME/bin/mpicxx \ + MPIFC=$MPI_HOME/bin/mpifort + +Build the program with the same MPI library. To switch MPI libraries, run +`make clean` first. + +A later `git pull` can change the DMTCP commit that MANA uses without +changing `dmtcp/`; `make` then warns, and `git submodule update` checks out +the new commit. -`make` puts MANA in the source tree's `bin` and `lib/dmtcp`. To install it, -with DMTCP, into another directory, give that directory to `./configure`: +To install MANA, with DMTCP, into another directory, give that directory to +`./configure`: ./configure --prefix=$HOME/mana-install make -j8 @@ -35,23 +61,6 @@ with DMTCP, into another directory, give that directory to `./configure`: MANA's scripts find its other files from their own directory, so keep the default `--bindir` and `--libdir` (`PREFIX/bin` and `PREFIX/lib`). -On Debian and Ubuntu, also install the package libc6-dbg: MANA needs the -symbols of the dynamic loader, which these distributions strip. - -`./configure` uses the MPI library whose `mpicc` comes first in `PATH`: MPICH -or a library based on it (such as Cray MPICH), or Open MPI 4.1 or 5.0. Build -the program with the same MPI library. To switch MPI libraries, run `make -clean` first. For example, on Fedora, with Open MPI: - - PATH=/usr/lib64/openmpi/bin:$PATH ./configure - PATH=/usr/lib64/openmpi/bin:$PATH make -j8 - -With the Cray Programming Environment, as on Perlmutter, `./configure` uses -the compiler wrappers `cc`, `CC` and `ftn` instead. To choose the MPI -compilers yourself, set `MPICC`, `MPICXX` and `MPIFC`. For example: - - ./configure MPICC=cc MPICXX=CC MPIFC=ftn - In a container that runs as root, Open MPI's `mpirun` needs `OMPI_ALLOW_RUN_AS_ROOT=1` and `OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1`. @@ -60,12 +69,71 @@ default seccomp profile of Docker and Podman forbids: run the container with `--security-opt seccomp=unconfined` (or a profile that allows it), as MANA's CI does. +### Quick start + +After building MANA, build its example program, `examples/ring`. Then start +a coordinator, and run the program under MANA on one host: + + make -C examples # builds examples/ring + bin/mana_coordinator + mpirun -np 4 bin/mana_launch examples/ring # 60 steps, 1 s each + +While it runs, checkpoint it from another shell, in the same directory: + + bin/mana_status --bcheckpoint # each rank N writes ckpt_rank_N/ + +The command returns when the checkpoint is complete. Then stop the job +(Ctrl-C), and restart it from the checkpoint, with a new coordinator: + + bin/mana_coordinator + mpirun -np 4 bin/mana_restart # goes on from the checkpoint + +The restarted job goes on from the step of the checkpoint, not from step 1. +On a cluster, run the same commands in an allocation, with `srun` in place of +`mpirun`. The man page has the other ways to checkpoint, such as +`mana_status --kcheckpoint` and periodic checkpoints. + +### Make targets + +| Target | What it does | +|---|---| +| `make` | Builds DMTCP and MANA, and puts them in `bin/` and `lib/dmtcp/`. It first prints the build environment and, in a git clone, links MANA's git hook. | +| `make mana` | The same build, without printing the environment or linking the hook. | +| `make tests` | Builds the end-to-end tests (`test/`), the example (`examples/ring`) and the benchmarks (`bench/`). `make` does not build them. | +| `make check` | Builds MANA and the end-to-end tests, and runs the tests with `test/autotest.py`. | +| `make install` | Installs MANA and DMTCP into the `--prefix` of `./configure`. `make uninstall` removes them. | +| `make tidy` | Removes checkpoint files from the top directory: `ckpt_rank_*`, `ckpt_*.dmtcp` and `dmtcp_restart_script*`. | +| `make clean` | Does `make tidy`, and removes MANA's build outputs in `src/`. It keeps DMTCP's build and the tests. | +| `make distclean` | Does `make clean` and DMTCP's `make distclean`, and removes `bin/`, `lib/` and the files that `./configure` made. Run `./configure` again after it. | + +To build only one set of programs, run `make` in its directory: +`make -C test` (the end-to-end tests), `make -C examples` (`ring`), or +`make -C bench` (the benchmarks). + +### Tests and benchmarks + To run MANA's tests, which checkpoint and restart each test program: - make tests # builds them; 'make' does not + make tests # builds the tests, examples and benchmarks test/autotest.py # or: make check test/autotest.py --help # e.g. --launcher 'srun -n {n}' +The benchmarks in `bench/` measure MANA's overhead. They are built with +`-O2`, and `make check` does not run them. + +- `virtid_bench` measures the cost that MANA adds to each wrapped MPI call. + Run it natively and under MANA, and subtract. +- `synthetic_app` has profiles inspired by the communication of applications + such as VASP and GROMACS. A run under MANA must print the same checksum + as a native run. See + [bench/README-synthetic_app.md](bench/README-synthetic_app.md). + +`synthetic_app` has less computation and more communication than the +applications that inspired its profiles, so it is closer to a +microbenchmark: MANA's overhead on a real program can be different. + +### License + MANA is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License, version 3 or (at your option) any later version. This is the same license as DMTCP. See @@ -75,7 +143,7 @@ autotools scripts carry their own license notices. --- -To cite this project in a publicatoin, please cite: +To cite this project in a publication, please cite: * "MANA for MPI: MPI-Agnostic Network-Agnostic Transparent Checkpointing", Rohan Garg, Gregory Price, and Gene Cooperman, HPDC'19.