Repository navigation
Replica count and index on every rank of a replica (#1443, part C) - #1481
Merged
carlocamilloni merged 4 commits intoOct 10, 2026
Merged
Conversation
multi_sim_comm is handed by the engine to a single rank of each replica (GREX setMPIIntercomm), so on the other ranks it keeps its MPI_COMM_SELF default and reports one replica of index 0. Actions that read their identity from it are therefore wrong on those ranks whenever a replica runs on more than one rank. Compute the replica count and index once, in a collective over the intra-replica communicator, and store them in PlumedMain. This is done in init(), before any action is created, and again at GREX init, so the result does not depend on which of the two the engine calls first. Expose them as Action::getNumberOfReplicas() and Action::getReplicaIndex(): correct on every rank, and needing no communication, so safe to call anywhere. No behaviour changes: nothing uses the new helpers yet.
DIMER selected its DSIGMA with multi_sim_comm.Get_rank(), which is 0 on every rank of a replica except the one holding the inter-replica communicator. With one DSIGMA per replica and more than one rank per replica, those ranks saw a single replica and aborted with "Mismatch between provided sigmas and number of replicas". Use getNumberOfReplicas() and getReplicaIndex() instead. The new regtest runs two replicas of two ranks each under a fake domain decomposition; every rank's forces now match the run with one rank per replica. Without this change the test aborts.
GATHER_REPLICAS sized itself with multi_sim_comm.Get_size(), which is 1 on every rank of a replica except the one holding the inter-replica communicator. Those ranks created a single rep-1 component, so any input referring to rep-2 failed on them; the rest of the job then deadlocked waiting for the rank that had died. Size from getNumberOfReplicas() instead. Only rank 0 of each replica takes part in the Allgather, so also broadcast the gathered values to the other ranks of the replica, which would otherwise hold a zero-filled buffer. The new regtest runs two replicas of two ranks each, under a fake domain decomposition, with different structures per replica, and matches the run with one rank per replica. GATHER_REPLICAS had no regtest before this.
Seventeen files worked out a replica count or index the same way: read multi_sim_comm on rank 0 of the replica, then broadcast or sum the value to the other ranks. Replace each with getNumberOfReplicas() or getReplicaIndex(), which give the same answer on every rank without any communication. Behaviour is unchanged, with two small exceptions. The per-step grid and hills writers in METAD and PBMETAD no longer do a broadcast each time they write. EFFECTIVE_ENERGY_DRIFT now raises its "needs multiple replicas" error on every rank of a replica rather than only on rank 0, so the job stops instead of waiting on the rank that raised it. Uses of multi_sim_comm that drive communication on the rank holding it are left as they are. OPESmetad.cpp and TTMetaD.cpp are not touched, to avoid conflicting with the open pull requests that modify them.
This was referenced Oct 10, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part C of #1443, as agreed there: a helper so that every rank of a replica knows the replica count and index, without API changes.
Core (
PlumedMain,GREX,Action)Action::getNumberOfReplicas()andAction::getReplicaIndex()return the same values on every rank of a replica.Maxover the intra-replica communicator, inPlumedMain::init()(before the input is read) and again inGREX init, so the order of the two calls does not matter. The helpers themselves do no communication, so they are safe to call anywhere, including incalculate().Two bugs fixed with the helper (both show up only with more than one MPI rank per replica)
DIMERreadmulti_sim_common every rank, so non-main ranks took replica 0's parameters. The run aborted with "Mismatch between provided sigmas and number of replicas".GATHER_REPLICASdid the same, and its values were gathered on rank 0 only and never broadcast to the other ranks. One rank failed on a missing component and the job deadlocked.Deduplication: seventeen files computed the count or index with the gate-and-broadcast idiom (read on rank 0, then
BcastorSum). They now call the helpers. Behaviour is unchanged, with two exceptions:EFFECTIVE_ENERGY_DRIFTraises its "needs multiple replicas" error on all ranks rather than only on rank 0.Not changed
multi_sim_commthat actually communicate on the rank that holds it.OPESmetad.cppandTTMetaD.cpp, to avoid conflicts with the open PRs that edit them.LOGMFD(see LOGMFD (LogPD) fails with more than one MPI rank per replica #1480).Target release
I would like my code to appear in release master (v2.11)
Type of contribution
Copyright
Tests
New regtests, each with 2 replicas × 2 MPI ranks and
driver --debug-dd yes:basic/rt-dimer-multi-dd: forces match the 1-rank-per-replica run exactly. Without the fix it aborts.basic/rt-gather-replicas-multi-dd: without the fix it deadlocks. This test cannot detect a missingBcaston its own, because the driver prints from rank 0 only.Locally the full suite gives 809 performed and 1 error,
basic/rt-make-fortran08. That test checks error propagation through the Fortran interface, and I believe the failure comes from mixing libc++ and libstdc++ in my build; it is unrelated to this change. CI will confirm.