Repository navigation
kenref: support more than one rank per replica (#1443) - #1490
Open
aalhossary wants to merge 2 commits into
Open
aalhossary wants to merge 2 commits into
aalhossary wants to merge 2 commits into
Conversation
calculate() read the replica count and index from multi_sim_comm on every step and on every rank. A rank that is not the first rank of its replica only has the MPI_COMM_SELF stand-in, which reports "replica 0 of 1", so the module refused to run with more than one rank per replica. - The replica count and index come from getNumberOfReplicas() / getReplicaIndex() (plumed#1481), once, at setup. - Only the first rank of each replica gathers and scatters over multi_sim_comm. The derivatives are then broadcast over comm, because every rank applies forces to the atoms it owns. - The energy is printed once per job, not once per rank. New regtest rt-kenref-sigma-multi-dd: 2 replicas x 2 ranks with --debug-dd. Its references are byte-identical to the same run with one rank per replica. This needs the matching KEnRef change on Smith-Group/KEnRef release/2.x.
The kenref module now needs the KEnRef change that goes with it, which is on release/2.x but not yet in the v2.0.0 tag. Before merge, v2.0.0 moves to that commit and this pin goes back to v2.0.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #1443 for
KENREF, in the same way as #1486 forTTMETAD.KENREFrefused to run when a replica spans several ranks; this makes it work.calculate()read the replica count and index frommulti_sim_common every step and on every rank. A rank that is not the first of its replica only has theMPI_COMM_SELFstand-in, which reports "replica 0 of 1", so it would have applied a one-replica restraint to the atoms it owns without any error. That is why the module refused this case until now.getNumberOfReplicas()/getReplicaIndex()(Replica count and index on every rank of a replica (#1443, part C) #1481), once, at setup.multi_sim_comm. The derivatives are then broadcast overcomm. The broadcast is required: every rank applies forces to the atoms it owns, so a rank without derivatives would apply none.Merge order. This needs the matching KEnRef change, Smith-Group/KEnRef@b4aa301 on
release/2.x. The new code uses members that only that KEnRef declares, and a PLUMED and a KEnRef that do not match fail to compile; they never build and give wrong forces. This PR deliberately pins.ci/install.kenreftorelease/2.x, which contains that commit, so CI builds the KEnRef it needs. Between approval and merge (or in a follow-up), I will movev2.0.0to that commit and return the pin tov2.0.0.Target release
I would like my code to appear in release master (v2.11)
Type of contribution
Copyright
COPYRIGHTfile with the correct license information. Code should be released under an open source license. I also used the commandcd src && ./header.sh mymodulenamein order to make sure the headers of the module are correct.Tests
New
kenref/rt-kenref-sigma-multi-dd: 2 replicas × 2 ranks withdriver --debug-dd yes, using real two-member data. The replicas start from different frames, so their forces differ (by up to 4.56). TheCOLVARfiles and all four force dumps are byte-identical to the same run with one rank per replica. Removing the broadcast makes all four force files fail.rt-kenref-sigmais unchanged and passes.Also checked locally:
--enable-debug(Raise an error in debug builds when multi_sim_comm is used where it is a stand-in (#1443) #1484), the new test never uses the stand-in. Re-adding an ungated gather trips the Raise an error in debug builds when multi_sim_comm is used where it is a stand-in (#1443) #1484 error on exactly the two non-first ranks. The-debug-CI jobs do not build kenref, so this check cannot run on this module in CI.