Repository navigation
Re-pin the JANUS pipeline tests to SOCRATES 2607.3 - #98
Merged
Merged
Conversation
SOCRATES 2607.3 corrects Simpson's rule for an odd number of intervals. This raises about half of the Planck table values that prep_spec writes into the spectral file of these tests, by up to 1.1 %. The upward longwave and net fluxes of the instellation test and the outgoing longwave radiation of the runaway-greenhouse test move by 1.6e-5 to 1.3e-2 relative, outside the 1e-5 tolerance. The 5 active rows now hold the 2607.3 values to 8 significant digits, and the tolerance stays at 1e-5. The Validation page and its plots show the new values.
The stored values of the instellation and runaway-greenhouse tests belong to one SOCRATES version, while CI builds SOCRATES main. A constant in tests/helpers now holds that version, and a failed comparison names it next to the version in use. The runaway-greenhouse cases get ids for their regime instead of the stored value, so the ids stay the same when the values change. The Validation page says that the commented-out rows are not kept at that version.
The note on a failed stored-value comparison now says when the SOCRATES version of the run differs from the version of the stored values, and states a match plainly otherwise. It is a module constant in tests/helpers. Both pipeline tests take their parametrize ids from a table keyed by the input, so a reordered or added row cannot take the id of another case, and the instellation cases are named for their orbital distance. The Validation page points to the version constant instead of repeating it, and both sections say that the commented-out rows are not checked against that version.
Member
Wouldn't this be better solved by packaging SOCRATES and pinning in a regular package manager? See also #96 (review) |
Member
Author
|
HI @egpbos Yes agree, my PRs were simply patches for CI failing. In the long run your solution is clearly better. Even though SOCRATES is kind of "special", as we manually try to remain consistent with the main code. @nichollsh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The JANUS nightly on main has failed since SOCRATES 2607.3 (FormingWorlds/SOCRATES#36, the Simpson's rule fix for an odd number of intervals): https://github.com/FormingWorlds/JANUS/actions/runs/40663649252 (5 failed, 321 passed, SOCRATES e34954c5). The 5 stored-value cases of
test_runaway_greenhouseandtest_instellationcompare the full JANUS + SOCRATES pipeline at a relative 1e-5. The SOCRATES change moves their output past that tolerance. The same commit of main passed the day before with SOCRATES 2607.2 (9b729fd8): https://github.com/FormingWorlds/JANUS/actions/runs/40475336776.This PR re-pins the 5 cases to SOCRATES 2607.3 and names the SOCRATES version in a failed comparison.
tests/helpers/__init__.pyholdsSOCRATES_PINNED_VERSION. The assert message says whether the SOCRATES version string of the run matches it. A SOCRATES release that raises the version string and moves these values then fails with a version mismatch, not as an unexplained regression. A change that leaves the version string as it is still fails, but the message cannot tell it apart from a JANUS regression. The parametrize ids now name each case (sub_runaway_200K,plateau_1705K,post_runaway_2800K,inner_0p3au,outer_1p4au). The Validation page and its plots show the new stored values. No source lines change.CI builds SOCRATES from its main branch, so a later SOCRATES commit can move these values again; this PR does not pin the SOCRATES build.
Validation of changes
PYTHONPATH=src pytest -m "unit or smoke" testson the branch with an older local SOCRATES: 321 passed, 5 deselected (the 5 pipeline cases need SOCRATES 2607.3 and ran only in the nightly above).janus.set_socrates_envwhose version string is2607.2\n:version mismatch: stored values are from SOCRATES 2607.3; this run uses SOCRATES 2607.2; with2607.3\nit states the match.git grep -n "9.07314e01\|2.78990e02\|6.58379e03\|2.34297e03\|1.94521e03\|4.54446e01"for the old stored values: no hits.macOS 26, Python 3.12.
Checklist
Relevant people
None beyond the code owners.