v5: Extend regression comparison helpers - #573
Open
mathomp4 wants to merge 1 commit into
Open
Conversation
|
This PR is being prevented from merging because you have not added one of our required labels: 0 diff, 0 diff trivial, Non 0-diff, 0 diff structural, 0-diff trivial, Not 0-diff, 0-diff, automatic, 0-diff uncoupled, github_actions. Please add one so that the PR can be merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This extends the nccmp-based regression comparison support introduced in the preceding PRs. The immediate driver is compiler-to-compiler roundoff: an absolute tolerance alone can reject a small relative difference in a large-magnitude field, while a relative tolerance alone can reject harmless near-zero differences. The new API allows a component to express both bounds explicitly.
Comparison API
compare_netcdf_files(baseline_file current_file [options...])is a new reusable helper for a single NetCDF file pair.compare_results(baseline_dir current_dir [options...])retains its directory-oriented API and delegates each discovered baseline file to this helper.The following options are accepted by both helpers:
ABSOLUTE_TOLERANCE <value>: adds nccmp lowercase--tolerance <value>, its absolute tolerance option.RELATIVE_TOLERANCE <value>: adds nccmp uppercase--Tolerance <value>, its relative-percent tolerance option. The case distinction is required by nccmp.JOIN_TOLERANCES: adds--join-tolerance, causing nccmp to accept a value that meets either the absolute or relative threshold. This option requires bothABSOLUTE_TOLERANCEandRELATIVE_TOLERANCE; configuration fails clearly if either is missing.TOLERANCE <value>: retained as the existing absolute-tolerance spelling, mapped to lowercase--tolerance.NANS_ARE_EQUALand its existing singular aliasNAN_ARE_EQUAL: add--nans-are-equal.EXCLUDE_VARS <var...>: retains existing behavior, passing a comma-separated--exclude=list to nccmp.For compatibility, when
ABSOLUTE_TOLERANCEis not specified the helper continues to resolve an absolute tolerance from, in order:TOLERANCE, a single positional argument,REGRESSION_TOLERANCE, and globalTOLERANCE. ExplicitABSOLUTE_TOLERANCEtakes precedence. The existingREGRESSION_NANS_ARE_EQUAL,NANS_ARE_EQUAL,REGRESSION_EXCLUDE_VARS, andEXCLUDE_VARSvariable fallbacks are unchanged. If nccmp is unavailable, the existingcmpfallback remains in use.Other Changes
compare_results()now discovers both.ncand.nc4baseline files.esma_add_regression_tests()accepts optionalLABELS <label...>. These are appended to the macro's always-presentREGRESSIONlabel, allowing a component to retain an additional suite label without rebuilding its own CTest registration.Example
This accepts a comparison if it is within
1e-4absolutely, which protects near-zero quantities, or within1e-4percent relatively, which permits compiler roundoff on large-magnitude fields.Validation
ctest -R "^(ll-ll|cs-cs|cs-ll|ll-cs)$" --output-on-failure: 4/4 passedctest -R "^(adv-dyn|dyn-sa)$" --output-on-failure: 3/3 passed