Skip to content

v5: Extend regression comparison helpers - #573

Open
mathomp4 wants to merge 1 commit into
developfrom
feature/v5-regression-test-helpers
Open

v5: Extend regression comparison helpers#573
mathomp4 wants to merge 1 commit into
developfrom
feature/v5-regression-test-helpers

Conversation

@mathomp4

@mathomp4 mathomp4 commented Aug 24, 2026

Copy link
Copy Markdown
Member

Purpose

This extends the nccmp-based regression comparison support introduced in the preceding PRs. The immediate driver is compiler-to-compiler roundoff: an absolute tolerance alone can reject a small relative difference in a large-magnitude field, while a relative tolerance alone can reject harmless near-zero differences. The new API allows a component to express both bounds explicitly.

Comparison API

compare_netcdf_files(baseline_file current_file [options...]) is a new reusable helper for a single NetCDF file pair. compare_results(baseline_dir current_dir [options...]) retains its directory-oriented API and delegates each discovered baseline file to this helper.

The following options are accepted by both helpers:

  • ABSOLUTE_TOLERANCE <value>: adds nccmp lowercase --tolerance <value>, its absolute tolerance option.
  • RELATIVE_TOLERANCE <value>: adds nccmp uppercase --Tolerance <value>, its relative-percent tolerance option. The case distinction is required by nccmp.
  • JOIN_TOLERANCES: adds --join-tolerance, causing nccmp to accept a value that meets either the absolute or relative threshold. This option requires both ABSOLUTE_TOLERANCE and RELATIVE_TOLERANCE; configuration fails clearly if either is missing.
  • TOLERANCE <value>: retained as the existing absolute-tolerance spelling, mapped to lowercase --tolerance.
  • A single positional tolerance: retained as the historical absolute-tolerance form. More than one unrecognized positional argument is now reported as an error.
  • NANS_ARE_EQUAL and its existing singular alias NAN_ARE_EQUAL: add --nans-are-equal.
  • EXCLUDE_VARS <var...>: retains existing behavior, passing a comma-separated --exclude= list to nccmp.

For compatibility, when ABSOLUTE_TOLERANCE is not specified the helper continues to resolve an absolute tolerance from, in order: TOLERANCE, a single positional argument, REGRESSION_TOLERANCE, and global TOLERANCE. Explicit ABSOLUTE_TOLERANCE takes precedence. The existing REGRESSION_NANS_ARE_EQUAL, NANS_ARE_EQUAL, REGRESSION_EXCLUDE_VARS, and EXCLUDE_VARS variable fallbacks are unchanged. If nccmp is unavailable, the existing cmp fallback remains in use.

Other Changes

  • compare_results() now discovers both .nc and .nc4 baseline files.
  • esma_add_regression_tests() accepts optional LABELS <label...>. These are appended to the macro's always-present REGRESSION label, allowing a component to retain an additional suite label without rebuilding its own CTest registration.

Example

compare_netcdf_files(
  "${baseline}" "${current}"
  ABSOLUTE_TOLERANCE 1e-4
  RELATIVE_TOLERANCE 1e-4
  JOIN_TOLERANCES
)

This accepts a comparison if it is within 1e-4 absolutely, which protects near-zero quantities, or within 1e-4 percent relatively, which permits compiler roundoff on large-magnitude fields.

Validation

  • ctest -R "^(ll-ll|cs-cs|cs-ll|ll-cs)$" --output-on-failure: 4/4 passed
  • ctest -R "^(adv-dyn|dyn-sa)$" --output-on-failure: 3/3 passed

@mathomp4
mathomp4 requested a review from a team as a code owner August 24, 2026 17:49
@github-actions

Copy link
Copy Markdown

This PR is being prevented from merging because you have not added one of our required labels: 0 diff, 0 diff trivial, Non 0-diff, 0 diff structural, 0-diff trivial, Not 0-diff, 0-diff, automatic, 0-diff uncoupled, github_actions. Please add one so that the PR can be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant