Skip to content

Replace NZ digest pins with legacy and selected-factor differentials - #1427

Merged
MaxGhenis merged 3 commits into
mainfrom
nzhub/nz-digest-differentials
Oct 30, 2026
Merged

MaxGhenis merged 3 commits into
mainfrom
nzhub/nz-digest-differentials

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Status: Sol's check of 010beac approves with Needs Max: no. The head, 5f86383, merges main into that commit and changes none of this PR's lines; waiting on CI.

Follow-up to the #1398 merge review (nits N1 and N2). Two tests pinned SHA-256 digests of outputs that NZ resources feed, so any change to those inputs forced a re-pin that said nothing about what the tests guard. A third test searched raw text for 1.654. This PR replaces all three with differential or value-level checks. It changes tests only; no source or data file moves.

Changes

Retired test Replacement
test_calibration_columns.py::test_prepare_matches_pinned_legacy_output_digest (pin 1b22f4e1…) tests/fixtures/calibration_columns_legacy.py.txt freezes the six pure functions of calibration_columns.py at 4f333f2, before 2f220f7 moved the predicates into calibration_predicates. Three tests compare today's prepare_calibration_columns against it: on the NZ fixture (frame-equal and byte-equal), on generated NZ-domain tables (one run for valid tables and one per defect class), and on generated interval inputs.
test_reference_profiles.py::test_nz_factor_receipts_are_byte_identical_after_additive_factors (three pins) For each aging profile, the full-document receipt must byte-match the receipt that resolution yields from only the selected factor. It must also match when the purpose vocabulary drops money_normalization_input, as it stood before #1398. A Hypothesis property drops and reorders the unread factors.
The "1.654" in path.read_text() scan in test_the_rate_has_one_source_and_the_bridge_points_to_it A JSON-value walk returns every location that states the exact rate (827/500): as a JSON number, as a numerator/denominator object, or as a whole numeric token in text, including signs and exponents. A token can't touch a letter, digit or underscore. A sign after a digit separates two numbers, so both ends of 1.650-1.654 count; after a letter, a sign leaves no number, so v1.654 and v-1.654 hold none. It must find exactly money_normalization.json → rate.text. A parametrized test pins the matcher on 38 cases.

The legacy loader follows the reference_selection_legacy.py.txt pattern. It execs the fixture with an explicit scope (math, Mapping, Integral, Real, Any, np, pd, and interval_indicators and validate_interval_document from reference_intervals). reference_intervals.py hasn't changed since 4f333f2 (git diff 4f333f2c origin/main -- …/reference_intervals.py prints nothing). Each of the six frozen functions matches its 4f333f2 source segment byte for byte; the reviewer checked this independently.

_outcome deep-copies every argument for each call, including the intervals option. It returns the call's result and a snapshot of what the call left in its own copies, both in a typed encoding:

  • each scalar carries its type and repr, so pd.NA, None and np.int64(7) differ;
  • each column keeps its dtype's repr, so a categorical's categories, their order and its ordered flag count, as does a string column's storage;
  • both axes keep their class, dtype, names and labels, and each frame keeps its attrs and duplicate-label flag.

So neither implementation can change what the other reads, and a write to any of those parts of an input shows as a difference even when outputs agree. The encoding doesn't claim more: state that a repr hides, such as a MultiIndex level's dtype, stays out of view. The generated tables give tenure a categorical dtype half the time, in any category order, ordered or not.

Invariants

  1. Refactor equivalence. For any NZ-domain tables (including empty tables, boxed NumPy identities and relabelled indexes) and the NZ bindings, today's prepare_calibration_columns and the frozen 4f333f2 version give the same outcome: the same typed snapshot of the prepared tables (values, dtypes, both axes, attrs) or the same ValueError message, and the same final state of their inputs. The same holds for interval outputs on either entity. The equivalence covers valid bindings; malformed binding schemas already refuse differently (ValueError against AttributeError) and stay outside it.
  2. Factor additivity. For a well-formed factor document (unique, nonempty factor ids), an NZ reference-profile receipt depends only on the factors its selection reads. Adding, removing or reordering any other factor, or adding a purpose those factors don't use, leaves the receipt byte-identical.
  3. Single rate source. The exact USD→NZD rate appears once across nz/**/*.json, at money_normalization.json → rate.text, as a JSON number, a numerator/denominator object, or a whole numeric token in text.

Evidence

  • The frozen legacy code, run on today's NZ bindings, reproduces the retired pin exactly: 1b22f4e11e6e6c901866010d709acadaaad638e0774e80751e0ff23257a2bde7. Today's three receipts also reproduce their retired hashes (b0f8648d…, 8989fa75…, e246f615…).
  • Local runs on CI's import path (every packages/*/src plus the repo root), one file at a time with -n 0:
    • test_calibration_columns.py: 104 passed (Hypothesis seeds 1427 and 1428);
    • test_reference_profiles.py: 129 passed;
    • test_nz_money_normalization.py: 55 passed.
  • Lint and plan. ruff check . passes, as do ruff format --check on the four changed files (including the fixture, which ruff reads as Python when named) and tools/ci_test_plan.py verify. A directory-wide ruff format --check packages/microcosm-build/tests fails on seven files this PR doesn't touch; each is byte-identical to its merge-base version.
  • Mutants. Each mutant ran against its target tests; the script then restored the files. The round-4 run covers the first 21 below, and round 5 reran the 11 bridge mutants, four of them new; both put pytest's basetemp outside the repository.
Mutant Expected Result
live ge → gt in calibration_predicates fail 2 failed (NZ fixture; valid generated tables)
live accepts negative ages fail 1 failed (the negative_age class)
frozen legacy lt → le fail 2 failed
live rewrites the first cut in its intervals argument (Sol's case) fail 1 failed
live writes a column into its input tables after copying them fail 4 failed (valid tables, and the three roster refusals that run after the copy)
receipt records the size of the whole factor document fail 3 failed
resolver validates every factor, not only the selected ones fail 4 failed
receipt lists the purpose vocabulary (module-level name import) fail 3 failed
receipt lists the purpose vocabulary (attribute access) fail 3 failed
currency_bridge.json restates "Rate 1.654." fail 1 failed
currency_bridge.json restates "Rate 1654e-3." fail 1 failed
currency_bridge.json adds {"numerator": 827, "denominator": 500} fail 1 failed
live swaps pd.NA for None in its inputs before returning (round-2 case) fail 1 failed
currency_bridge.json mentions 1.6541 and 1.654e1 pass 1 passed
currency_bridge.json mentions "Dependency version v1.654." (round-2 case) pass 1 passed
live reverses the category order of its inputs' categorical columns before returning (round-3 case) fail 1 failed (valid generated tables, the only run that reaches the return)
live flips its inputs' categorical ordered flag before returning fail 1 failed
live names its input frames' column axis before returning fail 1 failed
live sets an attrs entry on its input frames before returning fail 1 failed
currency_bridge.json mentions "Dependency version v-1.654.", a-1.654 and v-1654e-3 (round-3 cases) pass 1 passed
currency_bridge.json restates "Rate (+1.654)." fail 1 failed
currency_bridge.json gives the range "1.650-1.654" (round-4 case) fail 1 failed
currency_bridge.json gives the range "1.700-1.654" (round-4 case) fail 1 failed
currency_bridge.json gives the sum "0+1.654" (round-4 case) fail 1 failed
currency_bridge.json mentions the dates 2026-04-01 to 2026-11-03 pass 1 passed

In the first full run, the negative-age mutant survived: 60 examples spread over 11 defect classes didn't always draw it. Each defect class now gets its own 15-example run, and each run asserts that the live code refuses (or, for valid tables, prepares) before comparing it with the oracle.

git grep finds no remaining reference to the retired test names or the four retired digests.

Review

An independent GPT-6.1 Sol review at 08a2cd8 requested changes:

  1. The interval differential passed intervals to both calls uncopied, so a live call that rewrote it also changed the oracle's input.
  2. The rate matcher missed exponent forms such as 1654e-3, and matched 1.654e1 and -1.654 inside longer numbers.
  3. The receipt tests didn't guard against the purpose vocabulary leaking into receipts.
  4. Some prose overstated the old pins' reach or used passives.

Round 2 (09dc3fb) fixed all four; the mutant table includes Sol's interval case and both purpose-leak spellings.

Sol's round 2 at 09dc3fb found the four fixed and approved with three nits; round 3 fixes all three:

  1. to_dict turned object-dtype pd.NA into None, so the input snapshot missed an NA-to-None write. The snapshot now encodes each value with its type, and two @examples put a missing amount on an excluded row.
  2. Tokens could start or end against letters, so v1.654, a1654e-3b and 1.654e+ matched. Tokens now need a non-word boundary on both sides, and the matcher test covers those forms.
  3. ruff format --check wanted a second blank line after the fixture header. The six frozen functions still match 4f333f2 byte for byte.

Sol's round 3 at 2f82930 found those three fixed and approved with three more nits; round 4 fixes them:

  1. The new token boundary blocked a match at a sign that follows a letter, but matching then restarted at the next digit, so v-1.654, a-1.654 and v-1654e-3 matched as positive numbers. A token now can't follow a sign either.
  2. The body called the snapshot lossless, but str(dtype) dropped a categorical's categories and ordered flag and a string column's storage, and the column axis lost its name. The snapshot now keeps each dtype's repr, both axes' names and each frame's attrs and flag, and the body states what it covers.
  3. The CI line held a placeholder.

Sol's round 4 at 3d8610a found those three fixed, answered Needs Max: no, and approved with one nit, which round 5 fixes. The sign rule also hid the second end of a range such as 1.650-1.654 or 1.700-1.654, and the 1.654 in the sum 0+1.654. A sign that follows a digit now separates two numbers, and the matcher test covers both ranges, the sum, a date range and +-1.654.

Sol's round 5 at e0517e1 found that fix sound, answered Needs Max: no, and approved with one wording nit in the regex comment. 583d9e2 rewords that comment and credits _text_numbers, not the regex, with skipping 2.1.654; comments alone change, and the file's AST and non-comment tokens match e0517e1's. Sol's check of 583d9e2 agreed and flagged the same "X included" wording in two docstrings. The head, 010beac, rewords both; with docstrings blanked, both files' ASTs match 583d9e2's.

Out of scope

N3 from the same review stays open: CI never sets NZ_CHRONICLE_ARTIFACT, so the generator-reproduction tests skip there.

CI

Run 41087535556 at 5f86383 is pending. Earlier runs failed one engine-free (3.13) test: test_spec_source_citations.py::test_a_long_string_is_scanned_in_linear_time, a 2 s CPU limit on the citation scanners. This PR doesn't touch that test or those scanners, and #1421 dropped the test from main, so this head no longer runs it.

🤖 Generated with Claude Code

The #1398 review found that two tests pinned digests of outputs that NZ
resources feed, so any change to those inputs forced a re-pin unrelated to
what the tests guard. They now compare against oracles instead.

- Calibration columns: a frozen copy of 4f333f2's
  prepare_calibration_columns serves as the oracle. Tests compare the live
  code with it on the NZ fixture, on generated NZ-domain tables (valid and
  defective, empty, boxed NumPy identities and relabelled indexes) and on
  generated interval inputs. Each call gets its own copy of every argument,
  and the comparison includes what each call left in its copies. On
  today's bindings the frozen code reproduces the retired pin, 1b22f4e1.
- NZ factor receipts: a receipt must match the receipt resolved from only
  its selected factor, for each aging profile, and must not change when the
  factor purpose vocabulary drops money_normalization_input. A Hypothesis
  property drops and reorders the unread factors.
- The 1.654 scan reads JSON values. It finds the exact rate as a JSON
  number, a numerator/denominator object, or a whole numeric token in text
  (signs and exponents included), and ignores other numbers such as 1.6541
  or 1.654e1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the nzhub/nz-digest-differentials branch from 08a2cd8 to 09dc3fb Compare October 30, 2026 03:00
MaxGhenis added a commit that referenced this pull request Oct 30, 2026
The round-2 review of #1427 approved with three nits:

- The input snapshot used to_dict, which turns object-dtype pd.NA into
  None, so a write that swapped them went unseen. _lossless now encodes
  each scalar with its type and repr, and the index with its class, dtype,
  name and labels. Two examples put a missing amount on an excluded row.
- A rate token could touch a letter, so v1.654, a1654e-3b and 1.654e+
  matched. Tokens now need a non-word boundary on both sides; the matcher
  test covers those forms.
- ruff format wanted a second blank line after the frozen fixture's
  header. The six frozen functions still match 4f333f2 byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 30, 2026
The round-2 and round-3 reviews of #1427 approved with nits:

- The input snapshot used to_dict, which turns object-dtype pd.NA into
  None, and then str(dtype), which drops a categorical's categories and
  order flag and a string column's storage. _typed now encodes each scalar
  with its type and repr, each dtype by its repr, both axes by class,
  dtype, names and labels, and each frame's attrs and duplicate-label
  flag. Generated tables now give tenure a categorical dtype half the
  time, in any category order, ordered or not. Two examples put a missing
  amount on an excluded row.
- A rate token could touch a letter, or start right after a sign that
  did, so v1.654, v-1.654, a1654e-3b and 1.654e+ matched. A token now
  needs a non-word boundary on both sides and can't follow a sign; the
  matcher test covers those forms.
- ruff format wanted a second blank line after the frozen fixture's
  header. The six frozen functions still match 4f333f2 byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the nzhub/nz-digest-differentials branch from 2f82930 to 3d8610a Compare October 30, 2026 03:58
MaxGhenis added a commit that referenced this pull request Oct 30, 2026
The round-2 to round-4 reviews of #1427 approved with nits:

- The input snapshot used to_dict, which turns object-dtype pd.NA into
  None, and then str(dtype), which drops a categorical's categories and
  order flag and a string column's storage. _typed now encodes each scalar
  with its type and repr, each dtype by its repr, both axes by class,
  dtype, names and labels, and each frame's attrs and duplicate-label
  flag. Generated tables now give tenure a categorical dtype half the
  time, in any category order, ordered or not. Two examples put a missing
  amount on an excluded row.
- A rate token could touch a letter, or start right after a sign that
  did, so v1.654, v-1.654, a1654e-3b and 1.654e+ matched. A token now
  needs a non-word boundary on both sides. A sign after a digit
  separates two numbers, so both ends of the range 1.650-1.654 count;
  after a letter, a sign leaves no number, so v-1.654 holds none. The
  matcher test covers those forms.
- ruff format wanted a second blank line after the frozen fixture's
  header. The six frozen functions still match 4f333f2 byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the nzhub/nz-digest-differentials branch from 3d8610a to e0517e1 Compare October 30, 2026 04:13
MaxGhenis added a commit that referenced this pull request Oct 30, 2026
The round-2 to round-4 reviews of #1427 approved with nits:

- The input snapshot used to_dict, which turns object-dtype pd.NA into
  None, and then str(dtype), which drops a categorical's categories and
  order flag and a string column's storage. _typed now encodes each scalar
  with its type and repr, each dtype by its repr, both axes by class,
  dtype, names and labels, and each frame's attrs and duplicate-label
  flag. Generated tables now give tenure a categorical dtype half the
  time, in any category order, ordered or not. Two examples put a missing
  amount on an excluded row.
- A rate token could touch a letter, or start right after a sign that
  did, so v1.654, v-1.654, a1654e-3b and 1.654e+ matched. A token now
  needs a non-word boundary on both sides. A sign after a digit
  separates two numbers, so both ends of the range 1.650-1.654 count;
  after a letter, a sign leaves no number, so v-1.654 holds none. The
  matcher test covers those forms.
- ruff format wanted a second blank line after the frozen fixture's
  header. The six frozen functions still match 4f333f2 byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the nzhub/nz-digest-differentials branch from e0517e1 to 583d9e2 Compare October 30, 2026 04:26
The round-2 to round-4 reviews of #1427 approved with nits:

- The input snapshot used to_dict, which turns object-dtype pd.NA into
  None, and then str(dtype), which drops a categorical's categories and
  order flag and a string column's storage. _typed now encodes each scalar
  with its type and repr, each dtype by its repr, both axes by class,
  dtype, names and labels, and each frame's attrs and duplicate-label
  flag. Generated tables now give tenure a categorical dtype half the
  time, in any category order, ordered or not. Two examples put a missing
  amount on an excluded row.
- A rate token could touch a letter, or start right after a sign that
  did, so v1.654, v-1.654, a1654e-3b and 1.654e+ matched. A token now
  needs a non-word boundary on both sides. A sign after a digit
  separates two numbers, so both ends of the range 1.650-1.654 count;
  after a letter, a sign leaves no number, so v-1.654 holds none. The
  matcher test covers those forms.
- ruff format wanted a second blank line after the frozen fixture's
  header. The six frozen functions still match 4f333f2 byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the nzhub/nz-digest-differentials branch from 583d9e2 to 010beac Compare October 30, 2026 04:42
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 30, 2026 04:47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 98aef75 into main Oct 30, 2026
10 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the NZ hub (session local_e407a527) at 5f86383 with --merge. Gates: gh pr checks exit 0; MERGEABLE; not draft; no CHANGES_REQUESTED review. Reviewed head: 010beac; the merged head differs only by a base update, and the PR's own changed lines are identical to the reviewed head's. Sol r7 approved 010beac with Needs Max: no (review out-sol-r7.md); the head only merges main; #1421 dropped the timing test that failed earlier runs.

@MaxGhenis
MaxGhenis deleted the nzhub/nz-digest-differentials branch October 30, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant