Keep the record of set_input values in step with storage

[MaxGhenis]

Fixes #559.

Problem and behavior

Deleting an input could leave _user_input_keys pointing at a later calculation for the same variable, branch, and period. apply_reform then preserved that calculation as an input, and to_input_dataframe exported it. Copies also shared the record despite having separate storage, and calculating set_input handlers could register their intermediate calculations.

The mixed-storage witness makes the provenance problem explicit: January rent is calculated to disk as 0, then supplied in memory as 500. Deleting the year removes the memory input while disk storage retains the calculation. The disk 0 must not become a replayable input. A genuine disk input, however, must survive deletion of a newer memory input.

This PR changes outputs when a reform follows input deletion and the stale record would previously have preserved a calculation or default. Country constructor flows that move inputs such as employment_income make that behavior relevant to downstream validation.

Change

• Keep the existing (variable, branch, Period) record, and track the memory/disk tiers that received an actual input in Holder._user_input_storage. _record_input_storage updates provenance when a tier is written; _stores_user_input checks surviving input locations. A cached value in another tier cannot stand in for a deleted input.
• Holder.delete_arrays compares storage keys before and after deletion, then removes a recorded slot when no input tier survives. Simulation.delete_arrays applies this to the visible branches. Deletion does not load disk arrays to make that decision.
• _seed_input_storage handles inputs registered without tier metadata, including Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576's restore path. Existing storage input marks can seed an already-recorded slot; they do not register an ordinary cache value for export or reform replay.
• Reform replay consults the recorded tiers before choosing a value. An unrecorded memory cache cannot mask a genuine disk input. If a cache overwrite destroyed the only supplied input, replay drops the stale record instead of promoting that cache to an input. The registry is rebuilt from exactly the preserved snapshots. Non-derived external cache writes seed an existing register-only input's metadata before overwriting its tier, including restored inputs that did not yet have holder metadata.
• Normalize periods from their fields so the record names the actual storage alias: eternity, calendar and rolling twelve-month years, and valid early-year periods. This avoids reparsing an early-year storage string after the value has already been written.
• Give clones and branches independent input records, tier metadata, and running set_input contexts. Rebuild the input record during subsampling.
• Exclude put_in_cache writes from input registration and input-context branch redirection. Custom handlers store replayable inputs through holder._set or holder.set_input. The Holder.set_input docstring and changelog describe this handler contract and behavior change.

The storage-level derived mark remains distinct from replay provenance. _set preserves the existing positional derived argument and adds is_input after it. The record's tuple shape remains compatible with downstream readers such as PE-US _rebind_holders.

Invariants and regression coverage

For the tested set_input, calculation, deletion, clone/branch, and cache-invalidation operations:

1. The record names surviving supplied inputs, with provenance tracked independently in each storage tier. A retained disk calculation does not replace a deleted memory input; a retained genuine disk input does survive.
2. The record, stored values, and dataframe exports agree with a reference model seeded from supplied inputs. Formula results calculated inside input handlers are excluded.
3. Copies have independent records and tier metadata. Updating one simulation does not change another's input record.
4. Period normalization is idempotent and matches storage keys, including eternity and twelve-month aliases, without rejecting valid early years.
5. Reform replay retains recorded inputs and purges unrecorded calculations and broader carry-over marks.

These claims apply to the operations and storage behavior exercised by the tests; they are not a claim about every possible sequence. In particular, they do not guarantee preservation through pre-existing input-overwriting behavior such as calculate_add on a two-month input. This PR does not repair that separate storage behavior.

The new external-cache differential checks post-reform records, exports, supplied values, and repeated reform against a fresh simulation, including optional cloning. Its focused invariants are:

• A register-only restored slot seeds only its existing non-derived tiers before a holder-mediated overwrite; an unregistered cache cannot create an input record.
• After invalidation, the registry equals the preserved input snapshots. A slot whose only input was overwritten leaves no ghost key, export, or replay value.
• Repeating a no-op reform on the simulation or its clone leaves the same input record, exports, and calculated values as a fresh simulation containing the surviving inputs.

These assertions do not prove that an export made before reform ignores an arbitrary external cache overwrite. Direct storage mutations that bypass holder provenance remain outside that contract.

The existing #576 persistence contract is distinct: dump/restore deliberately registers every non-derived dumped value as a new leaf input, including an external cache value that live reform replay would drop. The restore differential preserves round-trip values, marks, and pre-reform answers, then checks live replay against an independent supplied-write reference and restored replay against the dump's non-derived value snapshot. These post-reform checks apply to every generated trace, including an explicit same-value input/cache overwrite; cross-simulation equality applies when those independent expected input snapshots agree. The #576 dump/restore input-registration contract is unchanged.

Existing example, property, and differential coverage is retained:

• test_user_input_keys_match_reference_model exercises families of simulations with memory storage, including handler calculations, exports, clones, and branches: 300 sequences of up to 30 operations.
• test_user_input_keys_follow_memory_and_disk_storage tracks supplied values independently in each tier and checks records, reads, and actual exports: 200 sequences of up to 25 operations, with an explicit mixed-storage counterexample.
• test_period_normalization_matches_storage_strings checks normalization against storage strings and the existing parser where applicable, with explicit calendar and rolling twelve-month examples.
• test_tier_provenance_replays_the_same_inputs_as_a_fresh_simulation compares mixed-storage deletion and reform replay against a fresh simulation containing only the surviving supplied inputs.
• Focused examples retain the prior soundness witnesses, early-year matrix, genuine-disk-input control, and register-only restore compatibility.

Fix-round regression modules:

• tests/core/test_input_tier_invalidation.py: genuine disk inputs versus unrecorded memory caches, overwritten-only record removal, register-only restored inputs, optional cloning, and the fresh-simulation differential across repeated reform (60 generated examples plus the explicit disk-input witness).
• tests/core/test_registered_input_deletion.py: register-only deletion for early years, plus a differential with set_input across period units, sizes, aliases, storage tiers, and underscored branches. Deleting the slot removes the record; a later cache write is purged by reform. At the reviewed source head, the newly added regressions produced 38 failed, 16 passed before the repair.
• tests/core/test_disk_period_filenames.py: compound and early-year disk periods under emulated Windows filename restrictions, replacement/clone paths, legacy restore and current-file precedence. Its memory/disk differential compares logical branch-period keys, values, and live derived marks after writes; source values remain independent of clone writes, and restore reads the original file values. Restore retains its existing behavior of clearing derived marks. At the reviewed source head, these newly added regressions produced 22 failed, 3 passed before the repair; the legacy/separator-free restore controls passed.

All three modules were exercised against the reviewed implementation at 7559e6952657d9c1af276c8268017ce09561d28b before their repairs and rerun afterward:

Regression module	Before repair	After repair
Input-tier invalidation	5 failed	5 passed
Registered-input deletion	38 failed, 16 passed	54 passed
Portable disk filenames	22 failed, 3 passed	25 passed

The filename controls and modern-year deletion controls passed before the fixes. These three after-repair results were obtained at a4f384d77ed0607b6e74e5d269dcbf09c979c50d; tier invalidation also passed all five tests after base reconciliation. The new property/differential invariants are stated above; existing coverage was retained.

Validation

Final head: 5a69ca8d835484fc582a6c638c231da7c9af2001.

The final head includes canonical master at 648f4e467d3c20afd206ce5f9be7f89752809257 through a merge commit, preserving history and the required #576-before-#561 order. The holder conflict was reconciled by retaining both #576's unconditional fast-cache eviction and #561's supplied-input provenance updates. Incoming restore, copy/pickle, and numeric-validation behavior was retained. The intermediate repair head conflicted with the updated base, so GitHub did not start its PR workflow; the final head is mergeable and started a fresh workflow.

Each file was run directly, in the foreground, one at a time with uv run --no-sync --python 3.13 pytest -q --tb=short; the three initial regression modules additionally used --noconftest. The replay-index check selected the single named test below. PYTHONPATH was set to the checkout root and PYTHONNOUSERSITE=1; uv selected a fresh isolated environment and a writable local cache. No lock wrapper, folder run, full suite, or microsimulation was used.

Environment: macOS arm64, CPython 3.13.9, NumPy 2.4.2, pytest 9.1.1, Hypothesis 6.168.3, Ruff 0.15.5, Plotly 5.24.1. Dependencies were installed into the isolated environment, not borrowed through another environment's .pth file. Import-origin and search-path checks confirmed core loads from the checkout and test/runtime packages from that environment. Editable package metadata was refreshed to core 3.32.25 after the base update; uv pip check verified all 65 installed packages are compatible.

In addition to the three regression modules above, these focused checks passed on the combined code tree (production code at f38247002d2c81939be9d848ff61447a71123eed, with the subsequent test-only follow-up now committed in the final head):

File or selected test	Result
tests/core/test_user_input_keys.py	29 passed
tests/core/test_user_input_keys_property.py	3 passed
tests/core/test_user_input_keys_soundness.py	67 passed
tests/core/test_apply_reform_preserves_user_inputs.py	9 passed
tests/core/test_on_disk_storage_clones.py	27 passed
tests/core/test_storage_input_index.py::test_apply_reform_keeps_only_the_replayed_inputs	1 passed
tests/core/test_holder_write_fast_cache.py	16 passed
tests/core/test_input_tier_invalidation.py (rerun after reconciliation)	5 passed
tests/core/test_restore_input_registry_property.py	1 passed
tests/core/test_restore_input_provenance.py	2 passed
tests/core/test_simulation_copy_pickle_property.py	2 passed

The retained restore property first failed after base reconciliation on the explicit-input then identical-cache overwrite trace. Its old key/mark-only comparison assumed equal replay provenance. The corrected independent model passes with the same 200-example budget and operation strategies, retains round-trip and fresh-input comparisons, and adds unconditional live and restored replay checks plus a fresh serialized-leaf reference for every post-reform request. The two deterministic controls pin both same-value and changed-value overwrites across repeated reform. This fixes the test oracle while preserving #576's documented persistence behavior.

Focused Ruff lint and format checks passed for all eight edited Python files; git diff --check passed. No mutation campaign or performance benchmark was rerun.

At the reviewed head 7559e6952657d9c1af276c8268017ce09561d28b, CI run 37847864480 has 15 successful checks and four failed Windows Test jobs, for Python 3.11–3.14. All four Windows jobs fail exactly the same eight early-year multi-day cases: disk storage, the simulation/holder entry points, and years 1, 99, 999, and 1000. The traceback reaches numpy.save with a basename such as default_day:1-03-17:2.npy and raises OSError: [Errno 22] Invalid argument. Colon is reserved in Windows filenames. Microsoft file naming conventions.

The storage repair now encodes colons as semicolons only in the physical filename's period suffix, including replacement and clone-write filenames. Logical storage keys remain unchanged. Restore reads legacy colon filenames, decodes the new representation without changing branch semicolons or underscores, and prefers the current portable file when a legacy file remains beside an updated value. Valid serialized periods contain no semicolons, so this encoding is reversible without lengthening filenames. Existing live readers still indexed to a legacy file see its previous value until they restore again. Filename compatibility is one-way: current Core restores legacy files, but older Core readers do not decode the new compound filename encoding.

The portable-filename regression module passes all 25 cases locally. The existing early-year cases are retained; native Windows verification is reported separately below.

Final-head CI at 5a69ca8d83 passed: all 19 checks succeeded. These include lint, the changelog check, the test matrix on ubuntu and native Windows for Python 3.11–3.14, and "Test Core and country packages" on both platforms. The native Windows jobs that failed at 7559e695 with OSError: [Errno 22] on colon filenames now pass.

A separate description evidence check failed seven targeted assertions against the posted body at the reviewed head: tier tracking, qualified sequence claims, finished review status, the explicit Impact section, the fresh-impact HOLD, the Windows failures, and removal of the stale performance table. These are documentation-only checks of the description, not production regression tests or evidence that population-impact gates have passed.

The completed description passes all nine targeted evidence assertions, including the Impact/HOLD, current tier design, qualified invariants, CI disclosure, removal of stale performance claims, and absence of local paths or unfinished placeholders.

No new performance benchmark or mutation campaign was run in this fix round. The former performance table and mutation totals were measured on earlier heads and do not measure the current tier-tracking implementation. Writes after a holder has stored an input do period normalization and provenance bookkeeping. Seeding register-only inputs before overwrite also adds normalization on the external non-derived cache-write path when the simulation has registered inputs; the added condition does not change ordinary core writes with derived=True. This description makes no measured cost claim for those writes. The focused no-disk-read deletion regressions remain, but they do not establish performance for all recording paths.

Full suites, documentation builds, country-wide tests, and microsimulations were not run locally in this fix round; CI and the hub own those gates. Documentation changes cover the handler contract in Holder.set_input and the existing input changelog fragment, and the portable filename/legacy restore contract in the storage class and restore docstrings plus a storage changelog fragment. The API reference renders those source docstrings; no generated documentation is hand-edited. Documentation build validation remains a CI gate.

Documentation review: impact medium, confidence medium. The durable source documentation describes the changed handler and persistence contracts; no new public import or developer workflow needs separate documentation. Focused Ruff and description checks validate the edited text and code formatting. Documentation builds were not run locally. The older-reader filename limitation is explicit above, and the PR-only review and impact judgments stay in this description.

Review status

The round-3 code approval didn't supersede the soundness r3b UNSOUND verdict at 7aeaac0e95dee171cb15a371f1b425849d6a4436. The mixed-tier provenance and early-year repairs were then reviewed at 7559e6952657d9c1af276c8268017ce09561d28b. That delta review found no P1 and requested description, impact and CI corrections.

The round-2 delta review at 5a69ca8d83 (static read, Opus via Subfleet): APPROVE. All three earlier P2s are resolved. Its P3s were the stale CI text (fixed here) and two limits that are already disclosed: the compound-period filename format, which older Core can't read, and an external-cache export edge case before a reform.

Impact

Hub A/B microsimulation, PE-US, current head. The hub ran PolicyEngine-US main 8b8ae4cc12 on the default dataset twice for each year. One run used Core at the merge base 648f4e467d; the other used this PR's head 5a69ca8d83. The runs went one at a time under the shared heavy-job lock.

Output	2025 change	2026 change	Records changed
household_net_income	0	0	0
income_tax	0	0	0
state_income_tax	0	0	0
household_benefits	0	0	0
household and person-household IDs	0	0	0

Every compared array is identical. Run time was also unchanged: 178s on both cores for 2025, and 187s (base) against 172s (PR) for 2026. The peak footprint was the same on both cores, at 11,264 MB (2025) and 12,288 MB (2026).

Not run: a PE-UK microsimulation. The PR changes country-agnostic holder bookkeeping. PE-UK is exercised only by CI's "Test Core and country packages" jobs, which pass. No partner baseline files change.

The historical results (PE-UK eFRS 2026, 36/36 arrays identical; PE-US eCPS, 3,000 households and marginal rates, 25/25 identical) predate the tier-tracking logic. The PE-US A/B above replaces them for this head.

Dependencies and rulings

• Preserve d899's landing order: Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 → Keep the record of set_input values in step with storage #561 → Cache ADD and DIVIDE results only where a plain read returns them #571 → Count only inputs as already set when a set_input helper splits a longer period #581, with Give each disk-backed holder storage its own directory #558 alongside. Count only inputs as already set when a set_input helper splits a longer period #581 remains dependent on Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576, Keep the record of set_input values in step with storage #561, and Cache ADD and DIVIDE results only where a plain read returns them #571 and held on d899.
• Preserve Max's existing HOLD decisions for Resolve parameter nodes at instant on first read #570 and Evaluate recursions over periods in full when they reach an input or the start of a formula #572. This repair changes no hold, draft status, or landing decision.
• Max's 2026-10-07 principle, “make inputs as leaf-nodey as possible,” remains in force: calculations are not promoted to supplied inputs merely because an input handler ran or a cache survived deletion.
• d1024 Proposal A remains in force: fresh parameter lookups reflect updates. d1026 Proposal A remains in force: each weight column keeps its own entity's total under subsampling. This PR does not supersede either ruling.

axiom: n/a: simulation infrastructure; no policy rule or statute.

[MaxGhenis]

US + core hub merge audit, core#561 at 5a69ca8d835484fc582a6c638c231da7c9af2001 (squash): supplied-input records now follow each storage tier, and compound-period disk filenames are portable to Windows.

• Order: Max's d899 sets Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 → Keep the record of set_input values in step with storage #561 → Cache ADD and DIVIDE results only where a plain read returns them #571 → Count only inputs as already set when a set_input helper splits a longer period #581, with Give each disk-backed holder storage its own directory #558 alongside. Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 and Give each disk-backed holder storage its own directory #558 are merged.
• Review: hub delta review r2 APPROVE at this head (Opus via Subfleet; static read). Its three P3s are disclosed or fixed in the body.
• Gates checked live at merge:
  • every check passes (19 of 19, including native Windows);
  • MERGEABLE;
  • not a draft;
  • no CHANGES_REQUESTED review;
  • head pinned.
• Overlap: master gained Give each disk-backed holder storage its own directory #558 after this PR's CI base, and Give each disk-backed holder storage its own directory #558 touches the same holder, simulation and disk-storage files. The hub merged master into this head locally and ran tests/core on the result: 2,023 passed, 19 skipped, 3 xfailed.
• Impact: a hub A/B microsim of PE-US main 8b8ae4cc12 on the default dataset, comparing Core at the merge base 648f4e467d with this head, for 2025 and 2026. Every compared array is identical, and run time is unchanged (table in the body).
• --admin: used only because a required approving review is the sole block.