Tie UK year files to the source dataset and package versions they were cut from

[MaxGhenis]

Fixes #600

Draft, stacked on #593 (fix/us-year-file-source-identity, not merged yet; rebased onto its head c7bb48d; this PR's head is a2244c8). The base is that branch, so the diff shows only this PR; retarget to main once #593 merges (merge order: #596 → #594 → #593 → this).

Problem

UK year files ({data_folder}/{stem}_year_{year}.h5) are named after the dataset's logical name, which drops the revision. ensure_datasets reused a year file whenever it recorded a data year, and load_datasets refused one only without it, so year files were reused after the bundle re-pinned a name (enhanced_frs_2024_25, populace_uk_2023, …), after a local source was edited, and after a policyengine-uk or policyengine-core upgrade.

Change

• Stamp. uk.create_datasets stamps each year file with {"packages": {"policyengine-core", "policyengine-uk"}, "revision": 1, "source": <dataset_source_identity>, "year": <int>} in its own H5 dataset policyengine_uk_year_file_stamp (uk.datasets.YEAR_FILE_STAMP_H5_DATASET). US stamps keep policyengine_year_file_stamp, so a year file of one country never passes as one of the other. The year means a copied or renamed year file is not taken for another year.
• Check. uk.ensure_datasets cuts the files again, and uk.load_datasets refuses them, unless the file records its data year and its stamp equals year_file_stamp(requested_dataset_source_identity("uk", dataset), year). uk.datasets.year_file_stamp_problem(path, source, year) reads only the stamp and whether the data_year node exists (h5py), never the tables or the source, and names what differs. Link cycles, dangling links and malformed stamps read as out of date. Permission, directory and lock errors are raised.
• Atomic cut. Each year file is written to a hidden temporary beside its path (.{stem}_year_{year}.<hex>.partial, created 0o666 less umask). After the source re-check (snapshot after materializing; downloads re-hashed, as in Tie US year files to the source dataset and package versions they were cut from #593), each temporary is stamped and then os.replaced onto its path. A path's tables and stamp therefore always come from one cut, even with concurrent cuts. A failed cut writes none of that dataset's year files, leaves the existing ones whole, and removes its temporaries.
• Guards from Tie US year files to the source dataset and package versions they were cut from #593. Two requested sources that share a stem are refused by create, load and ensure. create_datasets(years=[]) snapshots nothing (mirrors c7bb48d).
• Shared code. policyengine.provenance.year_file_stamp holds the country-neutral parts: JSON node I/O (read_json_dataset, read_stored_json, write_json_dataset), sentinels, installed_package_versions(names), source_and_package_differences, year_file_sources(country, …), temporary_year_file, publish_year_file. us/legacy_inputs.py and us/datasets.py use them; US behaviour and messages are unchanged (Tie US year files to the source dataset and package versions they were cut from #593's US suites pass with one patch target moved: legacy_inputs.json.loads → json.loads).

Invariants (each property-tested)

• For every year file and request: ensure_datasets reuses it ⇔ load_datasets accepts it ⇔ it records its data year ∧ its stamp == year_file_stamp(requested source, year, installed packages) (Hypothesis, with an oracle that decodes stored nodes independently of the checker).
• Totality: any stamp node × any data-year node gives a reason or None, never an error other than permission/directory/lock.
• Deciding never opens tables, PyTables, the source, the network, or imports policyengine-uk.
• No path ever holds one cut's tables under another cut's stamp; no stamp names a source that changed between materializing and stamping.
• No two different sources share year files.
• Differential: the UK and US checks name source and package differences with identical clauses.

Tests

• tests/test_uk_year_file_stamp.py: unit + Hypothesis, no engine.
• tests/test_uk_year_file_stamp_integration.py: real policyengine-uk on the seven-person 2024 fixture. Covers a re-pinned bundle dataset, an edited local source, policyengine-uk / policyengine-core upgrades, a source replaced mid-cut, interleaved concurrent cuts, a failed cut over existing files, save() failing mid-cut, two sources in one call, pre-stamp files, a float year, and file mode.
• tests/test_year_file_stamp.py: the shared module (publish ordering, temporary mode under several umasks, collision refusal).
• Local, at 9378b3c: 389 passed across the UK, US, shared, identity, runtime and US-integration modules, and 235 more in the other modules that touch the changed code.
• CI at a2244c8: all 12 checks pass (Lint, Mypy, changelog, smoke-import on 3.11–3.14, Test on 3.11–3.14, release inputs).

Review and mutation testing

• Independent reviews (Opus 5.5 via Subfleet; the GPT-6.1 Sol lanes were closed until ~17:00Z). Round 1 approved 4325d16 with a medium finding (concurrent cuts could mis-stamp) and four lows. All were fixed: atomic publish, year in the stamp, a UK stamp name, distinct no-data-year reasons, docs limits. Round 2 requested changes on c6f597b (published files became 0600 via mkstemp; float-year key). Fixed. Round 3 approved 568c9cf; its nits are applied in 9378b3c. A full review of the final head a2244c8 is queued (--task review --tier hard: GPT-6.1 Sol when the Codex lanes reopen at about 23:05Z, they were open only 16:40–17:06Z; Opus if a Claude lane admits it first).
• Mutation testing: all 44 mutants killed at 568c9cf (check, create_datasets, load/ensure, shared module; unmutated baseline passes). At the final head b8bff75, the create-path mutants were re-run against the integration tests without -x: non_atomic_cut (restores the old write-then-stamp cut) fails 4 tests including the concurrency test; snapshot_never and create_ignores_recheck fail the source-replaced tests.
• Evidence: ~/PolicyEngine/_reviews/pr601-uk-year-file-source-identity/.

Documented limits

Versions are read from metadata when the files are cut, so an upgrade inside a running process after import is recorded at its new version. numpy's seeded draws for student loan plans aren't recorded. A multi-year source is read as stored. Moving a year file into place needs write access to the data folder and gives it the cutting user's mode and group. Saved UK outputs aren't yet tied to the year-file stamp (follow-up proposed).

Rebase notes

#594 adds code in legacy_inputs.py that uses _read_json_dataset, _write_json_dataset, _ABSENT and _UNREADABLE. These are now imported from the shared module under the same names; add UNREADABLE as _UNREADABLE back when rebasing onto #594. Its _year_file_reasons should call _source_and_package_differences once #593 adds source and packages to it.

axiom: n/a: infra (year-file caching and provenance)

🤖 Generated with Claude Code