Stamp saved UK outputs with the year file they were calculated from

[MaxGhenis]

Fixes #606

Stacked on #601 (UK year-file stamp), which is stacked on #593. The base is fix/uk-year-file-source-identity; retarget to main once #601 merges. Merge order: #596, #594, #593, #601, then this. It mirrors #594 (the US output stamp) and will conflict textually with it in common/model_version.py (save() / load()), where the UK branches sit beside the US ones.

Status: reworked after the first independent review (REQUEST_CHANGES, four findings, all addressed below). Second review in progress; mutation testing queued behind a disk-space hold on the build host.

Problem

A saved UK output was checked only for a recorded data year. After #601 every UK year file carries a stamp (source identity, policyengine-core and policyengine-uk versions, year, format revision), but nothing tied a saved output to it. When a year file was cut again, because the bundle re-pinned the dataset, a local source was edited or policyengine-uk was upgraded, Simulation.ensure() and load() still reused outputs calculated from the old cut.

Change

• A UK dataset keeps its year file's stamp. PolicyEngineUKDataset.metadata is a new field, as on the US dataset. A dataset read from a year file, or returned by create_datasets, keeps the file's stamp under "year_file_stamp".
• A UK run stamps its output. It records uk.datasets.output_stamp(year_file) in the output's metadata["output_stamp"]:
  {"packages": {policyengine-core, policyengine-uk versions installed at the run}, "revision": 1, "year_file": <the input dataset's year-file stamp, or null>}.
• load() checks the whole stamp. Simulation.load() refuses a saved UK output, and Simulation.ensure() runs it again, unless its stamp equals the one a run of the simulation's dataset would record now (uk.datasets.output_stamp_problem). The error names the years, sources or package versions that differ.
• A UK output is saved in one step. uk.datasets.save_output writes the tables, the data year and the stamp to a temporary file beside the path and moves it onto the path, reusing Tie UK year files to the source dataset and package versions they were cut from #601's temporary_year_file / publish_year_file. A save that fails leaves the previous output whole.
• A load reads one file. provenance.year_file_stamp.read_one_file (new) holds the file open while it is read, then checks that the path still names that file with the same size and modification time, and reads again if not. PolicyEngineUKDataset.load() reads the year-file stamp and the tables through it, and the output load runs its check and reads the tables through it.
• PolicyEngineUKDataset.load() now raises ValueError for a dataset with no filepath (it raised pandas' TypeError before).
• provenance.year_file_stamp.package_differences(stamped, packages, done="cut") is split out of Tie UK year files to the source dataset and package versions they were cut from #601's source_and_package_differences, so the output check words the same clauses as "calculated with". The year-file clauses are unchanged: I compared the two implementations over every pair of package maps on four names (196,608 stamp pairs).

Design decisions

The output is compared with the simulation's dataset, not with the requested source. #594 compares a US output's year-file stamp with the one this release would write, which the release alone determined when #594 was written. A UK year-file stamp also depends on the source and the year, and an output cannot know which dataset was requested. So the check asks what a run of this simulation would record now: the stamp its dataset keeps of the tables it holds. That is what makes a re-pin or an edited source invalidate outputs: ensure_datasets cuts the year file again with another stamp, the simulation's dataset is read from the new cut, and the old output's recorded stamp no longer equals it. It also refuses an output saved under the same id for another year's file in the same folder.

The output records the versions that calculated it, separately from the year file's. The first review showed these can differ: a year file cut under one policyengine-uk and opened directly under another is run with the other. So the stamp has its own packages, and an upgrade makes every saved UK output out of date at once, including outputs calculated from data built in memory.

Whether a year file is itself current is left to load_datasets. An output calculated from an out-of-date year file opened directly is what a run of that file gives now, so it is reused with that file. This differs from #594, which refuses outputs from year files that are not this release's.

Invariants

• Exactness. output_stamp_problem(path, year_file, packages) is None exactly when the file records its data year, its stored stamp equals output_stamp(year_file, packages) as JSON, and year_file is null or a well-formed UK year-file stamp.
• Totality. Whatever the stamp, the data-year node and the dataset's stamp hold, the check returns a reason or None. It raises only for a missing file and for errors unrelated to the file's contents (no permission, a directory, a lock).
• Round trip. A stamp reads back as written; writing again replaces it.
• Named differences. The reason names every year, format revision, source and package version that differs, and nothing that agrees.
• No partial output is current. Every truncation of a current output is out of date.
• Distinct stamps. A UK output's stamp has its own H5 dataset name, so an output never passes as a year file, a year file as an output, or a US output as a UK one.
• One save per output. The tables and the stamp at an output's path come from one save: a failed save leaves the previous output byte-for-byte, and two saves to one path never leave one's tables under the other's stamp.
• One file per load. A loaded dataset keeps the stamp of the cut its tables came from, even when the file is replaced, replaced and put back, or rewritten in place while it is read.
• Ensure agrees with load. ensure() runs the simulation exactly when the output is missing or load() would refuse it. The output it then saves is current exactly when the dataset keeps no year-file stamp or a well-formed one.

Limits (documented in docs/microsim.md)

• The check compares the output with the simulation's dataset, not with the source requested now (see the third design decision).
• The stamp records policyengine-core and policyengine-uk, not policyengine.py's own version. A change to how policyengine.py runs a UK simulation is marked by raising OUTPUT_STAMP_REVISION.
• The versions are those installed when the run starts, read from the distributions' metadata, as for year files.
• A dataset built in memory, and a year file cut before the stamp existed and opened directly, keep no stamp, so two such datasets under one simulation id in one folder are not told apart.
• Editing a dataset's tables in memory does not change the stamp it keeps.
• Within one process, ensure() returns the output it holds in memory for a simulation id without checking its stamp. That cache is keyed by id only, for both countries, and is not changed here.
• read_one_file does not notice a file rewritten in place to the same size within one tick of the file system's modification time, or a file replaced and put back by moving another hard link to it onto the path. Neither create_datasets nor save_output writes that way.

Tests

• tests/test_uk_output_stamp.py (162 tests, engine-free): pytest and Hypothesis properties for the invariants above, including a file replaced and put back during a load, a file rewritten in place, and one save interleaved inside another.
• tests/test_uk_output_stamp_integration.py (26 tests): real policyengine-uk runs on the seven-person fixture. Each replays one cause: a bundle re-pin, an edited local source, a policyengine-uk or policyengine-core upgrade, a year file run with other versions than cut it and those versions then restored, another year's file, a raised year-file revision, an output saved by an earlier release, a failed save, and another process's output moved onto the path between the check and the read. Each checks that load() refuses the old output with the exact reason, ensure() runs once, the new output matches a direct policyengine-uk run of the new source, and it is then reused. A Hypothesis property checks that ensure agrees with load.
• No full-population simulation is built.

🤖 Generated with Claude Code

[juaristi22]

A coordination note from #621, which records the capital gains realisation response each UK run applies (part of #602), as Vahid asked on #621.

• Expected order: Record the capital gains realisation response each UK run applies #621 first. It isn't stacked, and step 5 of Adopt CenTax’s retention-rate elasticity of 1.0 as the default for UK CGT modelling #602 depends on it. This PR follows Read each US year file once in load_datasets #596, Stamp saved US outputs with the rename register they were calculated under #594, Tie US year files to the source dataset and package versions they were cut from #593 and Tie UK year files to the source dataset and package versions they were cut from #601.
• Where they overlap. Both add metadata to PolicyEngineUKDataset, and both add checks to the UK branch of load() in common/model_version.py. Whichever lands second moves its lines into the other's structure. If this one lands second, Record the capital gains realisation response each UK run applies #621's record would be written by save_output and checked in the output load beside output_stamp_problem.
• One rerun, not two. Both refuse saved UK outputs that predate them. Landing them in the same policyengine.py release means users rerun each saved UK output once.
• Two things Record the capital gains realisation response each UK run applies #621 adds that this PR could reuse:
  • StaleOutputError (a ValueError in policyengine.core). load() raises it for every refused output, and Simulation.ensure() logs the reason at INFO instead of a warning with a traceback. output_stamp_problem refusals could raise it too.
  • The policyengine-uk install source, read from direct_url.json. Record the capital gains realisation response each UK run applies #621 records it because a branch installed from git or a local directory can report the version of a published release with different code. The package versions in this PR's output stamp have the same gap.