Tie US year files to the source dataset and package versions they were cut from

[MaxGhenis]

Fixes #591

The defect

ensure_datasets() names each US year file {data_folder}/{stem}_year_{year}.h5, with the stem from dataset_logical_name(resolve_dataset_reference("us", dataset)). That drops the @revision, and create_datasets() uses the same stem (materialize_dataset(...).name). Since #590, a year file is reused when its stamp holds this release's register of renamed inputs and format revision. Nothing in the stamp says which source the file was cut from, or under which engine:

• A re-pinned bundle dataset. When the bundle points a logical name at another revision or sha256, year files cut from the old release keep their names, pass the stamp check and are reused. This is next for populace_us_2024_acs_local, whose replacement (Publish complete ACS-local WIC participation and remove the temporary .py assumption microcosm#1154) ends the temporary ACS WIC assumption (Remove temporary ACS WIC assumption after replacement certification #563).
• An edited local source. A local dataset rewritten in place (allow_unmanaged=True) keeps its year files. Two local files with the same name in different folders share year files.
• Another engine version. create_datasets stores each input as sim.calculate(variable, period=year) returns it. policyengine-core 3.33.0 uprates an input from its stored year by the policyengine-us parameter named in the variable's uprating (Simulation._calculate; 367 policyengine-us 2.2.1 variables set one, for example employment_income_before_lsr → calibration.gov.irs.soi.employment_income). So year-file values depend on both versions, and neither was recorded.

What changes

The stamp, revision 2 (legacy_inputs.YEAR_FILE_STAMP_REVISION 1 → 2). Example for the certified default:

{"packages": {"policyengine-core": "3.33.0", "policyengine-us": "2.2.1"},
 "register": {"would_claim_wic": "takes_up_wic_if_eligible"},
 "revision": 2,
 "source": {"sha256": "6496cc43…", "uri": "hf://policyengine/populace-us/populace_us_2024.h5@populace-us-2024-spm-20260915"}}

• source is dataset_materialization.dataset_source_identity(materialize_dataset(...)), which has one of three shapes:
  • a bundle dataset: {"uri", "sha256"}, the pinned hf://…@revision URI and the sha256 that materialize_dataset verified the local copy against;
  • an explicit Hugging Face URI: {"uri"}, as given;
  • a local file: {"path", "size", "mtime_ns"}, from one stat() of the resolved path, taken before the Microsimulation loads it.
• packages is legacy_inputs.installed_package_versions(): importlib.metadata versions of YEAR_FILE_PACKAGES = ("policyengine-core", "policyengine-us"), with None when one is not installed. No package is imported.

The check. year_file_stamp_problem(path, source, packages=None) returns None only when the stamp equals year_file_stamp(source, packages) exactly, and the renames record lies within the stamped register (as before). ensure_datasets and load_datasets pass requested_dataset_source_identity("us", dataset), which mirrors materialize_dataset's routing without downloading or reading anything: the bundle manifest entry, the URI, or file-system metadata. The reason names what differs: the renames (as before), both sources, and each package version that changed. For example:

US year file data/populace_us_2024_acs_local_year_2026.h5 is out of date: it was cut from hf://policyengine/populace-us/populace_us_2024_acs_local.h5@populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z (sha256 769756c3…), and the requested source is hf://…@<replacement> (sha256 <replacement>). Regenerate it with ensure_datasets() or create_datasets().

Guards around the cut. create_datasets records the source file's size and mtime right after materializing it (dataset_source_snapshot). Once a dataset's year files are all written, it stamps them only if dataset_source_change finds:

• the same size and mtime, whatever the kind of source;
• for a downloaded file, the same sha256: the one a bundle dataset was verified against, or the one an explicit URI was downloaded with (_download_hugging_face_dataset now hashes as it writes; DatasetSource.downloaded_sha256).

This catches another process that materializes another release, or another file with the same name, into a shared data folder. That includes a same-size download landing within one timestamp tick, and the pinned release downloaded back afterwards. On a change, the year files are left unstamped, so they are cut again, and DatasetMaterializationError is raised.

Two gaps are documented:

• a source swapped out while it loads, then restored to its exact bytes and times (checked when stamped, not as read);
• a local file changed at the same size within one timestamp tick.

Device and inode are deliberately not compared: some FUSE mounts renumber them, which would fail every cut.

Up-front refusals. requested_dataset_source_identity refuses what materialize_dataset would refuse, with the same error: unsupported schemes (s3://), malformed hf:// URIs (a shared _parse_hugging_face_uri), and, from create_datasets, unmanaged requests without allow_unmanaged. So a multi-dataset create_datasets fails before cutting any of them. Two requested datasets that are different sources with one stem, such as pop.h5@r1 and pop.h5@r2, would overwrite each other's year files, and with the new check they would be cut again on every call. ensure_datasets, load_datasets and create_datasets refuse them with ValueError before anything is read or cut.

HDF5 links. A soft-link cycle, or a chain too long to follow, in place of the stamp or the renames record made h5py raise RuntimeError out of year_file_stamp_problem. This was inherited from #590. It now reads as unreadable content.

Year files with a revision-1 stamp are cut again once. Revision 1 exists only on main since #590, which has not been released (6.3.2 is the latest tag).

Decisions

• Local paths: path, size and mtime, not a hash. The brief asked for a cheap check that doesn't open the source, so a local file is compared with the quick check rsync makes by default. Touching or copying the file triggers a recut, which errs on the safe side. A rewrite that keeps both size and mtime (touch -r, or a rewrite within the file system's timestamp resolution) is missed. That blind spot is documented, and a property test labels it as intended.
• Explicit URIs: compared as strings. Their contents were never verified, so the URI is all there is to compare without network. An unpinned or branch URI names whatever the branch holds. The docs say to pin a commit.
• policyengine-core is stamped too. Both packages are pinned exactly in the us extra (==3.33.0, ==2.2.1), so they move only when policyengine.py moves a pin, and core's uprating produces the values. An editable install keeps its version string while its code changes, which the docs note.
• The stamp stays in legacy_inputs, where Stamp US year files with the rename register they were cut under #590 put it. The source identity lives in provenance/dataset_materialization.py, next to materialize_dataset, so the UK year files could reuse it.

Invariants (all executed)

Invariant	Property tests (Hypothesis)	Example and integration tests
Exactness. A year file is current iff its stamp equals the requested one (revision, register, source and packages, compared as JSON, so true ≠ 1 and 1.0 ≠ 1) and its record lies within its register.	test_a_year_file_is_current_only_under_the_exact_stamp_requested (400 examples; sources, versions and registers drawn from small alphabets, so near misses are common)	test_a_year_file_cut_from_another_source_names_both, test_a_year_file_cut_with_other_versions_names_them, test_a_year_file_stamped_by_revision_1_is_cut_again_once
Totality. Any stamp or record content gives a non-empty reason or None, and raises only for permission, directory or lock errors; None exactly for the requested stamp, however the nodes are stored.	test_the_year_file_problem_is_total_and_exact_for_any_content (400 examples, including well-formed keys with wrong-typed values; both directions, with an independent decoder for raw text and bytes)	test_a_malformed_stamp_is_out_of_date (26 cases), test_links_that_cannot_be_followed_are_unreadable_content (cycle and long chain, stamp and record)
Named differences. Each differing source and package appears in exactly one clause; nothing that agrees is mentioned.	test_a_stale_year_file_names_each_source_and_version_that_differs (300), test_a_stale_year_file_names_each_rename_it_may_lack (200)	test_every_difference_is_named_in_order
Agreement (differential). requested_dataset_source_identity(dataset) equals dataset_source_identity(materialize_dataset(dataset, allow_unmanaged=True)) on every route.		test_a_request_is_identified_as_what_it_materializes (12 routes: default, default URI, bundle name, bundle name in another repo, explicit URI with or without a revision, explicit URI of a bundle dataset, data-release name, and local absolute, relative, ~ and symlinked paths)
Sensitivity. A bundle request's identity changes exactly when its entry's repo, path, revision or sha256 does, and never with another entry. A local file's changes exactly when its size or mtime does.	test_a_bundle_dataset_is_re_identified_exactly_when_re_pinned (150), test_a_local_file_is_re_identified_when_its_size_or_mtime_changes (150; equal size and mtime over other bytes is the labelled, intended blind spot)	test_a_local_file_elsewhere_is_another_source, test_re_pointing_a_symlink_re_identifies_the_source
Cheapness. Deciding reads only each year file's stamp and record. It never opens, hashes, downloads or loads the source, and never imports policyengine-us.		test_ensure_decides_from_the_stamps_alone (8 cases, bundle-free URI and local source), test_identifying_a_request_never_reads_the_source (5 routes)
Ensure agrees with load. ensure_datasets cuts again exactly when a file is missing or load_datasets would refuse it, logging each stale file with its reason.	test_ensure_cuts_again_exactly_when_load_would_refuse (80; 19 states, now including another source revision, a verified source, other core/us versions and revision 1)
No stamp names other bytes. A source replaced or modified after it is materialized leaves its year files unstamped. A downloaded file counts as changed whenever its bytes differ, whatever its size and mtime. A local file counts as changed exactly when its size or mtime moved.	test_a_replaced_explicit_download_has_changed_whatever_its_metadata (100), test_a_local_source_has_changed_exactly_when_its_size_or_mtime_has (100; same size and mtime over other bytes is the labelled gap)	test_a_source_replaced_after_its_snapshot_has_changed (bundle, explicit URI, local), test_a_downloaded_file_replaced_within_one_timestamp_tick_has_changed, test_a_bundle_file_swapped_out_and_back_has_changed, test_a_bundle_file_replaced_before_its_snapshot_has_changed, test_an_explicit_download_records_the_sha256_of_its_bytes, test_a_source_replaced_while_it_is_cut_leaves_the_year_files_unstamped (bundle, explicit URI and local, real engine), test_create_datasets_stamps_the_year_files_after_saving_and_checking
No two sources share year files. Requests are refused exactly when two distinct identities share a stem, before any year file is read or cut.	test_two_sources_with_one_stem_are_refused_before_anything_is_cut (60)	test_the_collision_refusal_names_both_requests_and_sources, test_one_source_requested_by_two_names_is_not_a_collision, test_create_refuses_what_materialize_would_with_its_error, test_create_refuses_a_uri_materialize_would_before_cutting_anything, test_a_uri_materialize_refuses_is_refused_with_its_error
End to end, real engine.		test_editing_a_local_source_cuts_its_year_files_again, test_a_re_pinned_bundle_dataset_cuts_its_year_files_again (the ACS-local replacement in miniature: a fixture bundle re-pins a logical name, no network), test_another_package_version_cuts_year_files_again

Mutation testing

Mutation testing used 26 distinct mutants, each applied in a throwaway worktree and run against the tests. All 26 were killed, including both mutants of create_datasets, which only the real-engine integration tests catch. The mutants cover the parts of the change that load-bearing behaviour depends on:

• The stamp check: ignoring the source or the packages in the comparison; no source reason; an empty package reason for a missing key; accepting a bool size; accepting any source as well formed.
• The source identity: a bundle identified without its sha256 or its revision; a local file without its mtime, its size or path resolution; the default URI treated as unmanaged.
• ensure and load: checking every request against the default dataset; installed_package_versions returning nothing; dropping the mtime's nanoseconds from the description.
• create_datasets: stamping the URI string instead of the identity; ignoring the post-cut check; skipping the collision check.
• The post-cut check: ignoring the snapshot, the bundle re-hash or the explicit-download hash; not recording the download hash.
• Up-front refusals: allowing collisions; skipping the hf:// parse; letting a link cycle escape.

Logs and the harness are in ~/PolicyEngine/_reviews/pr593-year-file-source-identity/ on the build machine.

Review

GPT-6.1 Sol reviewed this PR six times through Subfleet. Each round's findings were fixed or explicitly accepted:

1. 64f5eb3, changes requested. The source could be replaced between verification and load; same-stem sources would rebuild forever; HDF5 link cycles escaped the check; two documentation nits.
2. Changes requested. Explicit URIs and bundle files swapped out and back still passed.
3. 716fb0e, approved with low-severity findings. Unstable inodes on FUSE mounts, s3:// passing the up-front check, and documentation of a restore gap.
4. Changes requested. With inode dropped, a same-tick explicit download passed. This led to hashing explicit downloads. Malformed hf:// URIs also passed.
5. Changes requested. One documentation claim was too strong.
6. 8578cc8, approved. That commit is docs-only on top of 0e6b20b, confirmed with git diff --stat.
7. c7bb48d, approved. create_datasets now snapshots and re-checks the source only when year files are cut (years non-empty). test_dataset_runtime mocks a nonexistent bundle path with years=[], and it failed on the snapshot before this. The UK year-file session found the failure.

A sibling review of #594 found that the totality oracle took a raw node's descriptor rather than its decoded value. That is fixed here too.

Accepted, not changed:

• ensure_datasets enforces allow_unmanaged only when it has to cut, so existing year files for an unmanaged source still load without the flag. This predates this PR and is unchanged by it.
• managed_microsimulation could record downloaded_sha256 in its run provenance. That is left for a follow-up.

Verification

• At c7bb48d, the full pytest tests suite passed locally: 1543 passed, 9 skipped (21 min). The integration tests use tiny fixtures; no full-population Microsimulation is built.
• ruff format --check . and ruff check . are clean.
• mypy reports nothing new in the changed modules, compared with origin/main under the same configuration.

Related

• Remove temporary ACS WIC assumption after replacement certification #563 (draft) removes the temporary ACS WIC assumption. Once the bundle re-pins populace_us_2024_acs_local, year files cut from the old pin are cut again by this PR without a new revision. Year files cut from the old dataset requested by its explicit URI still need Remove temporary ACS WIC assumption after replacement certification #563 to raise YEAR_FILE_STAMP_REVISION, as its comment says.
• Support certified annual US projection datasets #524 (draft, stale since 2026-09-20) keys derived files by content-addressed paths instead. If it is revived, it should adopt this stamp or replace it, not both.
• Merge order: after Read each US year file once in load_datasets #596 and Stamp saved US outputs with the rename register they were calculated under #594. Both overlap this PR, in legacy_inputs.py, docs/microsim.md and tests/test_us_year_file_stamp.py. Their sessions plan to land first. This PR will then rebase and adapt them:
  • Read each US year file once in load_datasets #596's test writes year_file_stamp() for hf://fixture/{stem}.h5. That needs a source argument and a well-formed URI.
  • Stamp saved US outputs with the rename register they were calculated under #594's output_stamp_problem will accept a recorded year-file stamp exactly when year_file_stamp(stamp["source"]) == stamp. That is the rule agreed with its session.
• UK year files (uk/datasets.py) record only their data year, so they have the same gap. That is left for a follow-up.

axiom: n/a: year-file caching infrastructure, no policy change.

Country package versions are unchanged.

🤖 Generated with Claude Code

[juaristi22]

@MaxGhenis, two things from a local merge check of #593 against main. These came up while I was looking at #615, but neither needs a change in #615, and #615 leaves #593's rule as it is: an explicit URI stays unverified even when it names a bundle dataset. sim-api already turns a region's URI into the dataset name before it calls ensure_datasets.

1. Rebase break. main plus Tie US year files to the source dataset and package versions they were cut from #593 fails tests/test_us_year_file_stamp.py, for example test_ensure_decides_from_the_stamps_alone and test_ensure_cuts_again_exactly_when_load_would_refuse. They fail with NameError: name 'dataset' is not defined. The legacy-SPM check that Measure SPM poverty on legacy certified US sources (backward compatibility) #599 added calls _expected_legacy_spm_profile(dataset). Merged with Tie US year files to the source dataset and package versions they were cut from #593, that call lands inside the for dataset_stem, source in _year_file_sources(datasets) loop, where dataset isn't defined.

2. Measure SPM poverty on legacy certified US sources (backward compatibility) #599's helper follows the other rule (cc @anth-volk).

   • us/datasets.py _managed_source_sha256 maps a bundle dataset's URI to its certified sha256. load_datasets and ensure_datasets use that hash to pick the legacy-SPM profile a year file needs.
   • create_datasets picks the profile from the materialized file's hash instead. For an explicit URI with allow_unmanaged=True there is no hash, so no declarations are written.
   • So for a URI-form request of a registered legacy source, create and check disagree: the year file is cut without declarations, then judged out of date on every ensure_datasets and refused by load_datasets.
   • It's narrow (opt-out mode, legacy sources only). Making _managed_source_sha256 accept only names and the default URI would match Tie US year files to the source dataset and package versions they were cut from #593's rule.