Stamp US year files with the rename register they were cut under

changelog.d/532.changed.md

@@ -0,0 +1 @@
+`create_datasets` stamps each US year file with the register of renamed stored inputs it was cut under and a year-file format revision. `ensure_datasets` creates a year file again, and `load_datasets` refuses it, unless its stamp matches this release exactly. A year file cut before a rename was registered, which may have lost that input, is therefore not reused. `load_datasets` says why it refuses a file, naming the renames the file may lack when its register is the reason. Year files written before the stamp existed are created again once.

docs/microsim.md

@@ -401,15 +401,49 @@ The renames applied are recorded as `{old: new}` (`{}` when none applied):
 carried the draw: data that stores neither name runs with the new input's
 default, which for WIC is full take-up.
 
-Files written before this mapping existed have no record, and are not reused:
-
-- A saved US output may have been calculated without the mapped inputs.
-  `load()` refuses it, and `ensure()` runs the simulation again and saves the new
-  output.
-- A year file that `ensure_datasets` or `create_datasets` wrote stores
-  neither name, so the draw is lost from it. `ensure_datasets` creates such
-  year files again, and `load_datasets` refuses them. A year file opened
-  directly, as `PolicyEngineUSDataset(filepath=...)`, is not checked.
+A saved US output written before this mapping existed has no record and may
+have been calculated without the mapped inputs, so `load()` refuses it, and
+`ensure()` runs the simulation again and saves the new output. Saved outputs
+are checked only for the record, not for the register they were calculated
+under.
+
+### Year files
+
+`create_datasets` stamps each year file, after writing everything else, with
+the register it was cut under and a year-file format revision
+(`legacy_inputs.YEAR_FILE_STAMP_H5_DATASET`, UTF-8 JSON
+`{"register": {old: new, ...}, "revision": 1}`). `ensure_datasets` reuses a
+year file only when its stamp equals this release's exactly and its renames
+record lists only renames in the stamped register. Otherwise it logs why and
+creates the year files again; `load_datasets` refuses the file with the same
+reason, which names the renames the file may lack when its register is the
+cause. To decide, both read only the stamp and the record, never the tables
+or the source dataset. `legacy_inputs.year_file_stamp_problem(path)` returns
+the reason, or `None` for a current year file.
+
+A year file is out of date when:
+
+- it has no stamp. Year files that policyengine.py 6.1.1 and earlier cut,
+  with policyengine-us 2.x, from a source storing the draw only as
+  `would_claim_wic` (such as the certified default) store neither WIC take-up
+  name, so the draw is lost from them. Year files written with the renames
+  record but before the stamp are cut again once, and so is a year file whose
+  write did not finish;
+- its register lacks an entry this release has. It may have lost that
+  entry's input, as the older files lost the WIC draw;
+- its register has an entry this release lacks, or maps an old name to
+  another new name. It may store a new name this release's engine does not
+  define, and a run skips any column the engine does not define;
+- it has another format revision, or its stamp or record cannot be read.
+
+So any change to `LEGACY_INPUT_RENAMES` makes every year file out of date,
+and two policyengine.py versions with different registers that share a data
+folder cut each other's year files again. `ensure_datasets` raises, rather
+than cutting the files again, when a year file cannot be opened for reasons
+unrelated to its contents: no permission, a directory at its path, or a lock
+held by another process. A year file opened directly, as
+`PolicyEngineUSDataset(filepath=...)`, is not checked, and saving it again with
+`save()` drops its stamp.
 
 ## Pinned model versions
 

src/policyengine/tax_benefit_models/us/datasets.py

@@ -1,6 +1,7 @@
 import hashlib
 import importlib.util
 import json
+import logging
 import warnings
 from importlib import metadata as importlib_metadata
 from pathlib import Path
@@ -31,9 +32,14 @@
     pending_legacy_input_renames,
     read_renames_record,
     write_renames_record,
+    write_year_file_stamp,
+    year_file_stamp,
+    year_file_stamp_problem,
 )
 from policyengine.utils.hashing import sha256_file
 
+logger = logging.getLogger(__name__)
+
 
 class USYearData(YearData):
     """Entity-level data for a single year."""
@@ -371,6 +377,12 @@ def create_datasets(
 ) -> dict[str, PolicyEngineUSDataset]:
     """Create PolicyEngineUSDataset instances from logical dataset names or URLs.
 
+    Each year file is written to ``{data_folder}/{dataset_stem}_year_{year}.h5``
+    and stamped last with the register of renamed stored inputs it was cut
+    under and the year-file format revision (see
+    ``legacy_inputs.YEAR_FILE_STAMP_H5_DATASET``), which ``ensure_datasets``
+    and ``load_datasets`` check.
+
     Args:
         datasets: List of logical dataset names or HuggingFace dataset URLs
         years: List of years to extract data for
@@ -394,6 +406,8 @@ def create_datasets(
         )
         dataset_stem = source.name
         sim = Microsimulation(dataset=source.path, spm=resolve_spm_selection())
+        # The register the mapping below uses, stamped into each year file.
+        stamp = year_file_stamp()
         # Map stored inputs the engine has since renamed before extracting,
         # so each year's file stores the live input rather than dropping it.
         legacy_input_renames = apply_legacy_input_renames_to_microsimulation(sim)
@@ -555,34 +569,30 @@ def create_datasets(
                 ),
             )
             us_dataset.save()
+            # Stamp last. ``save()`` truncates the file first, so a year file
+            # whose write stopped part-way has no stamp and is cut again.
+            write_year_file_stamp(us_dataset.filepath, stamp)
 
             dataset_key = f"{dataset_stem}_{year}"
             result[dataset_key] = us_dataset
 
     return result
 
 
-def _year_file_records_renames(path: Path) -> bool:
-    """Return whether a year file records the renamed stored inputs mapped.
-
-    ``create_datasets`` writes the record into every year file (``{}`` when
-    nothing needed mapping). Files written before it did may have lost a
-    renamed input such as the WIC take-up draw.
-    """
-    return read_renames_record(path) is not None
-
-
 def load_datasets(
     datasets: Optional[list[str]] = None,
     years: list[int] = [2024, 2025, 2026, 2027, 2028],
     data_folder: str = "./data",
 ) -> dict[str, PolicyEngineUSDataset]:
     """Load PolicyEngineUSDataset instances from saved HDF5 files.
 
-    A year file without the record of renamed stored inputs that
-    ``create_datasets`` writes was cut before those inputs were mapped, and
-    may have lost one (see ``legacy_inputs.RENAMES_H5_DATASET``), so it is
-    refused.
+    A year file is refused unless ``create_datasets`` stamped it with this
+    release's register of renamed stored inputs and year-file format revision
+    (see ``legacy_inputs.year_file_stamp_problem``). A file cut before a
+    register entry was added may have lost that input, as files that
+    policyengine.py 6.1.1 and earlier cut from the certified default lost the
+    WIC take-up draw. The error says why, and names the renames the file may
+    lack when its register is the reason.
 
     Args:
         datasets: List of HuggingFace dataset paths (used to derive file names)
@@ -599,17 +609,14 @@ def load_datasets(
         dataset_stem = dataset_logical_name(resolved_dataset)
         for year in years:
             filepath = f"{data_folder}/{dataset_stem}_year_{year}.h5"
-            if Path(filepath).exists() and not _year_file_records_renames(
-                Path(filepath)
-            ):
-                raise ValueError(
-                    f"US year file {filepath} has no record of the renamed "
-                    "stored inputs mapped when it was cut, so it was written "
-                    "before policyengine.py mapped them and may have lost "
-                    "the WIC take-up draw (every WIC-eligible person would "
-                    "then take WIC up). Regenerate it with ensure_datasets() "
-                    "or create_datasets()."
-                )
+            if Path(filepath).exists():
+                problem = year_file_stamp_problem(filepath)
+                if problem is not None:
+                    raise ValueError(
+                        f"US year file {filepath} is out of date: {problem}. "
+                        "Regenerate it with ensure_datasets() or "
+                        "create_datasets()."
+                    )
             us_dataset = PolicyEngineUSDataset(
                 name=f"{dataset_stem}-year-{year}",
                 description=f"US Dataset for year {year} based on {dataset_stem}",
@@ -1255,10 +1262,13 @@ def ensure_datasets(
 ) -> dict[str, PolicyEngineUSDataset]:
     """Ensure datasets exist, loading if available or creating if not.
 
-    Year files without the record of renamed stored inputs that
-    ``create_datasets`` writes were cut before those inputs were mapped, and
-    may have lost one such as the WIC take-up draw, so they are created
-    again rather than loaded.
+    Existing year files are reused only when ``create_datasets`` stamped them
+    with this release's register of renamed stored inputs and year-file format
+    revision (see ``legacy_inputs.year_file_stamp_problem``). Only each file's
+    stamp and renames record are read to decide, never its tables or the
+    source dataset. If any year file is missing or out of date, every
+    year file is created again from the source dataset, and each out-of-date
+    file is logged with the reason.
 
     Args:
         datasets: List of HuggingFace dataset paths
@@ -1270,22 +1280,29 @@ def ensure_datasets(
     """
     datasets = datasets or [get_release_manifest("us").default_dataset]
 
-    # Check if all dataset files exist
-    all_exist = True
+    # Check that every year file exists and is current. A year file cut
+    # under another register of renamed stored inputs may have lost one, so
+    # it is created again rather than reused.
+    all_current = True
     for dataset in datasets:
         resolved_dataset = resolve_dataset_reference("us", dataset)
         dataset_stem = dataset_logical_name(resolved_dataset)
         for year in years:
             filepath = Path(f"{data_folder}/{dataset_stem}_year_{year}.h5")
-            # A year file written before renamed stored inputs were mapped
-            # may have lost them, so it is regenerated rather than reused.
-            if not filepath.exists() or not _year_file_records_renames(filepath):
-                all_exist = False
-                break
-        if not all_exist:
-            break
-
-    if all_exist:
+            if not filepath.exists():
+                all_current = False
+                continue
+            problem = year_file_stamp_problem(filepath)
+            if problem is not None:
+                all_current = False
+                logger.warning(
+                    "US year file %s is out of date: %s. Creating the year "
+                    "files again.",
+                    filepath,
+                    problem,
+                )
+
+    if all_current:
         return load_datasets(datasets=datasets, years=years, data_folder=data_folder)
     else:
         return create_datasets(