Skip to content

US year-file cuts can pair one cut's tables with another's stamp, and the stamp lacks the year #607

Open

Description

@MaxGhenis

Problem

tax_benefit_models.us.datasets.create_datasets writes each US year file at its final path and stamps it afterwards (#593). Two problems follow.

Interleaved cuts mix one cut's tables with another's stamp. Two processes cutting the same year files in one data folder can interleave:

  1. P1 saves the tables of pop_year_2026.h5.
  2. P2 saves its own tables there, then stamps them.
  3. P1 stamps the file.

The file then holds P2's tables under P1's stamp. Every later ensure_datasets in P1's environment takes the file as current and reuses it. This happens, for example, when two sources share a name (two pop.h5 files) or when two environments with different package versions share a data folder.

A failed cut destroys good year files. When the source changes mid-cut (DatasetMaterializationError), or any later step fails, the year files already at those paths have been overwritten and left unstamped.

The stamp lacks the year. A year file copied or renamed to another year's name passes the check and is taken for that year.

The independent review of #601 found the first two problems for UK year files. #601 fixed them there with policyengine.provenance.year_file_stamp.temporary_year_file / publish_year_file, and added the year to the UK stamp.

Fix

Port #601's approach to the US create path:

  • Write each year file, with its tables and renames record, to a hidden temporary file beside its path.
  • Re-check the source, then stamp each temporary file and os.replace it onto its path.
  • Remove the temporaries in a finally.
  • Record "year" in the US stamp. This raises legacy_inputs.YEAR_FILE_STAMP_REVISION to 3, and year_file_stamp_problem takes the year.
  • Saved outputs that record a year-file stamp (Stamp saved US outputs with the rename register they were calculated under #594) follow the revision.
  • Update docs/microsim.md "Year files", where the "takes no lock" limit changes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions