You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
US year-file cuts can pair one cut's tables with another's stamp, and the stamp lacks the year #607
tax_benefit_models.us.datasets.create_datasets writes each US year file at its final path and stamps it afterwards (#593). Two problems follow.
Interleaved cuts mix one cut's tables with another's stamp. Two processes cutting the same year files in one data folder can interleave:
P1 saves the tables of pop_year_2026.h5.
P2 saves its own tables there, then stamps them.
P1 stamps the file.
The file then holds P2's tables under P1's stamp. Every later ensure_datasets in P1's environment takes the file as current and reuses it. This happens, for example, when two sources share a name (two pop.h5 files) or when two environments with different package versions share a data folder.
A failed cut destroys good year files. When the source changes mid-cut (DatasetMaterializationError), or any later step fails, the year files already at those paths have been overwritten and left unstamped.
The stamp lacks the year. A year file copied or renamed to another year's name passes the check and is taken for that year.
The independent review of #601 found the first two problems for UK year files. #601 fixed them there with policyengine.provenance.year_file_stamp.temporary_year_file / publish_year_file, and added the year to the UK stamp.
Activity