Skip to content

Tie UK year files to the source dataset and package versions they were cut from #600

Open

Description

@MaxGhenis

Follow-up to #591 / #593, for the UK.

Problem

uk.create_datasets() names each UK year file {data_folder}/{stem}_year_{year}.h5, where the stem is dataset_logical_name(...), which drops the @revision. uk.ensure_datasets() reuses a year file whenever it exists and records its data year (DATA_YEAR_KEY), and uk.load_datasets() refuses one only when it lacks the data year. Nothing records which source a year file was cut from, or under which package versions:

  • A re-pinned managed dataset. When the bundle points enhanced_frs_2024_25, populace_uk_2023 or another logical name at a new revision or sha256, year files cut from the old release keep their names and are reused.
  • An edited local source. A local dataset rewritten in place (allow_unmanaged=True) keeps its year files, which still hold the old data. Two local sources with one file name in different folders share year files too.
  • Another policyengine-uk or policyengine-core version. policyengine-uk projects a single-year source to later years with extend_single_year_dataset, which uprates each year by policyengine-uk parameters read through policyengine-core's parameter tree. Year files cut under one version are reused under another.

#593 fixes all three for US year files.

Proposal

Stamp each UK year file after its tables, as #593 does for the US, with the source identity (dataset_materialization.dataset_source_identity) and the installed policyengine-core and policyengine-uk versions. ensure_datasets() cuts the files again and load_datasets() refuses them unless the stamp equals the one a file cut now from the requested dataset would carry, computed without downloading or reading the source (requested_dataset_source_identity). The check reads only the stamp (and the data-year key) from each year file. Mirror #593's guards: re-check the source before stamping, refuse two sources that share a stem, and treat link cycles and malformed stamps as out of date. Share the stamp code between the countries where it stays simple.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions