Repository navigation
ensure_datasets: consume Microcosm-published UK releases (manifest-verified), so analyses can swap bases by URI #495
Description
Doctrine says policyengine.py is the permanent interface and Microcosm is where UK data construction is migrating (PolicyEngine/microcosm#665). The consumption path is currently missing: pe.uk.ensure_datasets materialises per-year files from hf://policyengine/policyengine-uk-data/..., and there is no way to point it at a Microcosm-published UK release (certified single-year artifact + release manifest with pinned SHA-256s).
Concrete need, with a live consumer waiting: PolicyEngine/uk-cgt-paper#6 and PolicyEngine/uk-cgt-reform#1 — both reform analyses should rerun on the Microcosm UK build once it certifies (currently gated on microcosm#630), ideally changing nothing but a dataset URI.
Scope sketch:
- accept a Microcosm release identifier (HF repo + release id) in
ensure_datasets; - verify artifact SHA-256 against the release manifest before use (Microcosm releases pin these; the interface should check, matching Microcosm's own runtime verification discipline);
- resolve the release's
compatible_model_packages/compatible_core_packagesconstraints and fail loudly on mismatch with the installed bundle; - forward-year materialisation (uprating 2023 → simulation years) with the same semantics the uk-data path applies today, so results are comparable across the swap.
Related: #491 (publisher claim). Happy to pair on the design — the UK CGT arc (microcosm#552) is the first artifact that wants this path.
Activity
MaxGhenis commentedon Aug 19, 2026
Scope correction after reading the bundle machinery: the consumption path I claimed was missing largely exists — the bundle manifest's UK entry already carries a microcosm release as the default dataset (revision-addressed URI, release-manifest URI, certification block pairing it to the model version), and ensure_datasets resolves it. What remains of this issue: (1) the paired update ceremony for a NEW release (scripts/prepare_package_bundle_update.py handles package pins only; the dataset entry + certification block need the same scripted path), (2) runtime sha256 verification of the resolved artifact against the release manifest, matching microcosm's own discipline, and (3) forward-year materialisation semantics documented for the microcosm artifact. Next concrete consumer: pairing policyengine-uk 2.91.0 with the first CGT-family release (supersedes the closed #496 approach).
MaxGhenis commentedon Aug 20, 2026
Further scope collapse after reading docs/engineering/skills/data-certification.md (which I should have read first): the paired-update ceremony already exists —
scripts/bundle.py certify-data --country uk --data-producer populace --manifest-uri hf://…fetches and validates the release manifest, writes the bundle entry, exact-pins the model package, regenerates derived metadata, and writes the fragment, with the certification gate refusing pairs outside built-with or the publisher's compatible_model_packages claim.So the remaining substance here is exactly one gap, confirmed by reading the resolve path:
ensure_datasets→materialize_dataset_sourceperforms no byte-level sha256 verification (thebundle installCLI path hashes; the ensure path does not). PR for that incoming.One operational note for the next UK certification: the certification gate needs the new microcosm release to either be built with the model version being certified or declare it in
compatible_model_packages— so a release built with 2.89.x pairs with 2.91.0 only if the release manifest carries that claim.