Package UK area lists and certify UK regional releases

changelog.d/553.added.md

@@ -0,0 +1 @@
+Ship the UK constituency (2024, 650 including Northern Ireland) and local-authority (April 2023, 361 including Ards and North Down) lists with the package, so area regions and geography output labels no longer depend on GCS. Certify a UK regional release with `certify-data --regional-manifest-uri`: a Microcosm local-area release, or, for a national-first switch, the enhanced FRS with its area weight matrices (`--regional-release-kind area_coded_national`). Constituency and local-authority regions load the routed dataset and join the default registry when it is a local-area release. Add the Universal Credit coverage floor check (`scripts/check_uk_uc_floor.py`) for UK default switches.

changelog.d/553.fixed.md

@@ -0,0 +1 @@
+Constituency and local-authority regions were empty when requested, because their GCS download helper no longer exists in policyengine-core. Certification now names a build by its cut tag when the release has one, refuses a regional release that does not serve the pinned model version, and refuses a re-certification that would silently drop a bundle's regional routing (`--drop-regional-release` removes it on purpose).

docs/engineering/skills/data-certification.md

@@ -23,9 +23,75 @@ the checkout is dirty).
 
 ```bash
 python scripts/bundle.py certify-data --country uk --data-producer populace \
-  --manifest-uri "hf://dataset/policyengine/populace-uk-private@<tag>/releases/<tag>/release_manifest.json"
+  --manifest-uri "hf://dataset/policyengine/populace-uk-private@<cut-tag>/releases/<release-id>/release_manifest.json"
 ```
 
+Pin the cut tag (`@<cut-tag>`), never `main`: each cut of a Microcosm release
+line rewrites that line's `releases/<release-id>/` directory, for example
+`releases/microcosm-uk-2024-25-national/`. Microcosm records the line name as
+`build.build_id` on every cut and the cut in `build.cut_tag`; certification
+uses the cut tag as the build id (`data_releases.<country>.version`, the
+certification's `data_build_id` and the changelog fragment name), so two cuts
+of one line never look identical.
+
+### UK constituency and local-authority regions
+
+UK constituency and local-authority regions row-filter the dataset a regional
+release routes them to (`region_datasets.constituency` and
+`region_datasets.local_authority`), and the national dataset when nothing is
+routed. policyengine-sim-api accepts a UK default without area codes
+(`constituency_code_oa`, `la_code_oa`) only when both are routed. Certify a
+Microcosm national release with one of:
+
+- **A Microcosm local-area release (ship together).** The release declares
+  `dataset_role: non_default_local_area`, as for the US below.
+
+  ```bash
+  python scripts/bundle.py certify-data --country uk --data-producer populace \
+    --manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
+    --regional-manifest-uri "hf://dataset/policyengine/populace-uk-private@<local-cut-tag>/releases/<local-release-id>/release_manifest.json" \
+    --model-version "<policyengine-uk-version>"
+  ```
+
+  This records `regional_release_kind: local_area`, and the default UK region
+  registry then includes all 650 constituencies and 361 local authorities.
+
+- **The enhanced FRS (national first).** When the national release is ready
+  before a local one, route the area regions to a certified policyengine-uk-data
+  release whose households carry area codes:
+
+  ```bash
+  python scripts/bundle.py certify-data --country uk --data-producer populace \
+    --manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
+    --regional-manifest-uri "hf://model/policyengine/policyengine-uk-data-private@<revision>/releases/<version>/release_manifest.json" \
+    --regional-release-kind area_coded_national \
+    --model-version "<policyengine-uk-version>"
+  ```
+
+  Certification carries that release's national dataset
+  (`enhanced_frs_2024_25`) and its `weights` artifacts (the constituency and
+  local-authority matrices and `local_geography_weights`) at their own repo
+  and revision, records `regional_release_kind: area_coded_national`, and
+  leaves area regions out of the default registry: the API keeps serving them
+  from the matrices. The enhanced FRS release must itself claim the pinned
+  policyengine-uk version (built with it or listed in
+  `compatible_model_packages`); otherwise policyengine-uk-data must publish a
+  claim first. Its `frs_2024_25` is not carried, so keep it loadable with a
+  `dataset_overlays.uk` entry.
+
+Either way, certification checks the regional release's own model
+compatibility and HEAD-checks every artifact it carries.
+
+### Re-certifying a routed bundle
+
+Certification rewrites `data_releases.<country>` from the manifests it is
+given. While the bundle routes regions (`regional_source_manifest_uri` is set),
+a run without `--regional-manifest-uri` is refused; pass the current regional
+release again (or its replacement), or `--drop-regional-release` to remove the
+routing on purpose.
+
+### US
+
 For US Populace certification, certify the Populace release manifest directly:
 
 ```bash
@@ -109,8 +175,10 @@ Hard failures (certification refuses): missing national default dataset,
 default dataset absent from artifacts, any artifact without a revision pin,
 unreachable certified dataset, missing required supplemental release files
 (for example Populace-US `us_source_coverage.json`), a malformed shared
-local-area release or malformed legacy per-state artifacts when
-`--regional-manifest-uri` is used, unknown country.
+local-area release, area-coded national release or legacy per-state artifacts
+when `--regional-manifest-uri` is used, a regional release that does not serve
+the pinned model version, an unreachable regional artifact, a re-certification
+that would silently drop regional routing, unknown country.
 
 Certification gate: the model version must either exactly match the
 build-time model (`compatibility_basis: built_with_model_package`) or be
@@ -123,6 +191,24 @@ refused: a new data build or a published compatibility claim is required.
 Warnings (recorded, not blocking): artifacts without sha256, and the
 publisher-claim basis above.
 
+## Acceptance evidence for a UK default switch
+
+A PR that changes the UK default dataset carries, besides the UK candidate
+qualification gates, the Universal Credit coverage floor from #491: benefit
+units with Universal Credit and Universal Credit spend in Great Britain within
+10% of DWP's and the OBR's figures.
+
+```bash
+python scripts/check_uk_uc_floor.py --dataset <candidate .h5 or bundle dataset name> \
+  --year 2025 --output uc_floor.json
+```
+
+The benchmarks and their sources are in
+`src/policyengine/data/qualification/uk_uc_floor.json`. The receipt holds
+weighted national totals only. Before quoting any other figure from UK data
+in a PR, apply the UKDS output rules: no unit-record values, no cells under
+10 unweighted cases, no differencing.
+
 ## Release runbooks
 
 Concrete, fill-in-the-id runbooks that replay a specific certification live

docs/regions.md

@@ -91,13 +91,44 @@ england = pe.uk.model.get_region("country/england")
 Pass the registry entry's `scoping_strategy` to `Simulation` when using a
 national UK dataset.
 
+## UK constituency and local-authority regions
+
+The package ships the area lists: the 650 parliamentary constituencies (2024
+boundaries, including Northern Ireland) and the 361 local authority districts
+(April 2023, including Ards and North Down, `N09000011`), each with a name and
+a hex-map position (`src/policyengine/data/uk_geography/`, sources in its
+`SOURCES.md`). Building these regions needs no network access.
+
+Constituency regions (`constituency/<code>`) row-filter `constituency_code_oa`
+and local-authority regions (`local_authority/<code>`) row-filter `la_code_oa`.
+Their `dataset_path` is the dataset the certified bundle routes them to
+(`data_releases.uk.region_datasets`), or `None`, meaning the national dataset,
+when nothing is routed:
+
+| `regional_release_kind` | Area regions load | In the default registry |
+| --- | --- | --- |
+| none (today) | the national dataset | no |
+| `area_coded_national` | another release's area-coded national dataset (the enhanced FRS), while the default is a national file without area codes | no |
+| `local_area` | a certified local-area release | yes |
+
+Pass `include_constituencies=True` and `include_local_authorities=True` to
+`build_uk_region_registry` to include them regardless.
+
+```python
+from policyengine.countries.uk.regions import build_uk_region_registry
+
+registry = build_uk_region_registry(
+    include_constituencies=True, include_local_authorities=True
+)
+registry.get("local_authority/N09000011").label  # "Ards and North Down"
+```
+
 ## UK parliamentary constituencies
 
 Constituency-level impacts group household output rows by the longwise
-`constituency_code_oa` column carried by the dataset. If the constituency CSV is
-available locally or from the PolicyEngine UK GCS bucket, PolicyEngine uses it
-to attach names and map coordinates; otherwise results still compute and use
-the code as the label.
+`constituency_code_oa` column carried by the dataset. Names and map coordinates
+come from the packaged constituency list; codes it does not list use the code
+as the label.
 
 ```python
 from policyengine.outputs import compute_uk_constituency_impacts
@@ -109,10 +140,9 @@ impacts = compute_uk_constituency_impacts(
 impacts.constituency_results
 ```
 
-To force a specific metadata file, pass `constituency_csv_path`. To avoid
-downloading metadata and fall back to code-only labels, pass
-`download_missing_assets=False`. The legacy `weight_matrix_path` and `year`
-arguments are accepted for backward compatibility but ignored.
+To use a different metadata file, pass `constituency_csv_path`. The legacy
+`weight_matrix_path`, `year` and `download_missing_assets` arguments are
+accepted for backward compatibility but ignored.
 
 ## UK local authorities
 
@@ -126,11 +156,15 @@ impacts = compute_uk_local_authority_impacts(
 impacts.local_authority_results
 ```
 
-Local-authority impacts follow the same longwise pattern using `la_code_oa`.
-Pass `local_authority_csv_path` to use a specific metadata CSV, or
-`download_missing_assets=False` to skip metadata download and use code-only
-labels. The legacy `weight_matrix_path` and `year` arguments are accepted for
-backward compatibility but ignored.
+Local-authority impacts follow the same longwise pattern using `la_code_oa`,
+labelled from the packaged April 2023 list. Pass `local_authority_csv_path` to
+use a specific metadata CSV. The legacy `weight_matrix_path`, `year` and
+`download_missing_assets` arguments are accepted for backward compatibility
+but ignored.
+
+Both impacts need simulations on a dataset whose households carry the area
+codes. When the national default does not, run them on the dataset the bundle
+routes the area regions to.
 
 ## Region registries
 

docs/release-bundles.md

@@ -249,6 +249,13 @@ path. The national `default_dataset` remains unchanged. Ordinary bundle
 installation downloads only that national default; a regional runtime is
 responsible for materializing the additional certified dataset.
 
+`data_releases.{country}.regional_release_kind` records how certification
+routed those region types: `local_area` for a non-default local-area release,
+and `area_coded_national` (UK only) for another release's national dataset
+whose households carry area codes, certified together with that release's
+area weight matrices at their own repo and revision. It is absent when nothing
+is routed, or for a legacy US per-state release.
+
 Cross-package overlays must declare `data_package_name` and `repo_type`
 explicitly. Ordinary artifacts inherit these values from the country release's
 primary `data_package`. This prevents runtime code from guessing how a

scripts/bundle.py

@@ -44,6 +44,10 @@ def _certify_data(args: argparse.Namespace) -> int:
         argv.extend(["--regional-artifact-prefix", args.regional_artifact_prefix])
     if args.regional_path_template:
         argv.extend(["--regional-path-template", args.regional_path_template])
+    if args.regional_release_kind:
+        argv.extend(["--regional-release-kind", args.regional_release_kind])
+    if args.drop_regional_release:
+        argv.append("--drop-regional-release")
     if args.no_generate:
         argv.append("--no-generate")
     if args.no_changelog:
@@ -196,10 +200,25 @@ def _parser() -> argparse.ArgumentParser:
     certify.add_argument(
         "--regional-manifest-uri",
         help=(
-            "Optional regional release_manifest.json URI to merge into US "
-            "Populace certification."
+            "Regional release_manifest.json URI whose dataset serves the local "
+            "region types (US states and districts, UK constituencies and local "
+            "authorities)."
         ),
     )
+    certify.add_argument(
+        "--regional-release-kind",
+        choices=["local_area", "area_coded_national"],
+        help=(
+            "How to read the regional release. Defaults to local_area; "
+            "area_coded_national (UK only) routes area regions to another "
+            "national release's area-coded dataset and weight matrices."
+        ),
+    )
+    certify.add_argument(
+        "--drop-regional-release",
+        action="store_true",
+        help="Remove the bundle's current regional routing on re-certification.",
+    )
     certify.add_argument(
         "--regional-artifact-prefix",
         help="Legacy per-state artifact prefix to import. Defaults to states/.",

scripts/certify_data_release.py

@@ -8,7 +8,11 @@
 Usage::
 
     python scripts/bundle.py certify-data --country uk --data-producer populace \\
-        --manifest-uri "hf://dataset/policyengine/populace-uk-private@<tag>/releases/<tag>/release_manifest.json"
+        --manifest-uri "hf://dataset/policyengine/populace-uk-private@<cut-tag>/releases/<release-id>/release_manifest.json"
+
+Pin the cut tag, never a branch: each cut rewrites ``releases/<release-id>/``.
+Add ``--regional-manifest-uri`` to route local regions to a regional release
+(see docs/engineering/skills/data-certification.md).
 
 Private data (UK) requires ``HUGGING_FACE_TOKEN`` or ``HF_TOKEN`` in the
 environment. After running: commit the changed bundle manifest, pyproject.toml,
@@ -53,10 +57,29 @@ def main(argv=None) -> int:
         "--regional-manifest-uri",
         default=None,
         help=(
-            "Optional regional data release manifest to merge into US Populace "
-            "certification."
+            "Regional release_manifest.json URI whose dataset serves the local "
+            "region types (US states and districts, UK constituencies and local "
+            "authorities). Required on re-certification while the bundle routes "
+            "regions, unless --drop-regional-release is given."
+        ),
+    )
+    parser.add_argument(
+        "--regional-release-kind",
+        default="local_area",
+        choices=["local_area", "area_coded_national"],
+        help=(
+            "local_area: a non-default local-area release. "
+            "area_coded_national (UK only): another certified national release "
+            "whose households carry area codes, such as the enhanced FRS; its "
+            "national dataset and area weight matrices serve constituency and "
+            "local-authority regions."
         ),
     )
+    parser.add_argument(
+        "--drop-regional-release",
+        action="store_true",
+        help="Remove the bundle's current regional routing on re-certification.",
+    )
     parser.add_argument(
         "--regional-artifact-prefix",
         default="states/",
@@ -102,6 +125,8 @@ def main(argv=None) -> int:
         regional_manifest_uri=args.regional_manifest_uri,
         regional_artifact_prefix=args.regional_artifact_prefix,
         regional_path_template=args.regional_path_template,
+        regional_release_kind=args.regional_release_kind,
+        drop_regional_release=args.drop_regional_release,
     )
     print(result.summary())
 

scripts/check_uk_uc_floor.py

@@ -0,0 +1,57 @@
+"""Check a UK dataset against the Universal Credit coverage floor.
+
+Usage::
+
+    python scripts/check_uk_uc_floor.py --dataset <bundle dataset name or .h5 path> \\
+        [--year 2025] [--output receipt.json]
+
+A bundle dataset name (for example ``enhanced_frs_2024_25``) is materialized
+from the certified bundle; private UK data needs ``HUGGING_FACE_TOKEN``. Exits
+non-zero when either measure is outside the tolerance. The receipt holds
+weighted national totals only, so it can be attached to a certification PR.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(REPO_ROOT / "src"))
+
+from policyengine.provenance.uk_uc_floor import check_uc_floor  # noqa: E402
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--dataset",
+        required=True,
+        help="A bundle dataset name or a local UK dataset .h5 path.",
+    )
+    parser.add_argument("--year", type=int, default=2025)
+    parser.add_argument("--output", type=Path, help="Write the JSON receipt here.")
+    args = parser.parse_args(argv)
+
+    if Path(args.dataset).expanduser().is_file():
+        dataset_path = str(Path(args.dataset).expanduser())
+        dataset = Path(dataset_path).name
+    else:
+        from policyengine.provenance.dataset_materialization import (
+            materialize_dataset,
+        )
+
+        source = materialize_dataset("uk", args.dataset)
+        dataset_path, dataset = source.path, source.source_uri
+
+    receipt = check_uc_floor(dataset_path, args.year, dataset=dataset)
+    text = receipt.model_dump_json(indent=2)
+    if args.output:
+        args.output.write_text(text + "\n")
+    print(text)
+    return 0 if receipt.passed else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())