Revive the UK leg as an internal audit of policyengine-uk

docs/audit.md

@@ -117,15 +117,28 @@ The defaults round answers to the nearest dollar, which merges answers such as
 model whose own answer is within the tolerance of an amount reference counts as
 exact and never toward a wrong cluster, even when its rounded key misses.
 
+### The UK leg
+
+`consensus-flags` and `adversary-prepare` take `--country uk`. They read the
+`uk/data.json` that `export-full-run --country uk` writes, or the `uk` entry
+of a combined payload. UK cases get UK prompts: the 2026-27 fiscal year,
+legislation.gov.uk and GOV.UK as primary sources, pounds, the region, and a
+2026-10-10 law freeze (policyengine-uk 2.125.1). The UK board has six models,
+so pass thresholds that fit it, for example `--min-models 2 --top-k 3
+--min-top 2 --zero-cluster-min-models 2`. A UK derivation is the
+engine's own computation log, pruned to the steps that carry a value, in
+`uk_case_reference_explanations.csv`. US prompts are unchanged byte for byte.
+
 ## Two stages
 
 - **Stage 1, law first.** The judge sees the household prompt, the output's
   definition from `benchmark_specs.json`, the consensus answer with each
   member's model id, answer and explanation, and the reference value. It does
   not see the engine derivation. It works the answer from primary law
   (statutes, regulations, agency publications, forms and instructions), cites
-  each rule with its publication date and whether it predates the 2026-07-03
-  reference freeze, and says which answer the law supports.
+  each rule with its publication date and whether it predates the country's
+  reference freeze (US 2026-07-03, UK 2026-10-10), and says which answer the
+  law supports.
   - The Claude runner gives it only web tools (no file access) and denies
     WebFetch on the PolicyBench, PolicyEngine and GitHub domains
     (`BLOCKED_DOMAINS`). WebSearch has no deny rule, so its results reach the

docs/benchmark_card.md

@@ -409,6 +409,57 @@ as a public UK transfer path for benchmarking, not as a replacement for enhanced
 Family Resources Survey (FRS) microdata or as a population-representative UK
 household sample.
 
+The UK path samples households from the public calibrated transfer dataset in
+`PolicyEngine/policyengine-uk-data` (`enhanced_cps_2025.h5`, pinned to commit
+`6b1f80e` and its sha256). Those records are PolicyEngine US households
+re-weighted to UK targets, not UK survey records, and they carry no State
+Pension: every pension-age person in the sample has none, and the prompt says
+so by listing none. Since October 2026 the path works as follows.
+
+- **Reference-period inputs.** The artifact stores the 2025 survey year.
+  PolicyEngine UK uprates stored incomes, rents and savings when it calculates
+  2026-27, so the prompt shows the uprated 2026-27 amounts the reference uses.
+- **References from the prompted facts.** Each UK reference is a PolicyEngine
+  UK household simulation of exactly the facts the prompt lists, one simulation
+  per household, as on the US path. Facts the source record holds and the
+  prompt omits (for example Universal Credit deductions, or the seeded draw
+  that marks some 2026 health-element claimants as new) cannot move a
+  reference. `calculate_uk_transfer_microsimulation_values` recomputes the
+  outputs from the full records as a cross-check.
+- **Derived facts stated.** Where PolicyEngine UK derives a fact that changes
+  an output, the prompt states it: each person's sex and date of birth (State
+  Pension age and the Pension Credit savings credit cutoff), the education
+  status of anyone aged 16 to 19 (qualifying young person rules), and an
+  adult's limited capability for work and work-related activity for Universal
+  Credit, held since before 6 April 2026.
+- **Relationships and locality stated.** The prompt says two listed adults are
+  a couple who are not married or in a civil partnership, and the reference
+  supplies exactly that (claimant and partner flags, no marriage), so neither
+  PE-UK's presumed-child rule nor its presumption that couples are married
+  applies. A private renter's prompt states a Broad Rental Market Area: a
+  fixed area within each region, since the transfer data records none and
+  PE-UK otherwise reads Maidstone's Local Housing Allowance everywhere.
+  Someone in non-advanced education has the age it began stated (16 where
+  the scenario gives none).
+- **Facts as shown.** Every UK amount is the whole number the prompt shows,
+  and every rate the four significant figures it shows, so the reference uses
+  the stated figure, not an unrounded uprated one.
+- **Pension contributions.** PolicyEngine UK deducts both kinds of
+  contribution from taxable income and neither from adjusted net income, so
+  they do not lower the High Income Child Benefit Charge or the personal
+  allowance taper (policyengine-uk #2243). The prompt states that convention
+  for each: employee contributions under a net pay arrangement, and personal
+  contributions as a gross amount. It does not call personal contributions
+  relief at source, which works differently and which the reference does not
+  model.
+- **One canonical scenario.** The prompt renderer and the reference builder
+  both start from `canonical_uk_scenario`, which applies these conventions,
+  so a scenario loaded from a manifest gives each the same facts.
+- **Engine-version guards.** Reference generation stops if the artifact stores
+  a variable the installed PolicyEngine UK does not define, or if a derived
+  fact the prompt states is renamed, so a model release cannot silently drop
+  an input from the reference while it still reaches the prompt.
+
 ## Naming discipline
 
 Public prose should prefer:

policybench/cli.py

@@ -594,6 +594,12 @@ def main():
         help="Frozen run payload (data.json.gz) or a release dashboard-data.json",
     )
     consensus_parser.add_argument("--output", required=True, help="Report JSON path")
+    consensus_parser.add_argument(
+        "--country",
+        choices=["us", "uk"],
+        default="us",
+        help="Country payload to read (default us)",
+    )
     consensus_parser.add_argument(
         "--prototype",
         action="store_true",
@@ -624,6 +630,12 @@ def main():
         help="Write two-stage reference-adversary prompts for consensus-flagged cells",
     )
     adversary_prepare_parser.add_argument("--payload", required=True)
+    adversary_prepare_parser.add_argument(
+        "--country",
+        choices=["us", "uk"],
+        default="us",
+        help="Country payload to read (default us)",
+    )
     adversary_prepare_parser.add_argument(
         "--flags", required=True, help="consensus-flags report JSON"
     )
@@ -1434,7 +1446,7 @@ def main():
             ConsensusParams,
             consensus_report,
             file_sha256,
-            load_us_payload,
+            load_payload,
         )
 
         params = PROTOTYPE_PARAMS if args.prototype else ConsensusParams()
@@ -1454,7 +1466,7 @@ def main():
         params = replace(params, **overrides)
         payload_path = Path(args.payload)
         report = consensus_report(
-            load_us_payload(payload_path),
+            load_payload(payload_path, args.country),
             params,
             source=str(payload_path),
             source_sha256=file_sha256(payload_path),
@@ -1467,7 +1479,7 @@ def main():
         )
 
     elif args.command == "adversary-prepare":
-        from policybench.consensus import load_us_payload
+        from policybench.consensus import load_payload
         from policybench.reference_adversary import (
             build_adversary_cases,
             load_derivations,
@@ -1477,7 +1489,7 @@ def main():
         flags_report = json.loads(Path(args.flags).read_text())
         flags = flags_report["flags"]
         payload_path = Path(args.payload)
-        payload = load_us_payload(payload_path)
+        payload = load_payload(payload_path, args.country)
         adversary_dir = Path(args.adversary_dir)
         skipped: dict[tuple[str, str], str] = {}
         if args.skip_cells:
@@ -1570,7 +1582,7 @@ def main():
             }
         kept = [f for f in unheld if (f["scenario_id"], f["variable"]) not in skipped]
         derivations = (
-            load_derivations(Path(args.annotations_dir))
+            load_derivations(Path(args.annotations_dir), args.country)
             if args.annotations_dir
             else None
         )

policybench/consensus.py

@@ -123,13 +123,19 @@ def to_dict(self) -> dict[str, Any]:
 PROTOTYPE_PARAMS = ConsensusParams(answer_rounding="truncate", binary_outputs="skip")
 
 
-def load_us_payload(path: Path | str) -> dict:
-    """Read a US dashboard payload from ``.json`` or ``.json.gz``.
+PAYLOAD_COUNTRIES = ("us", "uk")
+
+
+def load_payload(path: Path | str, country: str = "us") -> dict:
+    """Read one country's dashboard payload from ``.json`` or ``.json.gz``.
 
-    Accepts both the per-country payload frozen under ``paper/snapshot`` and
-    the combined ``{"countries": {"us": ...}}`` shape of a dashboard release
-    asset, and returns the US payload either way.
+    Accepts both a per-country payload (frozen under ``paper/snapshot``, or
+    ``<run>/<country>/data.json`` from ``export-full-run``) and the combined
+    ``{"countries": {...}}`` shape of a dashboard release asset, and returns
+    that country's payload either way.
     """
+    if country not in PAYLOAD_COUNTRIES:
+        raise ValueError(f"country must be one of {PAYLOAD_COUNTRIES}: {country!r}")
     path = Path(path)
     raw = path.read_bytes()
     if raw[:2] == b"\x1f\x8b":
@@ -139,17 +145,22 @@ def load_us_payload(path: Path | str) -> dict:
         raise ValueError(f"{path}: payload is not a JSON object")
     if "countries" in payload:
         countries = payload["countries"]
-        if not isinstance(countries, dict) or "us" not in countries:
-            raise ValueError(f"{path}: combined payload has no 'us' country")
-        payload = countries["us"]
-    country = payload.get("country")
-    if country is not None and country != "us":
-        raise ValueError(f"{path}: payload is for country {country!r}, not 'us'")
+        if not isinstance(countries, dict) or country not in countries:
+            raise ValueError(f"{path}: combined payload has no {country!r} country")
+        payload = countries[country]
+    found = payload.get("country")
+    if found is not None and found != country:
+        raise ValueError(f"{path}: payload is for country {found!r}, not {country!r}")
     if "scenarioPredictions" not in payload or "modelStats" not in payload:
         raise ValueError(f"{path}: payload lacks scenarioPredictions or modelStats")
     return payload
 
 
+def load_us_payload(path: Path | str) -> dict:
+    """Read a US dashboard payload (see :func:`load_payload`)."""
+    return load_payload(path, "us")
+
+
 def file_sha256(path: Path | str) -> str:
     """The sha256 of a file's bytes, as lowercase hex."""
     digest = hashlib.sha256()

policybench/ground_truth.py

@@ -9,6 +9,7 @@
 
 from policybench.config import DEFAULT_COUNTRY, TAX_YEAR, get_programs
 from policybench.policyengine_runtime import (
+    get_uk_situation_simulation_class,
     get_us_situation_simulation_class,
     make_uk_transfer_microsimulation,
 )
@@ -480,6 +481,59 @@ def _calculate_ground_truth_uk(
     programs: list[str],
     year: int,
 ) -> pd.DataFrame:
+    """Calculate UK references from the facts each prompt states.
+
+    The scenarios come from the transfer microdata, but the reference uses
+    only what the prompt lists, as on the US path. Values the source record
+    holds and the prompt omits cannot move a reference.
+
+    Each scenario runs in its own simulation. PE-UK assigns the 2026 Universal
+    Credit health element by a seeded draw over the benefit units in a
+    simulation, so sharing one simulation would make a household's reference
+    depend on its position among the others.
+    """
+    Simulation = get_uk_situation_simulation_class()
+    period = str(year)
+    rows = []
+    for scenario in scenarios:
+        sim = Simulation(situation=scenario.to_pe_uk_situation())
+        for variable in expand_programs_for_scenario(programs, scenario):
+            pe_variable = _pe_variable_for_output(variable, "uk")
+            entity_key = sim.tax_benefit_system.variables[pe_variable].entity.key
+            if entity_key not in {"person", "benunit", "household"}:
+                raise ValueError(
+                    f"Unsupported UK entity '{entity_key}' for benchmark variable "
+                    f"'{variable}'."
+                )
+            rows.append(
+                {
+                    "scenario_id": scenario.id,
+                    "variable": variable,
+                    "value": float(
+                        np.asarray(
+                            sim.calculate(pe_variable, period), dtype=float
+                        ).sum()
+                    ),
+                    "impact_weight": None,
+                }
+            )
+    return pd.DataFrame(rows)
+
+
+def calculate_uk_transfer_microsimulation_values(
+    scenarios: list[Scenario],
+    programs: list[str] | None = None,
+    year: int = TAX_YEAR,
+) -> pd.DataFrame:
+    """Calculate UK outputs from the full transfer microdata records.
+
+    This was the scored UK reference before October 2026. It reads every
+    stored variable of each sampled household, including ones the prompt
+    omits, so it now serves as a cross-check: where it disagrees with the
+    scored reference, the source record holds a fact the prompt leaves out.
+    """
+    if programs is None:
+        programs = get_programs("uk")
     os.environ.setdefault("HDF5_USE_FILE_LOCKING", "FALSE")
     sim = make_uk_transfer_microsimulation(get_uk_dataset_path())
     period = str(year)

policybench/policyengine_runtime.py

@@ -74,19 +74,19 @@
     "runtime_dataset": "enhanced_cps_2025",
     "runtime_dataset_filename": "enhanced_cps_2025.h5",
     "runtime_dataset_repo": "PolicyEngine/policyengine-uk-data",
-    "runtime_dataset_pinned_commit": ("9514dfb7ec607897c9f7122a2e073b922c9fd8b6"),
+    "runtime_dataset_pinned_commit": ("6b1f80e0fdcd7ec149f347b4e0b2ae0081f0ff1b"),
     "runtime_dataset_pinned_url": (
         "https://raw.githubusercontent.com/PolicyEngine/"
-        "policyengine-uk-data/9514dfb7ec607897c9f7122a2e073b922c9fd8b6/"
+        "policyengine-uk-data/6b1f80e0fdcd7ec149f347b4e0b2ae0081f0ff1b/"
         "policyengine_uk_data/storage/enhanced_cps_2025.h5"
     ),
     "runtime_dataset_uri": (
         "policyengine_uk_data/storage/enhanced_cps_2025.h5 from the public "
         "PolicyEngine/policyengine-uk-data repository, pinned to commit "
-        "9514dfb7ec607897c9f7122a2e073b922c9fd8b6"
+        "6b1f80e0fdcd7ec149f347b4e0b2ae0081f0ff1b"
     ),
     "runtime_dataset_sha256": (
-        "199ebc61d29231b4799ad337a95393765b5fb5aede1834b93ff2acecceded866"
+        "c663daea31b6fb8300f5c4758ccdcd7a227835ad7de90655294392fb95543eec"
     ),
     "runtime_dataset_note": (
         "UK calibrated transfer dataset derived from benchmark-compatible "
@@ -647,6 +647,14 @@ def get_uk_single_year_dataset_class():
     return UKSingleYearDataset
 
 
+def get_uk_situation_simulation_class():
+    """Return the PE-UK situation Simulation class and record runtime metadata."""
+    policyengine_release_bundle("uk")
+    from policyengine_uk import Simulation
+
+    return Simulation
+
+
 def make_uk_transfer_microsimulation(dataset_path: str | Path):
     """Create a PE-UK Microsimulation for PolicyBench's public transfer data.