Repository navigation
UK scorecard: populace_uk_2023 vs enhanced_frs_2024_25 vs admin benchmarks (2026) #731
Description
cc @juaristi22 — María, could you take a look at why we're off on some of these?
I ran a benchmark scorecard of populace_uk_2023 against enhanced_frs_2024_25 (both simulated for 2026 through policyengine.py, pe.uk.ensure_datasets → Simulation with the pinned UK model version) and compared to approximate admin benchmarks.
Scorecard (2026)
| Metric | populace_uk_2023 | enhanced_frs_2024_25 | Benchmark (~2026) |
|---|---|---|---|
| Population | 69.9m | 70.0m | ONS ~68.3–69m |
| Households | 29.5m | 31.5m | ONS ~29m |
| State pension | £130.8bn | £127.5bn | OBR ~£140–146bn |
| Income tax | £419bn | £312bn | OBR ~£330bn |
| Income taxpayers | 36.7m | 42.7m | HMRC ~37–40m |
| Universal credit | £39.9bn / 3.3m families | £81.3bn / 6.35m | ~£80–86bn / ~6.7m hh |
| Child benefit | £17.4bn | £17.7bn | ~£13.5bn |
| Pension credit | £6.8bn | £7.5bn | ~£6bn |
| Council tax | £60.1bn | £52.5bn | ~£50bn |
| Poverty (BHC) | 16.7% | 13.5% | HBAI ~17% |
| Top 1% share (net hh income) | 6.9% | 6.9% | — |
| Gini (net hh income) | 0.385 | 0.370 | ONS ~0.34–0.36 |
Populace does well on demographics, household counts, taxpayer counts, pension credit, and poverty. The two large misses are:
- Income tax +27% vs OBR (£419bn vs ~£330bn) — consistent with the +20.8% vs the SPI-anchored benchmark already diagnosed in UK_COVERAGE_PROGRESS.md as distributional.
- Universal credit at less than half of admin (£39.9bn / 3.3m families vs ~£80–86bn / ~6.7m households) — consistent with Certified UK artifact carries less than half of admin UC caseload (3.26m vs ~6.7m; £36.9bn vs £75.8bn) #701 (UC not bound in the certified 148-target surface).
Smaller ones worth a look: child benefit ~+30% (though enhanced FRS shares this), council tax ~+20%, state pension ~£10–15bn low (also shared), and Gini a touch high.
Reproduction code
"""Benchmark scorecard for a UK dataset via policyengine.py (year 2026).
Usage: python scorecard.py <dataset-name-or-hf-uri>
e.g. python scorecard.py populace_uk_2023
"""
import sys
import numpy as np
import policyengine as pe
from policyengine import Simulation
DATASET = sys.argv[1]
d = pe.uk.load_datasets(datasets=[DATASET], years=[2026], data_folder="./data")
dataset = list(d.values())[0]
PERSON_VARS = [
"age",
"state_pension",
"income_tax",
"national_insurance",
"employment_income",
"self_employment_income",
"dividend_income",
"pension_income",
"capital_gains",
]
BENUNIT_VARS = ["universal_credit", "child_benefit", "pension_credit"]
HH_VARS = ["household_net_income", "income_tax", "council_tax", "in_poverty_bhc"]
sim = Simulation(
dataset=dataset,
tax_benefit_model_version=pe.uk.model,
extra_variables={
"person": PERSON_VARS,
"benunit": BENUNIT_VARS,
"household": HH_VARS,
},
)
sim.ensure()
out = sim.output_dataset.data
def arr(ent, var):
return np.asarray(getattr(out, ent)[var], float)
pw = arr("person", "person_weight")
bw = arr("benunit", "benunit_weight")
hw = arr("household", "household_weight")
print(f"SCORECARD {DATASET} year 2026")
age = arr("person", "age")
print(f"population_m {pw.sum()/1e6:.2f}")
print(f"children_u16_m {pw[age < 16].sum()/1e6:.2f}")
print(f"age_65plus_m {pw[age >= 65].sum()/1e6:.2f}")
print(f"households_m {hw.sum()/1e6:.2f}")
for v in PERSON_VARS[1:]:
print(f"{v}_bn {(arr('person', v)*pw).sum()/1e9:.1f}")
it = arr("person", "income_tax")
print(f"taxpayers_m {pw[it > 0].sum()/1e6:.2f}")
for v in BENUNIT_VARS:
x = arr("benunit", v)
print(f"{v}_bn {(x*bw).sum()/1e9:.1f}")
print(f"{v}_families_m {bw[x > 0].sum()/1e6:.2f}")
hni = arr("household", "household_net_income")
print(f"council_tax_bn {(arr('household','council_tax')*hw).sum()/1e9:.1f}")
pov = arr("household", "in_poverty_bhc")
print(f"poverty_bhc_pct {100*(pov*hw).sum()/hw.sum():.1f}")
order = np.argsort(hni)
h, w = hni[order], hw[order]
cw = np.cumsum(w) / w.sum()
inc = h * w
print(f"top1_share_pct {100*inc[cw >= 0.99].sum()/inc.sum():.1f}")
print(f"top10_share_pct {100*inc[cw >= 0.90].sum()/inc.sum():.1f}")
p = np.cumsum(w) / w.sum()
L = np.cumsum(np.clip(inc, 0, None)) / np.clip(inc, 0, None).sum()
print(f"gini {1 - 2*np.trapezoid(L, p):.3f}")Notes: national_insurance here is the personal-side variable only; benchmarks are approximate (OBR EFO / HMRC / DWP / ONS, nearest year) and meant for orders of magnitude, not precise scoring. All figures are weighted aggregates only — no microdata is included.
Metadata
Metadata
Assignees
Labels
No labels
Activity
juaristi22 commentedon Aug 20, 2026
Thanks @vahid-ahmadi — dug into this. TL;DR: the pattern is exactly what the June release's calibration surface predicts. The certified artifact was calibrated to 149 targets: 142 ONS region×age population, 6 SLC student-support, and 1 OBR private-school-students (chronicle#132); the calibration code never reached a pushed branch and the national build is gate-only by design (#623). Zero fiscal targets bound — so demographics, household counts, taxpayer counts, and poverty land well, and every fiscal aggregate free-floats. The eFRS column is the mirror image: it binds OBR receipts + HMRC income bands, so it nails income tax while overshooting households (31.5m vs ONS ~29m) and distorting poverty low (13.5% vs HBAI ~17%). Each dataset is accurate exactly where its surface binds.
Row by row
obr.income_taxdeclared in the 186-target contract. Direct HMRC banded route is adjudicated-fenced (all 208 facts banded by SPI Total Income, which FRS can't materialize —hmrc_income_replay_report.json,calibration_performed: false)would_claim_child_benefitis a dataset take-up draw defaulting to True; both datasets simulate ~full entitlement while the ~£13.5bn benchmark is net of HICBC opt-outs (~700k families) + non-take-upobr.council_taxdeclaredobr.state_pensiondeclaredHow the in-flight stack closes this
The architecture the June release lacked — facts → declaration → activation → fail-closed solve — is landing as one chain, UC first:
uc_benefit_unitsfacts).dwp.uc.householdsdeclaration + contract re-pin to uk-data 1.56.16. The contract (uk_national_targets.json, 186 declarations) already covers every missed family above — declarations resolve from chronicle facts only; the uk-data pin is parity accounting, not a value source.allowed_value_operationsis["identity"]today).Then a gated build + certification, at which point this scorecard is a ready-made acceptance test — worth re-running the script as-is against the first calibrated candidate. The bar on UC is beatable: eFRS at 1.56.14 sits at 6.40m/£77.6bn vs admin ~6.7m/£75.8bn, and populace's per-unit entitlement already matches admin while keeping the demographic/household/poverty fidelity eFRS gives up.
Evaluate a size candidate with one command: size_evaluation library, …
Evaluate a size candidate with one command: size_evaluation library, …
Evaluate a size candidate with one command: size_evaluation library, …
Evidence for the microcosm#968 BHC poverty diagnosis
Evidence for the microcosm#968 BHC poverty diagnosis