Repository navigation
UK source packages, wave 2: national fiscal/benefit families + uk_national target profile #133
Description
Lanes: packages under ledger-source-ingestor; the profile PR under ledger-target-profile-author (judges: ledger-target-profile + ledger-boundary)
Depends on: #132 (parity fixture established); #131 for the slc_repayments package; Stat-Xplore extracts follow the existing api_response artifact pattern — no new policy needed
Context
policyengine-uk-data (pinned ref for this migration: ebf733c, 2026-07-26) compiles its national calibration registry from 24 source modules under policyengine_uk_data/targets/sources/. Wave 1 ports the three families production last calibrated against; this issue ports the remainder of the national registry, one publication table per package, and then declares the uk_national target profile over them.
Ruling on vintages (2026-08-05): every source-backed original vintage is eventually portable — porting a fact never activates it; Populace owns the active subset. Tranche 1 = current vintages below. Historical vintages (earlier EFO rounds, prior SPI editions, …) land later through a standing backfill lane; the vintage-suffixed package-ID convention (obr/efo_march_2026_*) guarantees no collisions. Uprated/aged/PolicyEngine-computed values never enter Ledger (docs/adr-ledger-facts-only.md).
Package inventory (uk-data module → package)
| uk-data module (lines / Target() ctors) | package(s) | notes |
|---|---|---|
obr.py (730 / 13, ~28 realized targets × FY2024–30) |
obr/efo_march_2026_receipts, obr/efo_march_2026_expenditure |
assertion: source_projection, period_type: fiscal_year. uk-data commits the EFO workbooks under storage/obr_efo/ because obr.uk 403s CI runners — R2 archival is the proper fix for the same problem; ingest from the archived workbooks. |
dwp.py (323 / 21) |
dwp/benefit_stats_feb_2026, dwp/pip_stats_jan_2026, dwp/benefit_cap_nov_2025, dwp/two_child_limit_apr_2025, dwp/uc_payment_distribution |
Values are hand-transcribed in uk-data with reference URLs — the resolve_pending.py first-print lane fits most of these. UC payment distribution comes via Stat-Xplore: capture the API response as the artifact (sheet_name: api_response precedent, 5 existing packages; Census-CD API precedent for the extraction_method wording). A free DWP Stat-Xplore key is needed only by whoever runs the extract. |
hmrc_spi.py (366 / 4) |
hmrc/spi_collated_2324 |
Same collated ODS populace's national build parses natively (hmrc_income.py); one package serves both eventually, but do not touch populace's HMRC stage in this issue. |
hmrc_cgt.py (268 / 4) |
hmrc/cgt_statistics_2024 |
Single-homing: populace's native UK_CGT_TARGET_SPECS (populace/build/uk_runtime/fiscal_targets.py) and uk-data's module both retire onto this one package (populace retirement happens in PolicyEngine/microcosm#622). |
hmrc_salary_sacrifice.py (129 / 3) |
hmrc/salary_sacrifice_tables_6_1_6_2 |
Publisher CSV already. |
slc_repayments.py (100 / 1) |
slc/loan_repayments (or extend the wave-1 SLC package) |
Academic-year periods — needs #131. |
voa_council_tax.py (300 / 4) |
voa/council_tax_stock_2025 |
National band counts; LA-level stock goes to wave 3. |
ons_households.py, ons_tenure.py, housing.py (275 / 5) |
ons/families_households_2024, ons/tenure_2024 |
Curated extracts; the ONS families & households datasets also lack stable URLs — same lane as SNPP. |
ons_savings.py (61 / 1) |
ons/savings_interest_haxv |
ONS time series (HAXV, UKEA). |
ons_public_sector_employment.py (51 / 1) |
ons/public_sector_employment_2026 |
New module since June. |
nts_vehicles.py (49 / 3) |
dft/nts_vehicle_ownership_2024 |
Publisher is DfT (National Travel Survey); new agency directory. |
scottish_government.py (37 / 1) |
scottish_government/budget_2026_27 |
|
mhclg_regional_land.py, ons_land_values.py, _land.py (169 / 4) |
ons/land_values, mhclg/regional_land |
Verify the actual publisher/table for each before packaging — _land.py is a shared helper and the land series have moved between ONS/MHCLG publications. |
Survey-derived aggregates (any APS/LCF-based table encountered) additionally name survey_instrument per the README provenance rules.
The uk_national profile
After the packages above land: one PR in the ledger-target-profile-author lane adding policyengine_ledger/target_profiles/uk_national.json —
- selectors referencing the raw facts above (deterministic check: selector references raw Ledger facts),
measurement+bindings.policyenginemirroring uk-data'stargets/compute/*semantics (entity, map_to, filters),- no values (
FORBIDDEN_VALUE_KEYSintarget_profiles/model.py), no runtime hooks, - counterfactual/cross-tab targets (two-child limit, UC family-type splits, benefit cap) get declarative binding payloads only — the metric providers live in populace, following the JCT
neutralize_variableprecedent in populace'sus/fiscal_target_references.json.
Acceptance
- Every package: manifest
source_url+ sha256 + R2 URI; cell-geometry guards;concept_evidence_urlper measure; deterministic checks green; judge lanes as headed. - The
uk_nationalprofile validates with no embedded values and resolves every selector against built facts. - A consumer artifact covering waves 1+2 resolves every national-registry row that uk-data compiles at
ebf733cfor the current calibration year, with any intentional differences (vintage updates, dropped rows) enumerated in the PR description — no silent gaps. (The mechanical compile-parity gate itself lands populace-side.)
Out of scope
- Local-geography families (wave 3). Historical-vintage backfill (standing lane, separate issues as needed). Populace consumption/retirement (Consume Ledger UK facts: uk_target_references, mapping module, and a compile-parity gate microcosm#622, Calibrate the UK national build from Ledger-backed targets microcosm#623). The legacy
db/etl_obr.py/db/etl_ons.pydicts — superseded by these packages; their cleanup is tracked in UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135.
Sizing
~13–14 packages, batched per agency ≈ 5–6 PRs + 1 profile PR.
Activity
Add ONS savings-interest and families-households source packages (Pol…
Add HMRC salary-sacrifice relief source package, tax year 2024-25 (Po…
Add HMRC CGT statistics Table 1 source package, first ods_used_range …
Add HMRC SPI income-band and DWP benefit-cap source packages (PolicyE…
Add DWP two-child limit source package; freeze wave-2 PR scope (Polic…
juaristi22 commentedon Aug 6, 2026
Status: what belongs to which PR, and what remains.
Landed — draft PR #139 (frozen for review: 9 packages, 912 facts)
obr-efo-receipts-march-2026obr-efo-expenditure-march-2026ons-savings-interest-incomeons-families-households-2025hmrc-salary-sacrifice-relief-2024hmrc-cgt-statistics-2025hmrc-spi-income-bands-2023dwp-benefit-cap-november-2025dwp-uc-two-child-limit-2025Wave-1 note for cross-readers: the population families (
ons-mye-*,ons-uk-population-projections-2024) are in PR #137 under #132.Remaining in this issue's scope — successor PRs after #139 merges
_land.pyhelper spans moved publications).slc-student-support-england-2025,slc-student-loan-repayments-*) — gated on Add academic_year period type to the source-package contract #131 / PR Add academic_year period type to the source-package contract #136 (academic_year).undercount_relativeinflation to the family-type counts, which stays populace-side.uk_nationaltarget profile — its ownledger-target-profile-author-lane PR once the packages it selects exist.Corrections recorded during execution (affecting this issue's original text)
resolve_pending.pyfirst-print lane in this repo — hand-transcribed families are packaged from their actual publication files.ods_used_range); earlier "parser gap" scoping mistook zero usage for zero capability.ons_tenureEngland totals are not portable as national facts (LA-grain workbook; uk-data code-sums England) — moved to the wave-3 LA package under UK source packages, wave 3: back the uk_local_geography profile (constituency + local authority) #134.status: gone— its runtime download silently drops 3 targets) and the NRS statistics-by-theme URLs (current: the mid-2025 estimates publication, which also revises mid-23/24).🤖 Generated with Claude Code
Address adversarial review: entity split, curated concepts, live SPI …
juaristi22 commentedon Aug 7, 2026
Continuation draft up: #141 (stacked on #139's head so the drafts share one aliases/bundle-constants lineage) — six packages, 270 facts:
dwp-benefit-statistics-february-2026,voa-council-tax-bands-2025,scotgov-council-tax-bands-2025,scotgov-scottish-budget-social-security-assistance-2026,dft-nts-vehicle-ownership-2024,ons-public-sector-employment-2026. Also addsdocs/pe-uk-source-checklist.md, the per-family uk-data→Ledger migration tracker (fixes the harness doc's dead link).Scoping correction to the remaining-items list above: the PIP daily-living split (1,283,000 / 1,608,000) is not in the Jan-2026 publication tables — those are clearance/journey statistics. The values match DWP FOI response FOI2025/24990 (caseload at end-Jan-2025); its host blocks automated fetches, so that package is parked on a manual download of the response PDF, then the
pdf_text_numberslane.Still open after #141: SLC (gated on #131/#136), Stat-Xplore families (key), PIP FOI (above), land values (publisher-table verification pending), SPP Review 2025 PDF bases, ISC census (#132 remainder),
uk_nationalprofile.🤖 Generated with Claude Code
Add PIP daily-living FOI source package from FOI2025/24990 (PolicyEng…
Add four Stat-Xplore UC source packages via the Open Data API (Policy…
Add land, salary-sacrifice headcount and ISC census packages (PolicyE…
Address adversarial review: NTS geography, withdrawn TCLP allocation,…
Add OBR EFO March 2026 source packages: receipts and expenditure (Pol…
Add ONS savings-interest and families-households source packages (Pol…
Add HMRC salary-sacrifice relief source package, tax year 2024-25 (Po…
47 remaining items
Address adversarial review: NTS geography, withdrawn TCLP allocation,…
Record R2 storage for the wave-2 remainder after credentialed publish…
Re-measure the full bundle-expectation surface on the merged base (Po…
Merge pull request #141 from juaristi22/uk-wave2-remaining-national
Point the checklist housing row at the opened PR (PolicyEngine#133)
Add housing base-fact packages: PIPR bulletin rents and EHS weekly ho…
Re-measure the bundle-expectation surface with the housing packages (P…
Point the checklist housing row at the opened PR (PolicyEngine#133)
Consolidate the source checklist into one full-inventory table (Polic…
Re-measure the bundle-expectation surface on the post-PolicyEngine#155 …
Carry EHS note 5 per-fact; record note-2 recoverability (PolicyEngine…
Merge pull request #159 from juaristi22/uk-wave2-housing-base-facts
Make dimension pins selectable; port MYE sex sheets; validate kinds (P…
juaristi22 commentedon Aug 17, 2026
Closing: the Chronicle-side scope is complete, and the remaining deliverable moved repos per the #166 ruling.
docs/pe-uk-source-checklist.md.uk_nationalprofile: re-homed. Adjudicate: do target profiles and measurement bindings belong in Chronicle? The boundary records disagree #166 ruled selection contracts live in the consumer, so the contract this issue scoped as aledger-target-profile-author-lane PR landed instead as microcosm#707, carrying the full 651-row registry accounting (609 mapped + 42 signed exclusions atebf733c). Consumption/compile-parity remains microcosm#622/#623.