Skip to content

UK source packages, wave 2: national fiscal/benefit families + uk_national target profile #133

Closed

Description

@juaristi22

Lanes: packages under ledger-source-ingestor; the profile PR under ledger-target-profile-author (judges: ledger-target-profile + ledger-boundary)
Depends on: #132 (parity fixture established); #131 for the slc_repayments package; Stat-Xplore extracts follow the existing api_response artifact pattern — no new policy needed

Context

policyengine-uk-data (pinned ref for this migration: ebf733c, 2026-07-26) compiles its national calibration registry from 24 source modules under policyengine_uk_data/targets/sources/. Wave 1 ports the three families production last calibrated against; this issue ports the remainder of the national registry, one publication table per package, and then declares the uk_national target profile over them.

Ruling on vintages (2026-08-05): every source-backed original vintage is eventually portable — porting a fact never activates it; Populace owns the active subset. Tranche 1 = current vintages below. Historical vintages (earlier EFO rounds, prior SPI editions, …) land later through a standing backfill lane; the vintage-suffixed package-ID convention (obr/efo_march_2026_*) guarantees no collisions. Uprated/aged/PolicyEngine-computed values never enter Ledger (docs/adr-ledger-facts-only.md).

Package inventory (uk-data module → package)

uk-data module (lines / Target() ctors) package(s) notes
obr.py (730 / 13, ~28 realized targets × FY2024–30) obr/efo_march_2026_receipts, obr/efo_march_2026_expenditure assertion: source_projection, period_type: fiscal_year. uk-data commits the EFO workbooks under storage/obr_efo/ because obr.uk 403s CI runners — R2 archival is the proper fix for the same problem; ingest from the archived workbooks.
dwp.py (323 / 21) dwp/benefit_stats_feb_2026, dwp/pip_stats_jan_2026, dwp/benefit_cap_nov_2025, dwp/two_child_limit_apr_2025, dwp/uc_payment_distribution Values are hand-transcribed in uk-data with reference URLs — the resolve_pending.py first-print lane fits most of these. UC payment distribution comes via Stat-Xplore: capture the API response as the artifact (sheet_name: api_response precedent, 5 existing packages; Census-CD API precedent for the extraction_method wording). A free DWP Stat-Xplore key is needed only by whoever runs the extract.
hmrc_spi.py (366 / 4) hmrc/spi_collated_2324 Same collated ODS populace's national build parses natively (hmrc_income.py); one package serves both eventually, but do not touch populace's HMRC stage in this issue.
hmrc_cgt.py (268 / 4) hmrc/cgt_statistics_2024 Single-homing: populace's native UK_CGT_TARGET_SPECS (populace/build/uk_runtime/fiscal_targets.py) and uk-data's module both retire onto this one package (populace retirement happens in PolicyEngine/microcosm#622).
hmrc_salary_sacrifice.py (129 / 3) hmrc/salary_sacrifice_tables_6_1_6_2 Publisher CSV already.
slc_repayments.py (100 / 1) slc/loan_repayments (or extend the wave-1 SLC package) Academic-year periods — needs #131.
voa_council_tax.py (300 / 4) voa/council_tax_stock_2025 National band counts; LA-level stock goes to wave 3.
ons_households.py, ons_tenure.py, housing.py (275 / 5) ons/families_households_2024, ons/tenure_2024 Curated extracts; the ONS families & households datasets also lack stable URLs — same lane as SNPP.
ons_savings.py (61 / 1) ons/savings_interest_haxv ONS time series (HAXV, UKEA).
ons_public_sector_employment.py (51 / 1) ons/public_sector_employment_2026 New module since June.
nts_vehicles.py (49 / 3) dft/nts_vehicle_ownership_2024 Publisher is DfT (National Travel Survey); new agency directory.
scottish_government.py (37 / 1) scottish_government/budget_2026_27
mhclg_regional_land.py, ons_land_values.py, _land.py (169 / 4) ons/land_values, mhclg/regional_land Verify the actual publisher/table for each before packaging — _land.py is a shared helper and the land series have moved between ONS/MHCLG publications.

Survey-derived aggregates (any APS/LCF-based table encountered) additionally name survey_instrument per the README provenance rules.

The uk_national profile

After the packages above land: one PR in the ledger-target-profile-author lane adding policyengine_ledger/target_profiles/uk_national.json —

  • selectors referencing the raw facts above (deterministic check: selector references raw Ledger facts),
  • measurement + bindings.policyengine mirroring uk-data's targets/compute/* semantics (entity, map_to, filters),
  • no values (FORBIDDEN_VALUE_KEYS in target_profiles/model.py), no runtime hooks,
  • counterfactual/cross-tab targets (two-child limit, UC family-type splits, benefit cap) get declarative binding payloads only — the metric providers live in populace, following the JCT neutralize_variable precedent in populace's us/fiscal_target_references.json.

Acceptance

  • Every package: manifest source_url + sha256 + R2 URI; cell-geometry guards; concept_evidence_url per measure; deterministic checks green; judge lanes as headed.
  • The uk_national profile validates with no embedded values and resolves every selector against built facts.
  • A consumer artifact covering waves 1+2 resolves every national-registry row that uk-data compiles at ebf733c for the current calibration year, with any intentional differences (vintage updates, dropped rows) enumerated in the PR description — no silent gaps. (The mechanical compile-parity gate itself lands populace-side.)

Out of scope

Sizing

~13–14 packages, batched per agency ≈ 5–6 PRs + 1 profile PR.

Activity

  1. juaristi22 commented on Aug 6, 2026

    @juaristi22
    CollaboratorAuthor

    Status: what belongs to which PR, and what remains.

    Landed — draft PR #139 (frozen for review: 9 packages, 912 facts)

    uk-data family package facts
    OBR EFO receipts (income tax, NIC classes, cash-basis NI/VAT/fuel/CGT/SDLT) obr-efo-receipts-march-2026 63
    OBR EFO expenditure (council tax ×5, welfare ×13 incl. UC in/outside cap, TV licence) obr-efo-expenditure-march-2026 133
    ONS savings interest (HAXV annuals; flat-fill excluded as computed) ons-savings-interest-income 8
    ONS household types (Table 7, 2025 edition, LFS survey_aggregate) ons-families-households-2025 80
    HMRC salary-sacrifice relief (TY2024-25; uk-data's asset is dead + its ×1.03 series excluded) hmrc-salary-sacrifice-relief-2024 6
    HMRC CGT Table 1 (full 37-year history; uk-data's 378k/£65.9bn = publisher Total columns) hmrc-cgt-statistics-2025 333
    HMRC SPI Tables 3.6/3.7 (13 bands × 7 income types) hmrc-spi-income-bands-2023 182
    DWP benefit cap Table 4 (110,637 + 13 bands; £320.9m annualization excluded as computed) dwp-benefit-cap-november-2025 15
    DWP two-child limit (tables 01/03x/04x, all four status rows) dwp-uc-two-child-limit-2025 92

    Wave-1 note for cross-readers: the population families (ons-mye-*, ons-uk-population-projections-2024) are in PR #137 under #132.

    Remaining in this issue's scope — successor PRs after #139 merges

    1. PIP Jan-2026 + DWP benefit statistics Feb-2026 — publication tables pinned; next data batch.
    2. VOA council tax stock 2025 + gov.scot CT datasets; DfT NTS vehicles; Scottish budget; land values (verify the land publishers first — uk-data's _land.py helper spans moved publications).
    3. SLC (slc-student-support-england-2025, slc-student-loan-repayments-*) — gated on Add academic_year period type to the source-package contract #131 / PR Add academic_year period type to the source-package contract #136 (academic_year).
    4. Stat-Xplore extracts (UC payment distribution, UC households by children, Scotland UC child-under-1) — parked until someone holds a free Stat-Xplore API key; uk-data also applies a computed undercount_relative inflation to the family-type counts, which stays populace-side.
    5. SPP Review 2025 PDF base values (salary-sacrifice £24bn contributions, NI-relief bases) — small package; the growth-uprated series remain excluded per the facts-only ADR.
    6. uk_national target profile — its own ledger-target-profile-author-lane PR once the packages it selects exist.

    Corrections recorded during execution (affecting this issue's original text)

    • There is no resolve_pending.py first-print lane in this repo — hand-transcribed families are packaged from their actual publication files.
    • The ODS parser already existed (ods_used_range); earlier "parser gap" scoping mistook zero usage for zero capability.
    • ons_tenure England totals are not portable as national facts (LA-grain workbook; uk-data code-sums England) — moved to the wave-3 LA package under UK source packages, wave 3: back the uk_local_geography profile (constituency + local authority) #134.
    • uk-data link-rot found in passing: salary-sacrifice asset (status: gone — its runtime download silently drops 3 targets) and the NRS statistics-by-theme URLs (current: the mid-2025 estimates publication, which also revises mid-23/24).

    🤖 Generated with Claude Code

  2. juaristi22 commented on Aug 7, 2026

    @juaristi22
    CollaboratorAuthor

    Continuation draft up: #141 (stacked on #139's head so the drafts share one aliases/bundle-constants lineage) — six packages, 270 facts: dwp-benefit-statistics-february-2026, voa-council-tax-bands-2025, scotgov-council-tax-bands-2025, scotgov-scottish-budget-social-security-assistance-2026, dft-nts-vehicle-ownership-2024, ons-public-sector-employment-2026. Also adds docs/pe-uk-source-checklist.md, the per-family uk-data→Ledger migration tracker (fixes the harness doc's dead link).

    Scoping correction to the remaining-items list above: the PIP daily-living split (1,283,000 / 1,608,000) is not in the Jan-2026 publication tables — those are clearance/journey statistics. The values match DWP FOI response FOI2025/24990 (caseload at end-Jan-2025); its host blocks automated fetches, so that package is parked on a manual download of the response PDF, then the pdf_text_numbers lane.

    Still open after #141: SLC (gated on #131/#136), Stat-Xplore families (key), PIP FOI (above), land values (publisher-table verification pending), SPP Review 2025 PDF bases, ISC census (#132 remainder), uk_national profile.

    🤖 Generated with Claude Code

  3. 47 remaining items

  4. juaristi22 commented on Aug 17, 2026

    @juaristi22
    CollaboratorAuthor

    Closing: the Chronicle-side scope is complete, and the remaining deliverable moved repos per the #166 ruling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions