Repository navigation
UK source packages, wave 3: back the uk_local_geography profile (constituency + local authority) #134
Description
Lanes: packages under ledger-source-ingestor; any profile extension under ledger-target-profile-author
Depends on: conventions from #132 / #133 (GSS geography IDs, curated-extract lane, Stat-Xplore api_response pattern)
Context
policyengine_ledger/target_profiles/uk_local_geography.json declares 25 targets across five families (8 ons_population age bands, 5 dwp_universal_credit, 5 ons_housing, 4 hmrc income, 3 ons_income) at constituency + local_authority levels — with zero backing source packages. Of Ledger's 70 committed packages, all 4 UK ones are national firm/VAT. Meanwhile policyengine-uk-data feeds its local calibration from per-area CSVs under datasets/local_areas/{constituencies,local_authorities}/targets/ (built by targets/sources/local_age.py, local_income.py, local_uc.py, local_la_extras.py — these emit no registry Target()s; they are loss-matrix inputs).
Sub-national precedent exists: irs_soi/historic_table_2_state_*, census/pep_2024_state_age_sex, census/acs_s0101_state_2024 (state level), and per-CD Census API extractions.
Packages
| backing | package(s) | source mechanics |
|---|---|---|
ons_population age bands |
ons/census21_age_by_pcon24, ons/census21_age_by_lad |
Census 2021 / mid-year estimates by PCON24 and LAD; uk-data's local_age.py reads two per-area CSVs |
hmrc employment / self-employment income |
hmrc/spi_by_pcon24, hmrc/spi_by_lad |
SPI geography tables (collated 3.12–3.15a) — amounts and taxpayer counts |
earnings (profile hmrc count/amount complements) |
nomis/ashe_earnings_by_pcon24 |
NOMIS ASHE workbook uk-data commits (nomis_earning_jobs_data.xlsx) |
dwp_universal_credit households (+ by children) |
dwp/uc_households_by_pcon24, dwp/uc_households_by_lad |
Stat-Xplore extracts — archive the API response as the artifact per the existing api_response pattern |
ons_income equivalised income BHC/AHC + housing costs |
ons/small_area_income_msoa |
ONS small-area income estimates (MSOA), plus the aggregation lookup the profile's levels need — only source-published aggregation levels enter Ledger; any MSOA→PCON re-aggregation PolicyEngine performs is downstream build work, not a fact |
ons_housing tenure + private rent |
ons/tenure_by_lad, ons/private_rents_by_lad |
uk-data local_la_extras.py sources (PRMS rents, tenure) |
| LA council tax stock | voa/council_tax_stock_by_lad |
wave-2 VOA publication, LA cut |
Geography contract: GSS codes as geography_id (E14… PCON24, E06/E07/E08/E09… + devolved LAD codes), explicit geography_vintage (pcon_2024, lad_2023-style), geography_level: constituency / local_authority matching the profile's levels.
Row-count mechanics (verify on the first package)
650 PCON24 × band and 360 LAD × band tables must not become hand-enumerated per-row YAML (the VAT worked example enumerates 9 rows manually). Multiple committed extraction methods parse full tables ("delimited text full-row parse with selected source-row cells", "zip archive CSV member full-row parse", Census API unpivot) — confirm the row-spec form that pins geometry for full-table parses on the first constituency package, and document it in docs/agent-source-package-harness.md if the answer isn't already written down.
Profile work
uk_local_geography already declares selectors and bindings — the deliverable is that its selectors resolve against these new facts. Any selector/binding corrections discovered while backing it are a separate small PR in the profile lane (no values, sum-only).
Acceptance
- Every declared
uk_local_geographytarget resolves from a consumer artifact at both declared geography levels, every fact carrying full lineage (manifest sha256 + R2 + cell guards). - Coverage parity with uk-data's local CSV inputs: each per-area family uk-data feeds its constituency/LA loss matrices either has a Ledger package or an enumerated, signed exclusion in this issue — no silent drops.
- Deterministic checks + judge lanes green.
Out of scope
- Populace's local solve (already Ledger-profile-aware via
populace.build.uk.local_targets, populace#158). uk-data's local calibration remains as is until populace consumption switches over.
Sizing
~8–9 packages ≈ 3 PRs (ONS batch, HMRC/NOMIS batch, DWP/VOA batch) + possibly 1 profile-correction PR.
Activity
Add a single-sheet xlsx full-row lane and an msoa geography level (Po…
Add the batch-4 packages: LA income, tenure and rents (PolicyEngine#134)
Record the batch-4 families on the wave-3 checklist (PolicyEngine#134)
Re-measure the bundle expectation surface for batch 4 (PolicyEngine#134)
Add the batch-5 packages: per-area household counts and LA council ta…
Record the batch-5 families on the wave-3 checklist (PolicyEngine#134)
Re-measure the bundle expectation surface for batch 5 (PolicyEngine#134)
Pin the council tax levels and taxbase artifacts (PolicyEngine#134)
Add the council tax levels and taxbase packages (PolicyEngine#134)
Fold the local families into one inventory and reconcile every status (…
Re-measure the bundle expectation surface for batch 6 (PolicyEngine#134)
Refuse unreachable header rows and close the read-only workbook (Poli…
Add the PCON24 population package and its lane mechanics (#134)
Complete the constituency age family with Scotland and Northern Irela…
Add the UK local-authority age package (#134)
Document the whole-table package shape and open the wave-3 checklist (#…
Add the HMRC SPI by-area package (#134)
Add the DWP Stat-Xplore UC-by-area packages (#134)
Record the SPI and UC families on the wave-3 checklist (#134)
Keep the UC children facts out of the plain-total selector (#134)
Re-measure the bundle expectation surface for wave 3 (#134)
juaristi22 commentedon Aug 17, 2026
Closing via #173 (see its body): all wave-3 packages landed in #163, and the remaining scope here — backing the Chronicle-side
uk_local_geographyprofile — is moot under the #166 ruling (contracts live in the consumer; the local/firms contract migration is microcosm#708, and the profile surface retires under #172). #173 carries the migration's last outstanding facts.