Repository navigation
UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog) #495
Description
Max greenlit scoping the "bigger UK dataset" arm 2026-07-22. This epic is the UK analog of the US dense/local arm: one national rowwise artifact whose rows carry finest-available geography and whose weights are calibrated against a constituency/local-authority target surface — built by cloning the compact FRS-based pool, because the UK has no ACS-scale microdata to substitute for density. It subsumes the publication goal of #146 (rowwise H5 + long weights), sits under the #144 migration umbrella, and is the sibling of #355 (the pool should get a smaller national default at the same time as this arm gets bigger).
Everything below is designed against the machinery that already exists (uk_runtime/rowwise_geography.py, rowwise_dataset.py, geography_ladder.py, local_targets.py, local_solver.py, local_runner.py, tools/build_uk_rowwise_dataset.py); the epic is mostly wiring, fences, and targets, not new solvers.
Base artifact and pipeline order (fence-determined, not a preference)
The clone input is the national staging H5 — the output of the national orchestration seam after the 145/0 input-coverage gate and the adjudicated HMRC/SPI replay — not the raw certified compact:
certified compact (f17306cc…) |
national staging (829e843f…) |
|
|---|---|---|
| households | 535,080 (200,000 zero-weight dead SPI) | 535,080 (rebuilt SPI channel, positive mass) |
| persons | 1,157,100 | 1,157,739 (724,960 FRS + 432,779 SPI) |
| benunits | 618,980 | 618,884 |
| household mass | 28,840,551.182 | 28,840,551.182 (conserved; 1 mass-change record) |
| weight kind | (unlabeled) | importance + mass log in H5 attrs |
| bytes | 1,315,880,118 | 1,532,379,785 (local_untracked) |
(Compact figures verified directly against the cached certified H5; staging figures from the committed national_staging_build_record.json.)
The ordering is forced by the fences in hmrc_restoration.py: the HMRC replay byte-binds its input to the certified candidate (_assert_dataset_matches_certified_candidate checks the source-file fingerprint captured at SHA-256 verification), so cloning can never run upstream of the seam, and cloning the certified compact directly would clone 200k dead SPI households and skip the 145/0 restoration. Pipeline:
certified compact (byte-verified)
→ national seam (retained FRS leaves → HMRC/SPI replay, IMPORTANCE weights, mass log)
→ staging H5 ← the 145/0 gate passes here
→ rowwise clone + geography ladder assignment (this epic)
→ one uniform local+national calibration solve (this epic)
→ rowwise release, private HF only (this epic)
Clone count K: rows, memory, support
The pool is already a 10× clone: household_id = tier·10⁸ + base with exactly 53,508 unique base ids × 10 tiers (verified on the certified H5; #355 documents the construction). eFRS-grain source lineage is therefore recoverable as household_id mod 10⁸, and the rowwise build must carry it — per-area "distinct source households" diagnostics computed at pool grain would overstate real support 10×.
K below means the additional rowwise multiplier on top of that 10×. Clones add geographic resolution and solver degrees of freedom only — zero new information — so K is sized by per-area support, not ambition:
| K | households | persons | H5 (≈, linear) | mean rows/constituency (650) | smallest constituency (≈) |
|---|---|---|---|---|---|
| 1 | 535,080 | 1,157,739 | ~1.5 GB | 823 | ~240 |
| 2 | 1,070,160 | 2,315,478 | ~3.1 GB | 1,646 | ~480 |
| 4 | 2,140,320 | 4,630,956 | ~6.1 GB | 3,293 | ~960 |
| 10 | 5,350,800 | 11,577,390 | ~15 GB | 8,232 | ~2,400 |
The smallest-constituency column is the population-proportional expectation for a Na h-Eileanan an Iar-scale area and is approximate; the dry-run tool (increment 2) computes the exact expected support per constituency from the ladder/crosswalk populations and the pool's region mix. For calibration comparison: the US dense arm supports 436 CDs on a 337,704-record pool ≈ 775 records/CD (#355, #449) — K=1 is already at US-dense per-area scale, because the 10× pool cloning did the density work up front.
Proposal: pilot at K=1 (geography assignment only, no row growth), first published candidate at K=2–4, chosen from the dry-run support/ESS diagnostics. K>4 requires a measured justification. avoid_constituency_collisions stays on (no source household repeated in the same constituency across rowwise clones).
Geography assignment: the OA ladder is the operator
Two mechanisms exist. The crosswalk path (#142/#143) samples an OA within (country, region) ∝ population and covers all four countries. The OA ladder (#349 ratified, #354 merged) samples a 2024 Westminster constituency within the region ∝ census household counts, then an OA within the constituency ∝ population, carries sha-pinned best-fit lookups and per-layer vintages (vintage_policy: error), and ships a release-blocking gate — but is England & Wales only today (575 of 650 constituencies).
The epic declares the ladder as the assignment operator (it is the ratified US-block-ladder parallel with vintage discipline). Work items:
- extend the ladder artifact to Scotland (2022 census OAs, NRS sources) and Northern Ireland (Data Zones 2021, NISRA) —
geography_sources.pyalready pins those source URLs for the crosswalk path; they need per-layer vintage pinning inoa_ladder_sources.py; - until S/NI land, a full-UK rowwise build through the ladder fails loudly on the missing regions (already the ladder's contract — no silent partial coverage);
- rewire
tools/build_uk_rowwise_dataset.pyfrom the crosswalk sampler toassign_uk_geography_ladder+uk_geography_ladder_gate, keeping the crosswalk as the coverage-check path.
Local target surface: census first, then bind
local_targets.py computes today, at household grain:
- constituency: HMRC employment/self-employment amount+count (SPI frame via
income_tax > 0), age bands (8 × 10y), UC households, UC households by child count (0/1/2/3+); - local authority: the HMRC/age/UC families plus ONS equivalised net income BHC/AHC, housing costs, tenure (4 classes), private rent.
That mirrors the deprecated uk-data constituency surface. What official UK local data offers beyond it (ONS mid-year age structure by PCON24/LAD; HMRC Personal Incomes by constituency — taxpayer counts and employment/self-employment/pension totals; DWP Stat-Xplore UC/Pension Credit/Housing Benefit by constituency and LA; Census 2021/2022 tenure and composition; ONS PIPR rents by LA; ONS small-area income at MSOA) needs a census artifact, not prose: increment 1 ships a tool that enumerates every in-code metric family and every documented official family with area type, source, vintage, and status (bound_in_code / documented_unpinned / needs_adjudication), checked in and tested, so binding work is enumerable and claimable per family.
Two standing fences on targets:
- the 208 banded HMRC facts stay fenced (see below) — the constituency HMRC family is a different, unbanded surface and gets its own admissibility adjudication before it binds;
- the
income_tax > 0SPI-frame membership in the HMRC metrics is a modeled proxy frame; the census must record that honestly and the binding adjudication must accept or replace it explicitly.
The local solve: one uniform operator, declared bounds (doctrine)
Per the #492 lane and #493: no per-target calibration knobs. The solve is one operator over the whole surface — national families (149 targets in the published release, plus the natively declared CGT facts) and all area×metric rows together — with globally declared loss shape, cap, and weight-ratio bounds. A miss is local-support work (rows/clones/ladder) or target work (fix or fence the target), never a knob.
- Binding: the rowwise artifact solves one weight vector of length 535,080·K where each row supports only its assigned constituency's rows (the US Build-N CD shape). The stacked all-areas×all-households machinery (
build_stacked_local_matrix, Publish a Populace UK local variant with rowwise H5 and long weights #146 long weights) remains the research harness and can express the rowwise solve as its block-diagonal special case; the released artifact is the rowwise H5 with its solvedhousehold_weightcolumn. - Declared bounds to adjudicate once:
solve_stacked_local_weightscurrently defaultsmax_weight_ratio=100.0while the US dense design bound is 5× (max_weight_ratio anchors differently per arm: 5x vs design (dense) but ~25x effective (sparse refit re-anchors) — declare one stretch contract #493 wants one declared stretch contract). The epic picks the UK local bound explicitly and records it in the release manifest. - Diagnostics as contract: the Loss shape: flat cap at 1.0 makes past-cap rows free dumping grounds (17 rows pushed out in Build N); bounded-tail experiment + past-cap census diagnostic #492 past-cap census (init-past / final-past / escaped / pushed-out) ports to the local solve as a standard release artifact, alongside per-area support (min/median rows, ESS, distinct eFRS-grain sources) and the per-family fit table.
HMRC IMPORTANCE-weight fences under cloning
Read against hmrc_restoration.py, cloning survives the fences honestly if and only if the metadata rides along:
- Replay-before-clone is structural — the replay's byte-binding to the certified candidate means the clone consumes the seam's output; the fence cannot be reordered away.
- Mass conservation is exact under cloning: weights divide by K, every row duplicates, so total mass and every column's effective-mass share are invariant by construction. The 145/0 gate still re-runs on the rowwise output (fail-closed, never assumed), and the SPI channel's reviewed 50% mass share is re-checked post-solve as a declared bound.
- Live gap (increment 2 fixes):
rowwise_dataset.pyreads and writes only the four entity tables — it drops the staging H5'spopulace_household_weight_kind(importance) andpopulace_mass_log_jsonattrs. Cloning the staging H5 through today's tool silently launders IMPORTANCE weights into unlabeled weights. The clone step must carry both attrs, refuse an input whose weight kind it does not recognize, and assert exact mass conservation against the input. - No calibration against the 208 banded facts, before or after cloning: complete FRS Total Income band assignment remains unavailable, so those facts remain reviewed exclusions with canonical fences in every rowwise release exactly as in the national seam. Nothing in this epic touches
IMPORTANCE-weight replay internals, the reviewed release parameters, or the source contracts. - The local solve transitions weight kind explicitly (importance → locally calibrated) as a recorded step in the mass log, never implicitly.
Release contract for the rowwise artifact
The 145/0 contract covers input columns and carries over unchanged (the gate re-runs on the rowwise output). The rowwise artifact additionally promises, in its manifest:
- identity chain: staging H5 sha → certified candidate sha → source-stage shas (donor/ODS identities by reference to the committed replay artifacts, never row-level data);
- geography: ladder artifact sha + per-layer vintages, 650/650 (or the declared E&W 575 subset) constituencies assigned, zero missing-geography rows, ladder gate pass;
- weights: exact mass conservation vs staging, declared weight-kind chain (importance → locally calibrated), carried mass log, declared stretch bound;
- solve: uniform-operator config (loss shape, cap, ratio bound, epochs, seed), per-family fit, past-cap census, per-area support with eFRS-grain distinct-source counts;
- role:
dataset_role: non_default_local_areaper Release contract needs dataset-role classes: national-default checks reject valid non-default local-area artifacts #398 — never thelatest.jsonpointer (--no-latest), immutable tag, the compact (and eventually the UK small national default: L0-selected ~50k-household file from the 535k pool #355 small default) stays the national default. Until Release contract needs dataset-role classes: national-default checks reject valid non-default local-area artifacts #398's role-class validator lands, publication uses the Build-L routed path (all applicable gates recorded).
Publish mechanics (licensing)
FRS-derived microdata must never reach a public repo. The rowwise H5 and every sidecar that carries household-grain rows publish exclusively to the private HF dataset repo policyengine/populace-uk-private (the certified compact's home), via populace-publish-release --repo-id policyengine/populace-uk-private --no-latest, tagged populace-uk-2023-rowwise-<git>-<ts>. Aggregate diagnostics (manifest, fit tables, support summaries, past-cap census) are committable to the repo exactly like the HMRC replay artifacts — no row-level donor or FRS data, no local paths.
Increment ladder
- Local-target census tool —
tools/census_uk_local_targets.py+ checked-in census JSON + tests. Pure observability. (claimed, this lane) - Rowwise fence carriage + dry-run — carry weight-kind/mass-log attrs through clone read/write, refuse unrecognized weight kinds, assert exact mass conservation in the driver manifest, and add
--dry-run: the full clone plan (rows/bytes at K, exact expected per-constituency support from crosswalk populations × the pool's region mix, id-multiplier and lineage checks) with no H5 written. (claimed, this lane) - Ladder S/NI extension — pin NRS/NISRA sources with per-layer vintages; full-UK ladder artifact.
- Solve doctrine hardening — single-operator config surface, declared stretch bound (max_weight_ratio anchors differently per arm: 5x vs design (dense) but ~25x effective (sparse refit re-anchors) — declare one stretch contract #493 alignment), past-cap census port (Loss shape: flat cap at 1.0 makes past-cap rows free dumping grounds (17 rows pushed out in Build N); bounded-tail experiment + past-cap census diagnostic #492), structural refusal of per-target knobs.
- K=1 pilot build on the real staging H5 (licensed machine): geography + diagnostics, no solve; adjudicate K from the measured support.
- First calibrated rowwise candidate (K per adjudication): bound target families only, full contract above, cross-family review, private publish.
- Loader/consumer wiring (policyengine.py) — out of scope here; coordinate on the campaign index.
Refs: #144, #146, #355, #349, #354 (ladder), #420 (145/0 contract), #398 (dataset roles), #492/#493 (solve doctrine), #275 (one national dataset, filter by geography), #142/#143 (crosswalk + driver), UK_COVERAGE_PROGRESS.md (fence history).
Activity
MaxGhenis commentedon Jul 22, 2026
Claiming this epic (UK-dense scoping lane, greenlit 2026-07-22). Starting with the two unambiguous increments now, one PR each:
tools/census_uk_local_targets.py+ checked-in census artifact + tests (increment 1).--dry-runclone plan with expected per-constituency support (increment 2).Increments 3–6 (ladder S/NI, solve doctrine hardening, K=1 pilot, first calibrated candidate) stay open for claiming after design review of this issue. Coordination via comments here; campaign index cross-post on #449.
Harden the census per cross-family review: fail-closed matching, orde…
UK local-target census: drift-gated inventory of the local calibratio…
MaxGhenis commentedon Jul 22, 2026
Increment 1 merged: #498 (local-target census). The census landed with the cross-family review hardening: fail-closed exact/prefix family matching, per-area ordered-metric drift pinning, canonical-JSON currency comparison, register isolation, and six reviewed binding fences — the review pass minted three new ones (ons_bhc_ahc_noncomparable:
ons/equiv_housing_costshas no declared supplier; uc_unit_vs_household_grain: runtime benefit-unit counts vs DWP claim-household statistics; population_universe_private_households: FRS private households vs ONS usual residents). Those three are now standing inputs to the binding adjudications in increments 4/6.Increment 2 (rowwise fence carriage + dry-run) is implemented and in final verification; PR to follow.
MaxGhenis commentedon Jul 22, 2026
Increment 2 PR up: #501 — rowwise fence carriage (weight-kind/mass-log through the clone, closing the IMPORTANCE-laundering gap named in the epic), exact mass-conservation record with the declared 1e-9 bound,
--source-lineage-modulus(the verified 10^8 pool-tier structure), and--dry-runwith the sampler's exact expected per-area support. Sol cross-family review in progress; merge after adjudication + green CI.Rowwise clone: carry weight-kind/mass-log fences, exact mass conserva…
MaxGhenis commentedon Jul 22, 2026
Increment 2 merged: #501 (rowwise fence carriage + dry-run). Landed after a substantive cross-family review round — all five sol blockers confirmed and fixed, three verified against the pinned staging artifact. Design facts the epic should absorb (issue body remains authoritative for everything else):
source_household_idwith 335,080 distinct FRS-channel parents plushousehold_support_channel. Pool-grain lineage is therefore a separate derived layer (pool_source_household_id,--source-lineage-modulus 100000000), and on the staging pool it resolves 33,508 FRS bases — rebuilt-SPI households sit outside the tier scheme, so pool-lineage diagnostics are per-channel. The increment-5 pilot should adjudicate per-channel lineage semantics on the real artifact.realized_support, not the analytic block.Increments 3–6 (ladder S/NI extension, solve doctrine hardening, K=1 pilot on the real staging H5, first calibrated rowwise candidate) remain open for claiming.
MaxGhenis commentedon Jul 22, 2026
Continuing the lane: claiming increments 3 (ladder S/NI extension), 4 (solve doctrine hardening), and 5 (K pilot on the real staging H5). Order of attack: 5's measurement half first (dry-run K sweep on the real staging artifact with the merged #501 tooling — realized support per constituency/LA, lineage layers, weight chain — results posted here for the K adjudication), then the increment-4 doctrine PR, then increment 3.
MaxGhenis commentedon Jul 22, 2026
Increment 5a: K-sweep pilot on the real staging artifact
Ran against the byte-verified staging H5 (sha
829e843f…, re-hashed before use) with the #501 tooling; crosswalk built fresh from the official sources. Dry-run plans at K=1/2/4 plus a full K=1 build.Realized per-area support (real sampler at seed 42, zeros included)
650/650 constituencies and 361 LAs covered at every K; zero missing-geography rows. Collision-free vs realized minima differ by <5% at these K (groups have far more constituencies than clones), validating both blocks. The thin tail is exactly the expected protected/island class (S14000027 = Na h-Eileanan an Iar-scale, S14000051 Orkney-scale) plus the micro-LAs: Isles of Scilly 18 rows and City of London 32 at K=1 — LA-grain binding needs either K≥4 (Scilly 66) or an explicit micro-LA support adjudication.
Full K=1 build (the pilot artifact, local-untracked)
1.54 GB rowwise H5, sha
50c674f9…. The #501 fence chain held end-to-end on the real artifact: weight kind importance carried, 2 mass-log records (reviewed SPI allocation + clone record), conservation abs delta exactly 0.0, and the output loads underload_uk_national_dataset.Lineage measurements (design fact for increment 6)
Pool lineage (
mod 10⁸): 33,508 distinct pool sources; by channel: frs 33,508, spi 33,506 — the rebuilt SPI channel's ids resolve into the FRS base space because the seam samples its replacement households from the FRS base per stratum (spi_support._sample_replacement_household_ids, read this session). So pool lineage = eFRS pool-slot ancestry on both channels, which is stronger than the #501 caution suggested. Immediate layer: 335,080 distinct, as measured before. Pool copies per source: min 10 / median 16 / max 20.K adjudication proposal
Objections welcome here; increment 6 will treat K=2 constituency-grain as the working default absent adjudication otherwise.
MaxGhenis commentedon Jul 22, 2026
Increment 3 scoping: ladder Scotland/NI extension — the gap register
Measured against the pinned sources this session, the full-UK ladder is closer than the epic assumed:
Already solved UK-wide (no new work):
geography_sources.pyalready pins OA2022 → DZ2022 → IZ2022, OA → council/LAU, OA → PCON24, and OA population (the crosswalk path builds them today).The four genuinely missing sources (the whole remaining work):
MS-E01households table (Census 2021 person and household estimates for Data Zones) — publication verified; exact file + sha to pin.Mechanics:
join_uk_oa_ladder_layers/assemble_uk_oa_ladderare country-agnostic on their inputs; the extension is (a) a per-country layer-bundle assembly that concatenates E&W + S + NI OA frames with per-country, per-layer vintages in the artifact metadata (vintage_policy: errorextended to record e.g.ward: {ew: 2022, s: <vintage>, ni: DEA2014}), (b) the four loaders above with sha-pinned identities, and (c) the ladder gate's London-mass floor gaining S/NI analogues (Central Belt / Belfast mass floors). Until all four land, the ladder keeps refusing S/NI regions by name — no silent partial coverage.Remains claimed by this lane; if it hands off, this register is the complete work list.
MaxGhenis commentedon Jul 22, 2026
Increment 4 PR up: #503 — solve doctrine (uniform operator, declared cap 10.0 / stretch 100.0 as the explicit contract, structurally knob-free release-path solve, and the #492 past-cap census on every solve result). Sol cross-family review in progress; merge after adjudication + green CI.
UK local solve doctrine: uniform operator, declared bounds, past-cap …
MaxGhenis commentedon Jul 22, 2026
Increment 4 merged: #503 (solve doctrine). Post-review shape: the release-path solve has no doctrine injection point at all (bounds always come from the reviewed
UK_LOCAL_SOLVE_DOCTRINEconstant — cap 10.0, stretch 100.0), refuses per-target knobs structurally and duplicate (area, metric) surface rows as implicit weights;build_local_candidate(under_doctrine=True)is the release mode, refusing per-target solver options by name; and every solve carries the #492 past-cap census (every pushed-out row listed by default), persisted aspast_cap_census.jsonin candidate outputs.Lane state after today
Cross-family review protocol held for all three merged PRs: sol found 8 confirmed blockers across them, all fixed pre-merge.
MaxGhenis commentedon Jul 22, 2026
Increment 3 source adjudication: complete — all four gaps closed with verified sources
Follow-up to the gap register above; every source was fetched and inspected this session, not just documented:
census_2022_index.zip(nrscotland.gov.uk/media/utrbt5ze/census_2022_index.zip, on the 2022 Census Geography Products register):Postcode_To_OA.csvcarries HouseholdCount, spec-defined as the 2022 Census occupied household count (cell-key perturbed — the perturbation note must ride in the vintage metadata), summing to 2,508,715 across Scotland; aggregate byOutputArea2022Codefor the per-OA stage-one draw weight.OA_TO_HIGHER_AREAS.csvcarriesEW2022for all 46,363 OAs with zero blanks (point-in-polygon centroid methodology per the in-zip spec — record as the lookup method, analogous to ONS best-fit). One new pinned zip covers both Scotland gaps.build.nisra.gov.uk/en/custom/table.csv?d=HOUSEHOLD&v=DZ21— same table-builder API family as the already-pinned NI population source; fetched: 3,780 DZ rows with counts.DEA2014_cd/DEA2014_nm(verified on the downloaded file, 3,780 features) — the existing loader just needs to surface them.Net: increment 3 needs one new NRS zip pin + one new NISRA API pin + surfacing existing geojson properties, then the mechanical wiring (two loaders, per-country ladder frames concatenated into the NPZ with per-country layer vintages, S/NI mass-floor gate analogues, tests, and one full-UK artifact build). The spec PDF, column layouts, totals, and blank-counts above are the acceptance oracle for that wiring.
9 remaining items
MaxGhenis commentedon Jul 25, 2026
Increment 6d: the first calibrated UK rowwise candidate is built
Tool: #553. Real K=2 build against the byte-verified staging H5 and the full-UK ladder.
Result
populace_uk_2023_rowwise_candidate.h5, 3.14 GB, sha81523772…(local, unpublished)census_households/constituency— one family, named in the manifestThe artifact reloads through
load_uk_national_datasetwith the full fence chain intact.The finding worth more than the candidate
Calibration moved total household mass 28,840,551 → 28,097,269 (−2.58%), landing +0.13% from the census total. That is not solver noise — it is a real level disagreement between the two sources:
Census days are 2021-03-21 (E&W), 2022-03-20 (Scotland), 2021-03-21 (NI); the build period is 2023. So the gap sits in the right direction and rough magnitude for genuine household growth between census day and the build year — but I have not verified a UK household growth rate this session, so I am not asserting that as the explanation. What is verified is the arithmetic above and the vintage mismatch.
If the gap is mostly growth, then binding this family as a level control pulls the dataset back to a stale vintage and deletes ~780k households that exist in 2023. Two candidate treatments, for adjudication before this family is used as a level control:
This is exactly the class of thing the census's fence discipline exists to catch, and it argues for a fourth reviewed fence on the family (
census_vintage_level_mismatch) rather than silently accepting the −2.58%.Standing caveat on the candidate itself
Self-consistent by construction: the bound targets are the same ladder artifact that assigns geography. Strong end-to-end proof of the machinery, weak proof of calibration value. The real test is the first independent family (UC counts, HMRC income), gated on the SPI-frame-proxy / UC-grain / population-universe adjudications the census already records.
Not published. The candidate and its evidence are local only; publication is a deliberate human step.
First calibrated UK rowwise candidate builder (#553)
The ladder-clone result carries a populace Frame (#612 inc 2)
Ladder entry narrows to Frame | H5; #493 record; column-drop assessme…
The rowwise solve goes through the public calibrate() front door (#612 …
The ladder-clone result carries a populace Frame (#612 inc 2)
Ladder entry narrows to Frame | H5; #493 record; column-drop assessme…
The rowwise solve goes through the public calibrate() front door (#612 …
juaristi22 commentedon Sep 10, 2026
Closed with #870: increments 1–6 are merged (ladder clone route #540, rowwise solve #541, first bound family #542, K pilot #815, joint solve #852, rulings and registers #870), the doctrine constants are measured and pinned in
local_doctrine.py, and the first release candidate is assembled (A18,experiments/762-uk-rowwise-candidate-receipts.md). Private publication of the local variant and loader consumption move to #665 workstream I; the sized candidates continue under #355's follow-ups.Measure the WAS wealth predictors alike on donor and recipient (#1095)