Repository navigation
Port UK compact source stages and 2024-25 refresh from uk-data #145
Description
Parent: #144.
Why
Populace UK can replace the local-geography/matrix path, and the current compact H5 already carries several recent uk-data columns. But if policyengine-uk-data is archived, Populace needs to own the source-stage logic and provenance for those columns instead of relying on uk-data build history.
Scope migrated from uk-data
Open PRs:
- Impute mortgage and consumer debt from the Wealth and Assets Survey policyengine-uk-data#426:
mortgage_debtandconsumer_debtfrom WAS fornet_wealth. - Add PIP points from disability categories policyengine-uk-data#408:
pip_dl_points/pip_m_pointsfrom PIP categories once model variables exist.
Recent merged PRs that need Populace-owned source-stage equivalents:
- Regenerated ledger feeds fail the period contract: year-filtered bundles drop the CBO aging pivot and prior-year SOI tables #410/Add release-gate preflight: catch statically-knowable gate failures without a solve #436: FRS 2024-25 ingestion plus household-ID sort invariant for raw compact builds.
- Extend export-mass parity exclusions for the Build M frame #433:
employment_sector,sic_industry_division, and public-sector target source. - Measure household_weight from typed frame weights in the parity universe #428/Recognize series-identity ledger qualifiers as inert in the materializer guard #429/Gate the SSI swap delta at a runaway cap, not assignment granularity #431:
bus_fare_spending, bus subsidy scaling, and model-version floor. - Let voluntary filing predict from PUF-only survivor units #425: seeded stochastic draws for property purchase, BRMA, and capital-gains amounts.
- Restore ASEC ED_VAL via a pinned sidecar and close the source-carry class #421: SPI synthetic prior weights,
household_is_spi_synthetic,household_is_capital_gains_clone, and calibration source-share diagnostics. - Checkpoint and cache identity omit the staged frame: reform-vector cache ignores seed, constraint matrix unfingerprinted, manual materializer bumps #380/SYSTEM_REQUIREMENTS.md memory model is stale: attributes the 29 GB calibration peak to a dense-row vstack the compiler no longer does #383/Should *_would_be_qualified columns be derivations rather than QRF targets? #416/Child-support expense prevalence: survey-faithful 0.47% vs retired eCPS's drifted 2-3% vs OCSE-implied ~1.5-2% #417/SNAP state take-up: fill rates use pre-mass-repair weights and the export drops the eligibility surface — calibrated caseloads don't reproduce from the artifact #419: disability, PIP category, child/QYP, and transfer-side PIP component fixes.
- Assign SNAP take-up state-conditionally: reported receipt + national rate bakes in CPS state underreporting #372/Release-native loading: certificate-verified datasets by default; immutable-first publication #385/ACS-multispine staging: SSI probe finds spine-inconsistent incidence (2.6% vs 1.9% baseline rate) and weight-composition skew (ASEC 1.5 persons/hh, ESS 0.3%) #403-Fire the release Slack alert from publish_release; warn when the webhook is unset #407/Published release exports can no longer serve as --base-h5: QBI input reconciliation requires base-only PUF columns #412-Fix SNAP state take-up crash: Frame.broadcast only targets person #414: income/SPI/fuel/consumption prerequisite updates and road-fuel volume uprating.
Acceptance criteria
- Populace has UK source-stage modules/manifests for these compact-data inputs, with no
policyengine_uk_dataimports. - The compact artifact can be rebuilt for at least the current Populace UK release year and the FRS 2024-25 refresh path.
- Export-variable gates prove the compact H5 includes the same PolicyEngine-UK variables expected by
policyengine.py. - Determinism tests cover every stochastic assignment on the live build path.
- Source flags and diagnostics make synthetic SPI/capital-gains support inspectable without row-order inference.
2026-08-12 update — this issue is workstream E of the UK-migration master epic #665 (it absorbs the source-stage track of the now-closed #144).
Full item list to port, each as a declarative source stage + gate (children filed as picked up); order constraints: wealth before consumption (num_vehicles), income before capital gains.
- Full FRS table ingest (incumbent reads ~14 tables; microcosm reads 2): entities/IDs, grossing weights, demographics, council-tax imputation, LHA/BRMA, education, legacy-benefit proxies, disability categories
- Stochastic inputs: 11 take-up parameter files with reported-receipt anchoring, TV licence/FTB draws, seeded RNG discipline,
household_is_*provenance flags as first-class metadata - WAS wealth (11 outputs incl.
num_vehicles, student-loan balance allocation) — note theowned_landrelease-to-release instability recorded in Scottish/Welsh council tax ~20% above devolved budget figures; WAS owned_land imputation unstable across releases policyengine-uk-data#448 - LCFS consumption (18 outputs) + energy raking to NEED targets
- ETB VAT; ETB services + NHS usage
- Capital gains incl. top tail (Impute capital gains amounts from the published joint distribution #560 in flight; UK capital gains: imputed distribution has no top tail, missing 55% of CGT #552/UK CGT stage: HMRC outturn level anchors + Advani-Summers shape, with accrued/realized/taxable name separation #364/UK: banded CGT gains targets (by size of gain) to constrain the distribution, not just count + total #467)
- Salary sacrifice; student loan plans (SLC stocks); childcare take-up fitting; regional property uprating
- From Migrate remaining policyengine-uk-data backlog before archive #144: debt, bus fares (Add UK bus-spending imputation plan and DfT calibration targets #153), employer sector/SIC, PIP/disability, child/QYP
- New stages requested against the incumbent that land here instead: UC deduction attributes (Impute UC deduction attributes (latent rate, type combination) at dataset build policyengine-uk-data#450)
- Known incumbent defects the ports must not replicate: UC caseload ~40% below administrative counts (UC caseload runs ~40% below administrative counts, capping aggregate reform estimates policyengine-uk-data#452), Scottish/Welsh council tax 20–30% above devolved budgets (Scottish/Welsh council tax ~20% above devolved budget figures; WAS owned_land imputation unstable across releases policyengine-uk-data#448)
- FRS 2024-25 refresh (incumbent is already there; calibration year 2025)
Acceptance: the microcosm-built spine replaces the certified incumbent H5 in the national build; eFRS parity fixtures beat or match the incumbent on the frozen register with signed differences.
Activity
MaxGhenis commentedon Jul 8, 2026
Three portable lessons from policyengine-uk-v2#133 for the SPI source stage this issue tracks:
spi_incomestage spec should adopt the corrected map from day one and carry a test pinning all 12 regions — the bug lived only in one of the org's two SPI extractors, which is the cost of parallel extraction.2022_23, which is unambiguous).Also worth adopting repo-wide: uk-v2 runs a "Data pipeline dry run" CI job on PRs — a cheap wiring check populace's CI (tests-only) doesn't have; a build-plan dry-run job would catch stage-wiring breaks pre-merge.
juaristi22 commentedon Aug 14, 2026
Workstream E kickoff: child structure, sequencing, and verification plan (2026-08-14)
WS-E starts now, in parallel with the P1 critical path (#622/#623): only the final spine swap needs the fixed calibration harness — every ingest and imputation stage develops and verifies independently against a frozen parity bar (the incumbent's
enhanced_frs_2023_24.h5, the extraction already pinned inuk/efrs_parity_reference.json). Build at the 2023 vintage first; the FRS 2024-25 refresh follows parity sign-off.Children (filed as sub-issues of this epic)
source_stages.json, operation handlers, StagePlan migrationnum_vehicles)Critical path: E1 → E2 → E3 → E4 → E7 → E8 → E10, with E5 → E6 alongside and E9 floating. Incumbent releases after
ebf733cwiden only E11 — the parity bar stays frozen through E2–E9.Architecture doctrine (applies to every child)
Country-agnostic first; the US build is the model. Gates the US exercises that
uk/gates.jsondoesn't declare get declared into the shared battery per child (nonnegative_columns,enum_domain,support,aggregate_admin,macro_realism, …) — declarations and bindings only, no UK gate code. Operation kinds reuse or generalize existing vocabulary; UK-flavored kinds only where semantics are genuinely UK. Every child PR carries a UK-specificity note listing any UK-specific code and why it could not be general. Rules-engine dependencies run through a thin, country-parameterized adapter (sharedmaterialize_rules_engine_predictorsoperation; engine country must match dataset country; per-stage predictor lists declared in the manifest).Verification (per stage, then whole-spine)
Per stage:
StageRecord.nonzero_sharevs the committed 145-column parity reference (CI-checkable, tolerances as test literals); weighted-total deltas vs the licensed 131-column register (published SDC-safely, min count 3); tail-concentration recording (#609 instrument — accumulates toward arming the weighted-integrity gates); three-level determinism (bit-identity, row-permutation identity-stability, manifest-declared seeds). Whole-spine + swap (E10): candidate extraction diffed against the frozen reference, control-vs-candidate staging builds compared withtools/compare_uk_h5_payload.py, the full battery in release-candidate posture, and a committed signed-differences register — anything differing that is not in the register is a defect.Corrections to this issue's framing (found during planning)
household_net_income,capital_income,council_tax,total_income,current_education, …) through a mid-build Microsimulation. Hence the adapter doctrine above.Key adjudications recorded (2026-08-14)
WEEKS_IN_YEARuniformly; council tax ported post-uk-data#449 with per-nation signed deltas (uk-data#448);owned_landsupport receipt; UC take-up stays 0.55 for the parity build (uk-data#452 tracked against WS-D caseload binding).build/uk/resources with citations; the childcare fitting tool is ported (properly seeded) but stays out of the build, matching the incumbent.Full plan doc (child specs, unknowns register U1–U15, incumbent inventory):
uk-microcosm-migration-ws-e-plan.md, ask @juaristi22 for the latest copy.UK source stages on the shared manifest: source_stages.json + StagePl…
Ingest the full raw FRS table set and assemble the UK spine as a Frame
UK source stages on the shared manifest: source_stages.json + StagePl…
Ingest the full raw FRS table set and assemble the UK spine as a Frame
Ingest the full raw FRS table set and assemble the UK spine as a Frame
juaristi22 commentedon Aug 17, 2026
Adjudication recorded (2026-08-17):
receives_benefits_in_own_rightis produced by E3'sfrs_educationstage — settled with María on the E3 increment (PR #703).Context: the column is required by the 145-column coverage contract but named in no child issue's scope. It is a pure derivation over five E2 spine columns (UC/JSA/ESA reported > 0), computed in the QYP context E3 ports, and no other WS-E child claims it. E7 has no rewrite obligation: the incumbent's SPI stage-2 re-imputes the flag's inputs on synthetic donors but never re-derives the flag itself, so donors keeping the stacking default is port-faithful behavior (noted as a known incumbent quirk, not a defect to fix in WS-E).
Port the WAS wealth imputation and regional property uprating as decl…
juaristi22 commentedon Aug 19, 2026
End-of-workstream revisits registered from E5 (#681 / PR #714) adjudications (María, 2026-08-19):
owned_landinput-mass-parity reviewed exclusion expires 2026-09-19 (one-month deadline, entry ininput_mass_reviewed_exclusions.json). The E5 stability receipt measures a 37.7% national / 2.41× London weighted-total swing between adjacent seeds — the policyengine-uk-data#448 realization-variance class (sparse heavy-tailed WAS farm/estate records). The gate fails the expired entry until the adjudication is renewed, so this cannot silently rot. At the end of the E increments, either: adopt a variance remedy (winsorised donor variable or a separate land imputation per Carry the SSI reconcile per-pass trajectory through raise and record (#447) #448's suggestions) and drop the exclusion, or renew it with the E10 whole-spine evidence.num_bedroomspredictor quality in the WAS wealth QRF: the 2026-08-19 licensed audit foundHBedRmR8is −8 (UKDS sentinel) for 95.8% of round-8 household-tab rows — the bedrooms question is effectively unasked, and no alternative bedrooms column exists in the tab. E5 recodes the sentinel to 0 (signed difference; the incumbent trains on raw −8s), but a predictor that is ~96% structurally missing carries almost no legitimate signal either way. At the end of the E increments, decide: drop it from the predictor set (a method change vs the incumbent spec), source it differently, or keep-and-document.🤖 Generated with Claude Code
Adjudications: WAS sentinel recode, owned_land parity exclusion, boun…
Port the WAS wealth imputation and regional property uprating as decl…
Adjudications: WAS sentinel recode, owned_land parity exclusion, boun…
juaristi22 commentedon Aug 24, 2026
Filed #757 as the swap child of this issue: everything that must happen after the spine is approved — retiring the incumbent feed (W8/W9/W11, national driver to calibration-only,
UK_CANDIDATE_DATASET_NAMEtopopulace_uk_2024), arming the thresholds against the spine's own certified line per the US #327 doctrine, and retiring the workstream-E diagnostic markdowns into gates, regenerable artifacts, or dated archive.It is explicitly gated on #747 being reviewed and merged, since every step there either retires a fallback the spine depends on or re-points a threshold at the spine's own line.
Sign the five v13 target-fit deferrals as an expiring schema-2 register