UK dataset build: full migration from policyengine-uk-data (master epic) #665

Activity

1. juaristi22 commented on Aug 12, 2026

   

   juaristi22

   on Aug 12, 2026

   Last edited by juaristi22

   CollaboratorAuthor

   More actions

   The pipeline, and where each piece stands

   Status diagram for this migration, in the style of microcosm-dynamics#392. One node per pipeline piece, colored by state; updated as work lands (statuses as of 2026-09-10).

   flowchart LR
     subgraph CHRON["1 · Facts — chronicle"]
       W1["<b>Wave 1</b><br/>149-target national surface"]
       W2["<b>Wave 2</b><br/>national families landed<br/>#133 closed"]
       W3["<b>Wave 3</b><br/>24 local packages landed<br/>#134 closed"]
       SUITE["<b>UK consumer artifact</b><br/>suite + facts-only tooling<br/>#135 closed"]
       CLEAN["<b>Facts-only cleanup</b><br/>profile surface retired<br/>#172 closed · PR #206"]
     end
   
     subgraph TGT["2 · Targets — microcosm"]
       REFS["<b>uk_target_references</b><br/>388 active refs at 2025 +<br/>2 compile-parity gates<br/>#622 closed · PR #735"]
       CONTRACT["<b>Calibration contracts</b><br/>all re-homed per #166:<br/>merged into one population<br/>contract — #707 · #708 closed"]
       ADJ["<b>WS-C deferral registry</b><br/>#736 — in progress; scalers<br/>#790 + cross-grain #802 ruled;<br/>region grain · uprating holds open"]
     end
   
     subgraph SPN["3 · Spine + imputations"]
       FRS["<b>Raw FRS ingest</b><br/>+ derived + stochastic layers<br/>#678 · #679 · #680 closed"]
       SPI["<b>SPI donor + HMRC ODS</b><br/>income stages — #683 closed"]
       IMP["<b>Imputation stages, all landed</b><br/>WAS #681 · LCFS/ETB #682 ·<br/>CGT #560 #684 · E9 net-new #685 —<br/>retargeted to FRS 2024-25 #723"]
       SWAP["<b>Whole-spine parity + swap</b><br/>#686 + #757 closed —<br/>PRs #747 · #787 · #793 merged"]
       SPINE["<b>National spine</b><br/>microcosm-built at FRS 2024-25;<br/>incumbent feed + June path<br/>retired (#787)"]
       SPDEF["<b>WS-E deferral registry</b><br/>#796 — in progress; #750 #785<br/>#790 #792 closed · open: #791<br/>#688 #721 #725"]
       UPR["<b>Uprating / multi-year</b><br/>#148"]
     end
   
     subgraph GEO["4 · Geography"]
       OA["<b>OA ladder</b><br/>full UK"]
       CLONE["<b>Rowwise clone</b><br/>+ area assignment"]
     end
   
     subgraph CAL["5 · Calibration"]
       NATCAL["<b>National calibration seam</b><br/>#623 closed — doctrine, trio,<br/>diagnostics, scorer live;<br/>403 targets bound at 2025"]
       LOCCAL["<b>Rowwise local calibration</b><br/>#759–#762 closed — local +<br/>national solved as one matrix<br/>on the cloned spine (PR #852)"]
     end
   
     subgraph GATE["6 · Gates + record"]
       BATTERY["<b>Gate battery</b><br/>13 gates, signed schema-4"]
       SCHEMA4["<b>Schema-4 verification</b><br/>PR #663"]
       REBUILD["<b>Fresh rebuild passes gates</b><br/>#630 resolved"]
       ARM["<b>Weighted-integrity<br/>gates armed</b><br/>#609"]
       LOGBOOK["<b>Logbook: every attempt<br/>recorded</b><br/>#666"]
       CI["<b>UK CI lane</b><br/>suite tiered + run in CI<br/>#699 closed · PR #766"]
     end
   
     subgraph REL["7 · Products"]
       NAT["<b>National release</b><br/>first certified cut: queue down<br/>to 6 UC-side cells; assembly<br/>landed (#806 · PR #812)"]
       SPARSE["<b>Sparse exact-k default</b><br/>informed L0 + exact-count<br/>candidates — #355 closed"]
       LOCVAR["<b>Local variant</b><br/>rowwise candidate + registers<br/>+ receipts — #146 closed"]
     end
   
     subgraph DOWN["8 · Downstream"]
       PEUK["<b>policyengine-uk consumes</b><br/>microcosm datasets<br/>national · sparse · local"]
       VALID["<b>Reform-validation suite</b><br/>#365"]
       ARCHIVE["<b>Decommission</b><br/>uk-data archived"]
     end
   
     CLEAN ~~~ BATTERY
     CLEAN ~~~ SCHEMA4
     CLEAN ~~~ ARM
   
     W1 --> REFS
     W2 --> REFS
     SUITE --> REFS
     W2 --> CONTRACT
     W3 --> CONTRACT
     CONTRACT --> NATCAL
     CONTRACT --> LOCCAL
     CONTRACT -.-> CLEAN
     REFS --> NATCAL
     REFS -.-> ADJ
     SPINE --> NATCAL
     UPR --> NATCAL
     FRS --> IMP
     SPI --> SWAP
     IMP --> SWAP
     SWAP --> SPINE
     SWAP -.-> SPDEF
     OA --> CLONE
     SPINE --> CLONE
     CLONE --> LOCCAL
     LOCCAL --> LOCVAR
     NATCAL --> REBUILD
     BATTERY --> REBUILD
     ARM --> REBUILD
     REBUILD -.-> LOGBOOK
     REBUILD --> NAT
     SCHEMA4 --> NAT
     NAT --> SPARSE
     NAT --> PEUK
     LOCVAR --> PEUK
     NAT --> VALID
     PEUK --> ARCHIVE
     VALID --> ARCHIVE
   
     classDef done fill:#2f6f52,color:#fff
     classDef flight fill:#b8721a,color:#fff
     classDef blocked fill:#8b3a3a,color:#fff
     classDef transitional fill:#4a5d78,color:#fff
     classDef todo stroke-dasharray:4
   
     class W1,W2,W3,SUITE,CLEAN,SPI,OA,CLONE,BATTERY,SCHEMA4,ARM,LOGBOOK,REBUILD,CI,REFS,CONTRACT,FRS,IMP,SWAP,SPINE,NATCAL,LOCCAL,SPARSE,LOCVAR done
     class ADJ,SPDEF,NAT flight
     class UPR,PEUK,VALID,ARCHIVE todo
   
   

   Loading

   🟩	Done — merged and exercised
   🟧	In flight — open PR, or partially landed
   🟥	Blocked — fails today; needs adjudication or repair
   🟦	Transitional — works today but exists to be replaced (none remain — the incumbent-built spine retired 2026-08-27)
   ⬜	Not started

   Workstreams G and H are complete. The local lane closed end to end: the K=1 pilot (#761, PR #815), the local target surface + credibility gates ported with the uk-data parity register and masking doctrine (#147, PRs #830/#831), the first calibrated rowwise candidate solving local and national targets as one matrix on the cloned spine (#762, PRs #852/#870 — rulings, registers, receipts, and the dense release line), and the local variant itself (#146). The epic #495 closed 2026-09-10. On the national-default side, #355 closed the same morning via PR #877: informed-L0 and exact-count candidates for the ~50k default. WS-E also lost its last deferral — the E9 net-new stages (#685: WAS debt chain, UC deduction attributes) landed via PR #850.

   What stands between here and a certified national line is 6 cells, all UC-side. The #796 capstone queue is down from 13 to 6 (2 TCL disability cells → #808, 3 composition cells + 1 payment band → #809, all diagnosed as measurement-grain defects with the re-bind PR designed); before the cut, the candidate-of-record re-certifies on a post-#755 spine (spine-j needs the RC-strict battery + parity instruments + signed-differences re-mint). Release assembly is already mechanical: #806 closed via PR #812, and the September fix train (PRs #879–#892) repaired the UC measurement contracts and capital-graph inputs behind those cells.

   Both deferral registries keep shrinking. WS-C (#736): the four post-calibration scalers adjudicated (#790) and cross-grain reconciliation ruled — country wins (#802, PR #827); region-grain activation and the uprating holds remain. WS-E (#796): #750, #785, #790, #792 closed; #791 (household-composition frame column — its exclusions expire 2026-11-26), #688, #721, #725 remain open.

   Still ⬜ after this update: uprating/multi-year (#148, WS-F), the reform-validation suite (#365), and the two unfiled workstreams — I (policyengine-uk consumes microcosm datasets) and J (decommission). Those are now the whole remaining surface of the migration.

   Layout note (this update): with the entire spine lane closed, the serial imputation nodes (landed imputations → retarget → remaining stages) merged into one green node — that shortened the dependency chain and pulled boxes 5–8 roughly 600px left, closing the gap the previous layout left between blocks.

   Phase milestones (from the epic body): P1 national seam ✅ · P2 microcosm-built spine ✅ · P3 local variant ✅ · P4 sparse default ✅ + downstream integration ← current front (certified cut + WS-I) · P5 archive.

   Maintained by @juaristi22 — statuses re-checked against the linked issues and every open PR on each update.
2. juaristi22

   added 10 commits that reference this issue on Aug 14, 2026

   UK source stages on the shared manifest: source_stages.json + StagePl…

   ...

   925d0bb

   Ingest the full raw FRS table set and assemble the UK spine as a Frame

   ...

   16bdfc0

   UK source stages on the shared manifest: source_stages.json + StagePl…

   ...

   51f1151

   Ingest the full raw FRS table set and assemble the UK spine as a Frame

   ...

   93964d0

   Ingest the full raw FRS table set and assemble the UK spine as a Frame

   ...

   c199347

   UK gate adjudication (#630): reference registry, per-reference exclus…

   ...

   228cafb

   UK gate adjudication (#630): reference registry, per-reference exclus…

   ...

   a90cbf1

   Port the WAS wealth imputation and regional property uprating as decl…

   ...

   6fb284d

   Port the WAS wealth imputation and regional property uprating as decl…

   ...

   e503541

   Port the LCFS consumption, ETB VAT, and ETB services/NHS layers as de…

   ...

   e12be04
3. MaxGhenis commented on Sep 6, 2026

   

   MaxGhenis

   on Sep 6, 2026

   Last edited by MaxGhenis

   Contributor

   More actions

   UK handoff for Tuesday 8 September — current PRs and release checks

   Updated 2026-09-08 11:11 UTC. #870 merged at 09:52:11 UTC as 289437765a2d5caca97f2974f4d104e9e1e066d5, from head 74e7d8c80a923b80c581f6da83b94260a7442dd2. Its last change aligned tests with the six local gates; the reviewed production code was unchanged. The signed-input binding and malformed-identity findings below remain follow-up requirements on main. No new releasable UK file has been established by this local work.

   New activity. #883 is no longer draft; its code head remains 9d13d1dda79083a80348928645e3d774a4506cf7. #881 and #879 retain their reviewed code heads. Draft #877 now targets main with head 56aa4e25df91b10ce00e623bfe52bc3c5caef8f6, and is MERGEABLE/UNSTABLE: 9 successful and 13 unfinished checks at the saved read. Its new head and shared-solver changes have not yet received a fresh local review. The former conflict/no-CI status is superseded. #878 remains closed without merge.

   Shared architecture progress. Local US integration includes main's #870 merge. The accepted catalogue-first ACS+ASEC graph constructs both surveys before complete PUF support cloning; fresh-process reuse reconstructs source authority. The corrected sampling budget passes 42 cases, including the three failures found by independent review. Original ASEC wage observations pass 14 corrected cases; allocation/universe flags remain unresolved and feature eligibility stays false. Separate numeric/transport components pass 45 focused tests. The seven-node calibration runner is receiving manifest and independent-diagnostics corrections; its full execution is not yet claimed. These are local invented-data components, not a new calibrated US or UK file. Methods and both papers have been updated locally. No remote branch or release has been published.

   New #870 review and next actions

   The repair makes local target-fit, per-family-fit, weight-ratio and ESS checks release-blocking and adds source-manifest checks. A remaining join is insufficient: the dense validator verifies that an unsigned candidate manifest points to the signed report, while the local producer supplies no signed commitment to the measured inputs.

   An invented reproduction retains the signed gate report byte-for-byte, changes an active diagnostic from target 10 to estimate 1 (90% error), and refreshes the unsigned packaging/score/evaluation hashes. The release validator still accepts it despite the 25% local-fit limit. No signing key is used after the original fixture report is created. Bind the signed verdict to the actual measured dataset/household identity, support and target-error inputs, then verify that commitment against the shipped dataset and original diagnostics. Add a regression that retains the old signed report while replacing those inputs; the bundle must refuse. This is a stale-evidence reproduction, not a claim about the actual UK candidate.

   Also validate nested identity mappings before calling .get: a list-valued identity.code currently escapes as AttributeError instead of ReleaseContractError. It fails closed, but bypasses callers' named error handling.

   The reviewer ran 49 isolated contract tests and two invented probes. Assembler/evaluator and full gate-executor integration remain unexecuted. An initial broader collection loaded packaged UK configuration/reference resources before a guard refused; that deviation and the failed attempt are retained. No microdata, reserved US targets, country engine, fit or score ran. This is an incremental contract review, not full-restack or population validation.

   #883 review context. #883 is now at 9d13d1dda79083a80348928645e3d774a4506cf7, with its latest commit reviewed as an incremental graph/source delta. The earlier full review at 316b4af7 remains PARTIAL / COMMENT, zero confirmed critical defects and two items to address. The latest exact-head check found no CI, with CLEAN/MERGEABLE status. A releasable calibrated UK file remains outstanding.

   Latest #883 delta. The new commit preserves is_uc_claimant as a Boolean graph column and binds the shared relationship helper into the three consuming stages’ cache identities. Local graph checks passed 11 tests with two UK-engine skips at 9d13d1d; the 162-row invented person fixture preserves all previous fields. No new defect was confirmed in this delta. The earlier 283-test review below belongs to 316b4af7, not a full rerun at the new head. Both action items below remain unchanged. The Boolean fix closes part of the claimant-input graph gap; companion-engine behavior, the full FRS build and release-coverage validation remain open.

   New #883 review and next actions

   The new code aligns FRS couple predictors across UC measurement and redraw, keeps child/disability reductions within the claim's benefit unit, adds GB predicates, caps the final included monthly-payment band at £2,500 inclusive, and pins the nine April–December 2025 source observations. The separately published £2,500.01-or-over category is still unbound. The code records positive-award, annual-state and birth-year proxies explicitly; those remain approximations to the administrative claim/date definitions.

   Independent code review ran 283 passing tests and five engine/feed-dependent skips across 12 files, plus invented solver/claimant reproductions. Source review checked four official DWP/FRS methodology resources. This review did not regenerate the licensed population or approve a release.

   1. Record selected-iterate provenance for UK firm calibration. The shared optimizer's new best-iterate behavior also reaches solve_firm_weights, which does not pass the selection receipt. An invented reproduction selects an earlier iterate; the firm result recomputes final_loss correctly, but the selected epoch and the meaning of the trajectory's last entry are undocumented. Either pass the receipt through that caller or explicitly scope the behavior.
   2. Complete the capital-stage required-input list. cohere_uc_capital now uses is_benunit_head and is_parent through the shared relationship mask, but its entry validation does not declare those two required person columns. The sibling redraw stage does. Both current error paths refuse; this is input-contract/diagnostic consistency, not demonstrated data corruption.

   Before a release build, verify the locked UK engine variables and run the claimant-aware path with the companion PolicyEngine-UK #1850. Microcosm's dependency lock is unchanged. The new is_uc_claimant column and the one-or-two-claimant condition need full FRS/spine and release-coverage validation; the frozen coverage candidate does not certify the new input. Exact PIP/disabled-child and Scottish youngest-child source-table numerators remain a specific source-review gap. No new population fit or target-value equivalence is established by these code checks.

   Optional cleanup: give zero-epoch solves a consistent selection receipt/label, and clarify the capital-stage is_uc_couple derivation name. The formal local report is retained in the review archive; this paragraph is the actionable GitHub handoff. No PR comment, code push or merge was made by this review.

   Earlier #879 / #881 handoff, retained with its original context

   New #881 review

   #881, head 19172e8f1baa754f728ac6cdb1d9fe7b09afe17d, makes the FRS disability survey-year and DSA calibration-year rules explicit. Local review ran 190 passing tests with seven UK-engine skips. Both hosted UK engine jobs and both wheel jobs now pass, with all 23 checks successful at the reviewed head. Review status is partial, with no confirmed calculation defect and two documentation corrections. The newly activated DSA path was not independently executed through a local UK engine to the final benefit.

   • In the receipt and signed-difference entry, distinguish the allocated £18,000 from the policy maximum. The 2.94.0 parameter sets the 2025 ceiling to £27,783. The allocation code also caps at reported grants; this discrepancy is in the evidence label and does not establish that the recorded allocation is wrong.
   • Narrow the SPI invariance assertion. The receipt runs seven FRS stages, ending before the SPI redraw or calibration. Shared derivation with identical inputs does not establish unchanged post-redraw counts or fitted results. The new test pins implementation agreement, not population invariance.

   Current shared work

   • #879, head eb7ca8a541425ff6029f60017d1acf30e29372ff, now targets main at 5ab1b056, including merged Port the post-freeze childcare / Tax-Free Childcare inputs and targets, re-pin the Chronicle feed, and activate the DfT bus targets (#834, #789) #874. It is no longer draft. Hosted CI now has all 23 checks successful, including both wheel jobs, and GitHub reports the unchanged head mergeable/clean. The old stacked-base/no-CI description is superseded.

   • The review repairs restore optional-engine test boundaries, add an authenticated retained-evidence builder and 84 explicit UC matrix-row checks per run, name both source-spine origins, and annotate four historical UC deferral reasons. The annotations preserve operative decisions and dates; they do not renew approvals. The author reports 588 passes/19 skips on each engine-free Python version and 81 passes/two skips in the engine environment across overlapping scopes. These are not additive or release-certification counts.

   • #870 merged on September 8 at 09:52:11 UTC as 289437765. The final head changed tests only. The two production findings above remain follow-up work on main; a merge does not supply a new measured-input commitment or candidate acceptance.

   • #874 merged at 13:54:58 UTC as 5ab1b056f4d3e8a71873b13fe566f0bb899899b6, from head 659adb1a. Its fiscal-year, DfT and take-up-contract changes are now in Correct SPI income inputs and UK calibration measurements #879’s base. The retained comparison was not recalibrated on this new base.

   • Draft #877 now has head 56aa4e25df91b10ce00e623bfe52bc3c5caef8f6 against main; its live status is recorded above. This supersedes the old 60e10421 conflict. The new version still requires local review, and its size-selection evidence remains separate from Correct SPI income inputs and UK calibration measurements #879’s calibration comparison. #878, head baac62fa, was closed without merge on September 8 at 08:28 UTC. Its earlier stacked-parent/mergeable status is historical; this closure establishes no validation of the CGT-tail work.

   What the retained comparison establishes

   The corrected comparison and paired evidence retain control spine d43b4203, SPI spine fc49b482 and measurement code 117860ef: UK 2.94.0/Core 3.31.0, Chronicle 6fb700e, 371 active targets, 1,500 epochs, seed 0 and maximum weight ratio 10. They are a matched national, single-seed experiment at those original pins.

   Loss falls 39.8%, with 337/371 → 341/371 targets within 10%; interest supplies 91.5% of the net gain. Income-tax error worsens −12.067% → −12.709%, and state pension −7.909% → −11.430%. Both corrected runs fail the target-fit gate and refuse calibrated H5 export. Both lack childless-couple support above £26.4k; the earlier apparent high-payment support was a duplicated lone-parent case. The treatment also retains self-employment and pension-band failures. No later rebase changes those historical results.

   Next actions

   1. Finish exact-head review and CI. Address Declare survey- and calibration-year rules for UK FRS parameter reads (#862) #881’s receipt labels and downstream evidence claim, then review Correct SPI income inputs and UK calibration measurements #879’s repairs against Vahid’s findings, and fix the merged UK rowwise local candidate: rulings, registers, receipts, and the dense release line (#762 PR B) #870 signed-input binding and malformed-identity findings on main before rechecking release-contract tests and CI. Keep original receipts and failure evidence intact.
   2. Repair UC measurement and generated support before the next matched calibration. The inventory records 25 included positive-payment bands per family while DWP publishes 26. The omitted £2,500.01-or-over monthly tail is currently absorbed by the highest included row despite its bounded source label. Resolve endpoint and missing-tail semantics under WS-C deferred adjudications and the reconciliation layer #736. Separately align redraw and calibration family definitions, verify FRS role codes, and review the model’s standard-allowance relationship and partner eligibility. Relabelling families does not fix entitlement or create missing unique source support.
   3. Reassess the remaining joint-fit and release decisions. Investigate the UC tail, self-employment and pension failures, and freshly adjudicate stale exclusions rather than adding exceptions to obtain a pass. State-pension uprating sensitivity remains unmeasured. Keep Stabilize UK CGT open-tail source amounts with paired build evidence #878’s CGT-tail work and UK CGT: translate the 2024-25 gains outturn to the 2025 calibration year (forestalling year; OBR CGT +44.2% signed on the stack register) #875’s timing reconciliation separately attributable. After those decisions, run a new matched comparison on the chosen integrated source, expose its actual diagnostics and incomplete checks in the dashboard, then run the release battery for that exact candidate.

   The overnight local SPI integration and small UK graph/dashboard remain historical source/invented evidence. They do not replace the shared genuine comparison or establish local-area accuracy. The original morning Slack link points to this updated handoff; no second Slack message has been sent.