UK spine: make SPI support households internally consistent (SPI-first order, housing shell)

changelog.d/uk-energy-rake-convergence-check.changed.md

@@ -0,0 +1 @@
+Check, rather than assert, that the UK `energy_rake` stage-health gate reads a converged NEED rake: each fuel's per-sweep residual may move by at most `maximum_residual_change_over_window` (0.1 point) over the last `convergence_window_sweeps` (10) sweeps, or the receipt fails as a truncated rake. The residual tolerance stays 2.5 %.

changelog.d/uk-gas-disconnection-identity-order.changed.md

@@ -0,0 +1 @@
+Impose the UK LCFS stage's published gas-connected share by disconnecting gas-positive households in an identity-keyed uniform order (seed 0) instead of smallest drawn gas first, so every group of households loses the same expected share of its drawn connected mass. Once the SPI support households drew gas from their own incomes, the smallest-first walk piled the disconnections onto low-draw flats and left the NEED rake a Scottish level conflict it could not close. The recipient NEED rake now runs 200 sweeps instead of 50, so its terminal residual is converged rather than truncated; the 2.5 % tolerance is unchanged.

changelog.d/uk-spi-before-donor-imputations.changed.md

@@ -0,0 +1 @@
+Run the UK SPI support block before the income-conditioned donor imputations, so WAS wealth, LCFS spending, ETB VAT and services and NTS bus travel are imputed onto the SPI rows from their SPI incomes instead of being copied from the FRS parent. Regional property uprating takes its factor over FRS-base owners, the UC reporter redraw keeps its place after the WAS stage behind a graph isolation boundary, the identity receipts follow the new stacking order, and a new instrument measures SPI-half coherence against the FRS parents.

changelog.d/uk-spi-housing-shell.added.md

@@ -0,0 +1 @@
+Impute the UK SPI support households' housing from their own incomes in a new `spi_housing_shell` stage: tenure, dwelling type, bedrooms and council tax band by weighted classifiers, housing costs and the housing-linked benefit reports by a chained regime-gated QRF, trained on the FRS base households with identity-keyed draws and learned structural zeros. A holdout tool compares the imputation with donor matching.

docs/uk-income-anchors-280.md

@@ -212,19 +212,11 @@ microcosm#1006, comments 5812328880 and 5814299594):
   on everything upstream, the band donors included. The limit stays at 1.0 in
   this tree; the fix belongs in the gate (a tolerance for a top-up of a couple
   of rows) and needs a ruling. Assessment builds raised it to 2.0 locally only.
-- **The SPI copies, band donors included, inherit their FRS parent's wealth,
-  spending and VAT.** The support channel and the band donors copy whole
-  households after `was_wealth`, `regional_property_uprating`,
-  `nts_bus_travel`, `lcfs_consumption`, `etb_vat` and `etb_services` have run,
-  and only incomes, pension contributions and reported benefits are replaced
-  afterwards. On spine-u all 45 WAS, LCFS and ETB columns on the SPI rows equal
-  the parent's; the rank correlation of investment income with gross financial
-  wealth is 0.14 on the SPI half against 0.54 on the FRS half; and the band
-  donors' median gross financial wealth is flat across the bands (£48k, £67k,
-  £50k and £44k from the 200k band to the 2m-and-over band, where median income
-  is £3.57m). The order is inherited from the enhanced FRS. The reorder that
-  runs the SPI block right after `frs_brma` is microcosm#1012, stacked on this
-  branch; it is deferred there, not folded in here.
+- **The SPI copies, band donors included, inherited their FRS parent's wealth,
+  spending and VAT** while this lane was open: on spine-u all 45 WAS, LCFS and
+  ETB columns on the SPI rows equalled the parent's, and the band donors' median
+  gross financial wealth was flat across the bands (£48k, £67k, £50k and £44k).
+  The next section runs the SPI block before those stages (microcosm#1012).
 - **The national calibration at this head blocks on the CGT projection
   fence.** With the spine rebuilt from the rebased head, the terminal gate
   `uk_cgt_projection_entrants` (introduced by #979) refuses: 146,920 weighted
@@ -241,6 +233,112 @@ microcosm#1006, comments 5812328880 and 5814299594):
   count, spread the band-donor mass over lighter rows, or cap their weight
   ratio).
 
+## SPI rows before the donor imputations
+
+The SPI block (`frs_hmrc_spine_leaves`, `spi_support_channel`,
+`spi_income_band_donors`, `hmrc_spi_income_spine`) runs right after
+`frs_brma`, ahead of `was_wealth`, `regional_property_uprating`,
+`nts_bus_travel`, `lcfs_consumption`, `etb_vat` and `etb_services`. Those six
+stages condition on income, and the support stages copy whole FRS households.
+In the enhanced FRS order, which the spine used to follow, each SPI copy kept
+the wealth, spending, VAT rate, public-service use and bus travel drawn for its
+FRS parent's income, while its adults' incomes were replaced by SPI draws. A
+£2m band donor then carried the financial wealth of the median FRS household it
+was copied from.
+
+With the SPI block first, every row, the FRS base rows included, is imputed
+from its final incomes. On the base rows this includes the SPI stage-1
+dividend redraw.
+
+Three details follow from the order:
+
+- `uc_reporter_redraw` stays after the six stages. Its award screen runs the
+  engine, and for benefit units without an FRS capital answer, UC capital falls
+  back to the household's WAS `savings`, property and `corporate_wealth`. The
+  graph opens a reader-isolation version before it, as it does for
+  `uc_capital_coherence`.
+- `regional_property_uprating` takes each region's factor over FRS-base owners
+  only, the population the factor was defined on, and applies it to every
+  owner, the SPI rows included.
+- The six stages now see the SPI support mass allocation, so their
+  weight-dependent steps run at the frame's prior (importance) weights: the
+  gas-connection walk, the NEED rake and DESNZ level, the ETB NHS
+  normalisation and the NTS receipts.
+
+## The SPI households' housing
+
+The support copies and band donors are whole FRS households, so after the
+reorder their wealth and spending followed their SPI incomes while their
+housing was still the parent's. On the reordered spine the SPI half's detached
+share sat at 27-29 % in every income band (the FRS half rises from 19 % to 47 %
+by £200k-1m), social renting stayed at 11-16 % above £50k and 16 % above £1m,
+and 402k of the SPI channel's 506k prior-weighted Housing Benefit households
+were owner-occupiers.
+
+`spi_housing_shell` runs right after the SPI income chain and imputes the
+housing of every SPI-channel household, the household counterpart of the SPI
+stage-2 FRS-only fill. It trains on the FRS base households and conditions on
+region, household type, adults, children, single-adult status, the reference
+person's age and the household's income components; no engine run, so housing
+benefits do not become circular. Tenure, dwelling type, bedrooms and council
+tax band are drawn in that order by weighted multiclass classifiers with an
+inverse-CDF draw; council tax, rent, both mortgage repayments, insurance,
+service charges, water, Northern Ireland rates, subletting, Housing Benefit and
+council tax benefit by one chained regime-gated QRF conditioned on those
+categories. The two benefit targets are trained on the head benefit unit's
+Housing Benefit and the reference person's council tax benefit. Every draw
+reads uniforms keyed on household id. Structural zeros are learned from the
+training households (an amount is zero wherever the FRS has no positive value
+for that tenure or region); mortgage payments need a mortgaged tenure, Housing
+Benefit a rented one, and council tax benefit is capped at council tax. Both
+drawn benefits land on the household reference person, with every other
+person's zeroed, and the stage checks it. Region, single-adult status, BRMA and
+composition stay the recipient's; FRS rows are unchanged.
+
+Imputation was chosen over whole-record donor matching on a holdout of FRS
+households (`tools/validate_uk_housing_imputation.py`). Within FRS support the
+two, and a forest-weighted whole-record draw, sit near the sampling noise; on
+the top 5 % of incomes held out and filled from the rest, which is the
+extrapolation the SPI copies need, imputation came closest on ownership (0.818
+against 0.855 observed; matching 0.787), detached share (0.396 against 0.446;
+matching 0.328) and four or more bedrooms (0.456 against 0.557; matching
+0.355), with the most distinct bundles. All three under-draw council tax band
+F and above at the top (about 0.19 against 0.35).
+
+## The gas connection
+
+`lcfs_consumption` imposes each region's published gas-connected share (DESNZ
+gas meters over electricity meters) on the drawn gas before the NEED rake.
+It used to disconnect the households with the smallest drawn gas first. Once
+the SPI support households drew gas from their own, higher incomes, that walk
+piled the disconnections onto low-draw FRS flats: Scottish flats' connected
+share fell from 76 % to 71 % while terraced and semi-detached houses rose. NEED
+publishes Scotland's gas means by property type and for all dwellings; on the
+tilted mix the property-type means implied a Scottish level 2.6 % above the
+all-dwellings mean, and the rake could not close it. At 300 sweeps the income
+margins converge to zero and Scotland's property-type cells all settle 2.54 %
+(reorder alone) or 2.56 % (with the housing stage) below target, against the
+gate's 2.5 %. The housing stage is not the cause.
+
+The walk now takes gas-positive households in an identity-keyed uniform order
+(seed 0, salt `lcfs_consumption:gas_disconnection`) with the same
+weight-fitting skip, so every group of households loses the same expected
+share of its drawn connected mass and only the level of the drawn connection
+moves to the published share. Its converged residual is 1.05 % (Scotland
+gas by property type), with the income margins at zero.
+
+The recipient rake also runs 200 sweeps instead of 50. With the SPI
+households' energy drawn from their own incomes, the income margins start
+further from NEED's gradient, and at 50 sweeps they were still falling
+(electricity 3.14 %, gas 2.76 %). The per-sweep receipt shows both under
+2.5 % from sweep 59 and flat by 200 (1.16 % and 1.36 % at 100, 0.91 % and
+1.12 % at 200, 0.82 % and 1.05 % at 300), so the gate reads a converged
+residual rather than a truncated one, as its governance note asks. The 2.5 %
+tolerance is unchanged. The gate now checks the convergence instead of
+asserting it: each fuel's residual may move by at most 0.1 point over the last
+10 sweeps (the final head moves 0.014 points; the 50-sweep receipts on main and
+after the reorder were still falling 0.11 to 0.75 points).
+
 ## Not done here
 
 - The property-income amount rows stay signed out: the spine's

packages/microcosm-build/src/microcosm/build/source_manifest.py

@@ -125,6 +125,7 @@
         "impute_disability_benefits_to_puf_support",
         "impute_energy_subsidy_to_puf_support",
         "impute_cell_means",
+        "impute_spi_housing_shell",
         "impute_housing_assistance_to_puf_support",
         "impute_other_health_insurance_premiums_to_puf_support",
         "impute_prior_year_income_to_puf_support",

packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json

@@ -7593,6 +7593,111 @@
       }
      }
     },
+    {
+     "type": "object",
+     "additionalProperties": false,
+     "required": [
+      "kind",
+      "training_rows",
+      "rows",
+      "identity",
+      "predictors",
+      "categorical_predictors",
+      "categorical_targets",
+      "categorical_fit",
+      "amount_targets",
+      "benefit_targets",
+      "amount_fit",
+      "support_clip",
+      "structural_zero_keys",
+      "rules",
+      "landing",
+      "n_estimators",
+      "seed"
+     ],
+     "properties": {
+      "kind": {
+       "const": "impute_spi_housing_shell"
+      },
+      "training_rows": {
+       "type": "string"
+      },
+      "rows": {
+       "type": "string"
+      },
+      "identity": {
+       "type": "string"
+      },
+      "predictors": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "categorical_predictors": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "categorical_targets": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "categorical_fit": {
+       "type": "string"
+      },
+      "amount_targets": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "benefit_targets": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "amount_fit": {
+       "type": "string"
+      },
+      "support_clip": {
+       "type": "string"
+      },
+      "structural_zero_keys": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "rules": {
+       "type": "array",
+       "minItems": 1,
+       "items": {
+        "type": "string"
+       }
+      },
+      "landing": {
+       "type": "string"
+      },
+      "n_estimators": {
+       "type": "integer",
+       "minimum": 1
+      },
+      "seed": {
+       "type": "integer"
+      }
+     }
+    },
     {
      "type": "object",
      "additionalProperties": false,
@@ -8835,9 +8940,14 @@
       },
       "disconnect_rule": {
        "enum": [
-        "lowest_drawn_gas_first"
+        "lowest_drawn_gas_first",
+        "identity_uniform_order"
        ]
       },
+      "seed": {
+       "type": "integer",
+       "minimum": 0
+      },
       "margins_resource": {
        "type": "string"
       },