diff --git a/changelog.d/1063-cgt-residential-split-and-table-7.changed.md b/changelog.d/1063-cgt-residential-split-and-table-7.changed.md new file mode 100644 index 000000000..497e575f8 --- /dev/null +++ b/changelog.d/1063-cgt-residential-split-and-table-7.changed.md @@ -0,0 +1 @@ +The UK spine carries the residential-property flag as weight instead of drawing it (microcosm#1063 item 7). A new expansion stage, `cgt_residential_split`, runs after `cgt_incidence_anchor` and before `hmrc_cgt_asset_type_spine`, which moves after the anchor: it solves the residential logistic over the liable gainers exactly as the asset-type stage did (intercept to Table 8a's individuals-basis count, slope to its gains, the stock shift on `other_residential_property_value`), and then splits every household carrying liable gainers into arms, identical in every cell except `capital_gains_residential_property`: one liable gainer gives a residential arm at `p x w` beside the incumbent at `(1 - p) x w`; `k` liable gainers give `2^k` arms at product weights, with more than three refused. The arm index is `cgt_residential_clone_index` under the kernel's `id + k x offset` rule, the flag `household_is_cgt_residential_clone`, and the gainer's probability `cgt_residential_probability`. Mass and every household's composition are conserved (graph policy `conserve`, a declared factor of 1.0 under the stage's own reason, a `mass_conserving` family in the coverage manifest), the incumbent is corrected to the exact total, and no draw or seed is involved. At design weights the residential count and gains are then identities on Table 8a in every HMRC gain band, where the walk this replaces was bounded by one row's weight for the count but by one row's stake for the gains, so the open band was either short by its lottery expectation or over by a stake several times the band's whole expectation. The stage gate `uk_stage_cgt_residential_split_mass` holds the conserved mass, the arm weights as the declared products, the arm count implied by the households by liable gainers, the solve and the identities by band; `uk_stage_hmrc_cgt_asset_type_spine_summary` holds the identities the arms restate and drops the walk's bounds, offsets and seed 553; the `assign_residential_property_flag` operation leaves the manifest. The terminal `uk_weight_ratio` and `uk_weight_ess` gates fold the arms onto their household as they fold the support copies, the geography identity kernel keys an arm on its explicit index as the `cgt_residential_split` branch, the export surface carries the three columns, and the E8 identity receipt reconstructs the arms from the stored frame (every arm weight against the recomputed probability, the folded masses against Table 8a) before it checks the clone pairs and support copies on the arm-free frame. The main asset type is fitted once per claimant status (item 8): Business Asset Disposal Relief and Investors' Relief claimants to Table 7's shares among the three types the reliefs apply to, non-claimants to the shares that restore the published composition overall, realised by an error-diffusion walk in descending gain order so the realised gains by type track the fit; that assignment is deterministic and seed 554 is retired. The receipt reports the composition by claimant status against Table 7; the fact fence and the diagnostic-only status of the composition are unchanged. diff --git a/changelog.d/1063-child-benefit-take-up.changed.md b/changelog.d/1063-child-benefit-take-up.changed.md new file mode 100644 index 000000000..8c685c3a3 --- /dev/null +++ b/changelog.d/1063-child-benefit-take-up.changed.md @@ -0,0 +1,5 @@ +Redraw UK Child Benefit claims and opt-outs after the SPI income chain against HMRC's August 2025 statistics (#1063 fold-in). `frs_take_up` drew `would_claim_child_benefit` at a flat 89% (HMRC, 2022) over every benefit unit with the FRS reporters added on top, and `child_benefit_opts_out` at a flat 23% over every benefit unit whatever its income; the engine pays on the first and never reads the second, so 96% of eligible children were claimed for and paid, and `obr.child_benefit` sat about 20% over. + +A new spine stage, `child_benefit_take_up`, runs after `pension_credit_take_up`. One temporary engine materialization gives each person's eligibility (child or qualifying young person) and adjusted net income. A family claims for all its children or none, so the draw is by family, keyed on the age of its eldest eligible child: a reporter claims, and the other families claim at rates solved from the eldest age down so that the claimed share of the eligible children of each single year of age is HMRC's published claim rate (Table 15, May 2025: 68.8% at age 0 rising to 96.6% at 15, 86.6% overall, opted-out families included). The published opted-out share of registered families (684,635 of 7,552,330 at August 2025) of the claiming families then opts out, drawn among claiming families that do not report receipt and whose highest adjusted net income is at or above the income at which the High Income Child Benefit Charge takes the whole benefit, then inside the taper for any remainder. The stage stores claims-and-not-opted-out in `would_claim_child_benefit`, the flag the engine pays on, and the opt-out in `child_benefit_opts_out`. + +On the 1 October spine at design weights the redraw claims for 86.7% of eligible children, opts out 9.0% of claiming families (all from families charged the whole benefit) and leaves 11.1m children in payment, down from 13.5m. The published rows are vendored from the pinned Chronicle feed as `hmrc_child_benefit_statistics.json`. A stage gate, `uk_stage_child_benefit_take_up`, holds the claimed share overall and by age and the opted-out share; the terminal take-up signal gate no longer measures the two flags. The roster moves from 36 to 37 stages. diff --git a/changelog.d/1063-identity-receipts-after-salary-sacrifice.fixed.md b/changelog.d/1063-identity-receipts-after-salary-sacrifice.fixed.md new file mode 100644 index 000000000..d1d13a8ca --- /dev/null +++ b/changelog.d/1063-identity-receipts-after-salary-sacrifice.fixed.md @@ -0,0 +1 @@ +The UK E8 identity receipt passes again on a spine whose `salary_sacrifice` stage lowers a converted record's pay (microcosm#1063). Since microcosm#1069 c9 the conversion rewrites `employment_income` and `employee_pension_contributions` in place, after the capital-gains support split and the incidence anchor have banded every carrier on that income; on the 2 October spine the receipt's recomputes therefore banded 2,797 clone carriers on post-sacrifice pay, moved 82 of them down a Table 3 band, derived different anchor targets (55,622 households off by up to 286 weight units) and selected a different support family in two bands. The stage now keeps a converted record's pre-conversion pay on `salary_sacrifice_pre_conversion_pay` (zero elsewhere; an internal carrier dropped at the release boundary), and every receipt of a stage before `salary_sacrifice` restores the pay and the employee contribution from it before recomputing. The residential split's logistic solve walks the liable gainers in `person_id` order and breaks a tie for the exact-total correction row by household id, so its arm weights are bitwise identical for any person order (the same receipt found 1e-13 between two orders). diff --git a/changelog.d/1063-identity-receipts-green.fixed.md b/changelog.d/1063-identity-receipts-green.fixed.md new file mode 100644 index 000000000..be2c6b6bf --- /dev/null +++ b/changelog.d/1063-identity-receipts-green.fixed.md @@ -0,0 +1 @@ +The UK identity-stability receipts pass again on a spine built from main (microcosm#1063). E7 read every SPI-synthetic household as support clone index 1, so it failed on every spine carrying the income band donors, which are stacked at index 2; it now reads the donors' own index. E8 compared the stored and recomputed `student_loan_plan` with `Series.equals`, which calls two equal columns of different string dtypes unequal; it now compares the values. E8's reversed-order recompute of the capital-gains incidence anchor differed from the forward one by a few ulps (1e-13 on the licensed spine): the sums the anchor's factors depend on were pairwise float sums, which move in their last bits when the rows are permuted. The anchor now takes correctly rounded sums, so its weights are identical for any row order. diff --git a/changelog.d/1063-release-cut-blockers.fixed.md b/changelog.d/1063-release-cut-blockers.fixed.md new file mode 100644 index 000000000..7bce8c6ba --- /dev/null +++ b/changelog.d/1063-release-cut-blockers.fixed.md @@ -0,0 +1 @@ +Clear the UK release-cut certifier's blockers found by the rehearsal on the 2026-09-30 national build (microcosm#1063 c9). The candidate label is `microcosm_uk_2024_25`; the certifier and the full-build gate evidence read the export surface through `uk_export_candidate_columns`, which strips the structural id columns and carries `household.household_weight` as the enhanced-FRS reference does, so ids are no longer reported as unreviewed extras nor the weight as missing. Every declared stage output the incumbent never carried is allow-listed (ruling 2026-10-02): the source and support-channel lineage of the three entities, the council tax family, the reported-benefit inputs the take-up stages read, the SPI channel's `hmrc_spi_*` leaves, the FRS education and housing fields, the two fenced raw FRS codes and `other_investment_income`. The one reviewed export exclusion, `person.incapacity_benefit_reported`, and the five internal disability carriers (`attendance_allowance_reported`, `dla_sc_reported`, `dla_m_reported`, `pip_m_reported`, `pip_dl_reported`, stage-internal inputs the export registers keep off the release) now leave at the release boundary: `uk_release_export_frame` drops it from the written candidate, the graph readback expects the same frame, and the spine keeps the column for the SPI income stage. The FRS 2024-25 vintage's recoded training variable (`TRAIN2`, where 9 is "None of these" and 10 the Kickstart Scheme) is read, so `is_in_approved_training` carries signal again. The LCFS donor's diary consumption is floored at the declared `floor` (0) before the imputation, the support-clip ranges and the rake see it, with a receipt of the rows raised; `uk_stage_lcfs_consumption_support` declares the floor and requires the receipt, no remaining negative donor row and no clip range below it (250 release rows had inherited a minimum of -14,165 in housing, water and electricity consumption). The two FRS education inputs the enhanced-FRS reference barely carries (`adult_ema`, `dfe_education_spending`) enter the per-reference input-mass register, and the sparse charity-gifts leaf re-enters the QRF-tail register, each a signed receipt expiring 2027-04-02. `uk_stage_child_benefit_take_up` declares `population_fact_check` like the Pension Credit gate, so a synthetic smoke build records its failure and continues, and the anchor composition check allows the absolute rounding tolerance the other mass identities use. diff --git a/changelog.d/1063-retire-mass-increasing-support.removed.md b/changelog.d/1063-retire-mass-increasing-support.removed.md new file mode 100644 index 000000000..c9ab87026 --- /dev/null +++ b/changelog.d/1063-retire-mass-increasing-support.removed.md @@ -0,0 +1 @@ +Retire the `mass_increasing_support` semantics from the UK release-input coverage validator (#1063 item 3). Its last user, the SPI income band donor stage, is now a funded support channel, so every stage family declares `mass_conserving` and the terminal family gate accepts only a household mass record that conserves the total and declares factor one. A manifest that names the retired value is refused when it is read. diff --git a/changelog.d/1063-salary-sacrifice-earners-only.fixed.md b/changelog.d/1063-salary-sacrifice-earners-only.fixed.md new file mode 100644 index 000000000..63f419808 --- /dev/null +++ b/changelog.d/1063-salary-sacrifice-earners-only.fixed.md @@ -0,0 +1 @@ +Predict UK salary sacrifice only for people who are paid (#1063 fold-in). The `salary_sacrifice` stage fitted its QRF on the FRS respondents asked the question and predicted for everyone else in the frame, children and people without a job included: on the 30 September national build 1.94m of the 7.6m people with `pension_contributions_via_salary_sacrifice` had no employment income (0.53m of them under 16) and carried GBP 7.1bn of the GBP 22.5bn, and the stage's headcount receipt counted them as users before its conversion top-up. The prediction population is now the people not asked who have employment income; a person not asked with none is set to zero, and the receipt records how many. The conversion then tops the headcount up from the true starting count. diff --git a/changelog.d/1063-spi-band-amounts-band-aware-uprating.fixed.md b/changelog.d/1063-spi-band-amounts-band-aware-uprating.fixed.md new file mode 100644 index 000000000..087944c83 --- /dev/null +++ b/changelog.d/1063-spi-band-amounts-band-aware-uprating.fixed.md @@ -0,0 +1 @@ +The UK SPI Table 3.6 employment, self-employment and private-pension amounts by total-income band are projected from 2023-24 to the calibration year by HMRC's own band-specific growth (Income Tax Liabilities Table 2.5 total income by band, the calendar-year window the count, total-income and tax rows already use) instead of a flat national index (microcosm#1063). An amount held in a fixed nominal band moves with the band's membership as earners cross its edges; the flat average-earnings index raised the GBP 15,000-20,000 band's employment amount 10.7% while HMRC projects the band's total income down 2.3%, which left the frame 20-31% short of the three lowest employment bands at design weights and pulled the calibration into over-weighting low-paid employees. Under the band growth the three lowest employment targets move from GBP 18.7bn, 53.8bn and 190.4bn to 17.7bn, 47.5bn and 172.5bn. Dividend and savings-interest amounts keep their own indices, since those amounts move with rates rather than band membership. diff --git a/changelog.d/1063-spi-income-band-donors-channel.changed.md b/changelog.d/1063-spi-income-band-donors-channel.changed.md new file mode 100644 index 000000000..cbaf2e0b7 --- /dev/null +++ b/changelog.d/1063-spi-income-band-donors-channel.changed.md @@ -0,0 +1,7 @@ +Make the UK SPI income band donors a mass-conserving support channel (#1063 items 1, 2 and 4). `spi_income_band_donors` used to copy 120 FRS households per HMRC Table 2.5 band from GBP 200,000 and add the band's published taxpayers on top of the incoming weights: about 450,000 households of created mass, on rows starting at up to 2,992 each, recorded with no declared factor under graph policy `free`. + +Each band now seats `max(120, ceil(published taxpayers / 250))` donors at equal weights of the band's published taxpayers over its count (1,436, 244, 120 and 120 donors on the 2024-25 counts, none above 250), and the mass they carry is taken in proportion from the households of each donor's region, exactly as the SPI support channel takes its copies' mass. Total household mass and every region's mass are conserved; the mass record carries `declared_factor=1.0` under its own reason; the graph policy is `declared`; and the release-input coverage manifest gains a `spi_income_band_donors` family with `mass_conserving` semantics, so the terminal family gate demands the record. + +The seating is drawn from identity-keyed uniforms (`stable_identity_uniforms` on person ids with the declared seed and the salt `spi_income_band_donor_draw`): Efraimidis-Spirakis keys on household weight times the tape's band propensity, one carrier per household, the highest band first. The carriers' band-conditional leaf draw in `hmrc_spi_income_spine` is keyed the same way on the FRS person each carrier copies (salt `spi_income_band_donor_leaf_draw`). The stage now prepares the tape as the income stage does (its stage-1 seed and the ONS age draw), so the propensity table and the income stage's band pools read one set of donor ages. A frame too small for the plan scales the seats to at most 15% of its eligible households and the donor weights so that no region gives up more than half its mass, and records both scales; the published band masses are asserted only when both are one, and a full-sample build must be at full scale. + +The stage gate `uk_stage_spi_income_band_donors_support` checks the conserved total, the maximum donor weight, the minimum count, the funding factors and the band masses. The E8 identity receipt reconstructs the pre-donor frame from the artifact and, given the licensed SPI tape (`--spi-tab`), reruns the seating from ids in original and permuted row order against the stored donors, carriers and band weights. diff --git a/changelog.d/1063-was-wealth-tenure-coherence.changed.md b/changelog.d/1063-was-wealth-tenure-coherence.changed.md new file mode 100644 index 000000000..c650dd0d4 --- /dev/null +++ b/changelog.d/1063-was-wealth-tenure-coherence.changed.md @@ -0,0 +1 @@ +The UK `was_wealth` stage holds its wealth columns coherent with tenure and with each other (microcosm#1063). Tenure enters every chain segment as one four-way category built the same way on both sides: the donor's from `Ten1R8` and `DVPriRntR8`, the recipient's from the frame's `tenure_type`, with council and housing-association renters together as social renters. The engine's `is_renting`, which omits `RENT_FROM_HA`, is no longer a predictor, so housing-association renters stop matching owner donors. `main_residence_value` is fitted and drawn on the owner tenures only and the main-residence mortgage (`TotMortR8`) on the mortgaged tenure only, each zero by rule elsewhere; the mortgages on other property (`HMortGR8` less `TotMortR8`: buy-to-let, second homes and the rest, about a tenth of the donor's mortgage mass) are drawn on every tenure without a stratum, so an outright owner's or a renter's buy-to-let keeps its mortgage beside the property it is secured on, and `mortgage_debt` is derived as their sum. Totals are derived, not drawn: `property_wealth` is the sum of land, the main residence, other houses, buildings and a drawn remainder; `gross_financial_wealth` is savings, the cash ISA, the share-like holdings and a drawn remainder; `net_financial_wealth` is gross less consumer debt and the student loan balance. The pinned round-8 tab satisfies each identity on every row, and the cleaning step refuses a donor that does not. The chain runs eight segments with their own child seeds (the first four roots are unchanged). A new release-blocking stage-health gate, `uk_stage_was_wealth_coherence`, requires zero rows with a main-residence mortgage off a mortgaged tenure, zero with a main-residence value off an owner tenure, and zero identity violations, and bounds the share of owners without a main-residence value by the donor's own; `uk_stage_was_wealth_support` now also gates `private_pension_wealth`. The four-way tenure grouping lives in `uk_runtime.tenure_constants`, which the local targets also read. diff --git a/experiments/1063-uk-spine-followups.md b/experiments/1063-uk-spine-followups.md new file mode 100644 index 000000000..0932a52ea --- /dev/null +++ b/experiments/1063-uk-spine-followups.md @@ -0,0 +1,361 @@ +# microcosm#1063 receipts: UK spine follow-ups + +Plan: `repos/microcosm-1063-implementation-plan.md` (approved 2026-10-01) and its continuation +(2026-10-02, rulings R1–R3 below). Branch `uk-spine-followups-1063`, stacked on `uk-pensions-1069` +at 004b5c9cc (PR #1084); the PR targets that branch until #1084 merges. + +Commits, in order: + +- 80261bf11 c1: the SPI income band donors as a mass-conserving support channel (items 1, 2, 4). +- e900bb68b c2: `mass_increasing_support` retired from the coverage validator (item 3). +- cccec91bf c8: `child_benefit_take_up`, a new stage after the Pension Credit redraw (fold-in). +- 245702287 c7: salary sacrifice predicted only for people who are paid (fold-in). +- ba702f3e0 c3+c4: WAS wealth coherent with tenure and with its own totals (items 5, 6). +- 9161ad6f5: the identity receipts (E7, E8) pass on a spine from main. +- b98667cfb c5+c6: the per-band residential walk and Table 7 by claimant status (items 7, 8). +- c53686355: the two plan-2 measure exclusions renewed to 2026-11-03 (R1). +- 991134b0c c5b: `cgt_residential_split`, the residential flag carried as weight (item 7, R3); + replaces the walk of c5. +- 48eb73bf4 c9: the release-cut certifier's blockers (R2). +- c10: this note. + +## Rulings (2026-10-02) + +- R1 plan-2 measure exclusions: renewed one month (approved 2026-10-02, expires 2026-11-03), the + coverage gap staying with microcosm#868. +- R2 export surface: every declared stage output the incumbent lacks is allow-listed; only + `incapacity_benefit_reported` is dropped, at the release boundary. +- R3 residential flag: no closing rows, sub-bands or weight-keyed eligibility; the probability is + carried as weight (the incidence clone and US PUF-clone pattern), calibration decides. + +## Builds + +Every arm is a licensed local build in the detached measurement tree `repos/populace-1063-shard` +(`data/ukds/acceptance/1063-spine-followups/scripts/arm.sh [calibrate]`): the spine +with the build_930 recipe (the five NTS tabs, `--was-person-tab`, `--no-staging`), then the national +calibration (local staging only) against the Chronicle artifact 505e0e7 and the 2026-09-30 build as the +incumbent. Nothing is released from an arm. The engine is policyengine-uk at the repo's pin. + +The arms, in order: + +- C (e0b41a59b, the walk): all 27 spine gates pass. Residential gains gap £38.8m, 0.3% of the £12.2bn + target (main: £469m, 3.8%); count gap 1,428 against a bound of 10,331; the ten largest flagged stakes + carry 18.5% of residential gains (22.7% on the 2026-09-30 build); largest flagged stake £557m; the + largest stake in the pool £10.9bn, 89% of the target. The open band (£5m and over) holds 84 rows with + £47.7bn of gains, 5 flagged, achieved £1.27bn against £1.44bn expected: the deterministic bound of a + band whose heaviest row weighs 254 households and whose largest gain is £522m is vacuous (£132.6bn), + which is why the walk was replaced (R3). Table 7 by claimant status: claimants realise 87.2 / 5.3 / + 7.5 against the same targets to five decimals; non-claimants within 0.4 points of their restored + shares. Calibration: loss 0.00853 (main 0.00836), ESS 5,217 (5,198), max-to-median weight 1,115 + (1,079; bound 1,151), projection entrants 6,715 (6,711; bound 73,000), 1,090 targets. +- B (b561270df, WAS and identities): all 28 spine gates pass including the new + `uk_stage_was_wealth_coherence`: zero mortgage debt off mortgaged tenure, zero main-residence value + off owner tenures, zero owners without one, zero identity violations. (Superseded after review + round 1: Arm B's rule dropped the donor's off-tenure mortgage mass, £127bn of £1,266bn, all + other-property mortgages. Since 558325016 only the main-residence mortgage is tenure-stratified and + the other-property mortgage, `HMortGR8` less `TotMortR8`, is drawn on every tenure; on the final + build's spine the donor holds £127.3bn of mortgage debt off the mortgaged tenure, £111.6bn of it + other-property mortgages, and the recipient frame holds £273.1bn on 1,386 households, with zero + main-residence mortgage off the mortgaged tenure and zero `mortgage_debt` identity violations.) + Mortgaged + households owing more than their home: 5.4% against the donor's 2.7% (26% on the 2026-09-30 build). + Renters with property wealth: private 16.5% against the donor's 10.3%, social 4.6% against 1.1% + (10.7% before). The calibration stopped before an H5 on + `hmrc/state_pension_income_band_50_000_to_70_000@2025`, a target-fit exclusion inside its bound on + this spine, which the pensions branch retired at b19f14d98; dense loss at epoch 1,500: 0.0087. +- C2 (0ce1a5235, the residential split, on the arm C spine): all 28 spine gates pass. 2,276 + households carry a liable gainer, none more than one, so 2,276 residential arms; the residential + count and gains are identities on Table 8a in every band (solve error 1e-16 on the count, 7e-10 on + the gains, identity error 0); 106 arms weigh less than one household (smallest 0.14, largest + 1,263); the ten largest residential arms carry 9.4% of residential gains (18.5% on the walk, 22.7% + on the 2026-09-30 build); the largest residential arm stake is £176m (£557m flagged on the walk); + the largest liable stake in the pool stays £10.9bn, 89% of the target. Calibration: dense loss + 0.00909 (C 0.00853, main 0.00836); on the family fold the max-to-median weight is 610 (C 618, main + 609) and the ESS 5,207 (C 5,209, main 5,191), with 7,028 rows folded (4,752 support copies and the + 2,276 arms); the unfolded ESS fraction falls from 0.0895 to 0.0862 because the arms are extra rows. + The seam battery blocked the H5 on `uk_target_fit`: the North West £12,570–15,000 income-tax cell + (`hmrc.spi_region.income_tax_by_region_12570_15000@E12000002@2025`) at +25.3% against the 25% bound + (23.2% on main, 22.6% on C; its South East twin 22.6% and 23.3%: the regional band cells hover just + under the bound on every arm and the solver moves a point among them), and the stale + `hmrc/state_pension_income_band_50_000_to_70_000@2025` exclusion back inside the bound (24.9%; the + pensions branch retired it at b19f14d98, so the stack does not carry it). + On the calibrated weights (review item 4, `tools/uk_residential_readout.py` on the C2 solution): the + two bound national totals hold (count 202,628 against 202,630, gains £12.246bn against £12.242bn), + and within them the solver moves mass between bands: £250–500k 3,680 → 4,256 taxpayers, £500k–1m + 2,004 → 2,173 (gains £1.25bn → £1.67bn), £1–2m 730 → 552 (£0.94bn → £0.77bn), £2–5m 427 → 274 + (£1.24bn → £0.87bn), the open band 134 → 139; the bands below £250k move by under 3%. So the + by-band identities are a design-weight property; the release carries the national totals. Row + level with the arms in place: the smallest positive calibrated weight is 0.012 (an arm, 0.050; the + smallest design arm 0.144), the positive-weight median 26.6 (arms 11.8), no row at zero. +- Stack (ebff84125, the final-build candidate): the spine passed its gates in six minutes; the first + calibration refused the Chronicle artifact 505e0e7 because the stack pins the pensions feed + (facts 28b7105…, artifact 825406f). Against that feed the dense loss is 0.00818 (main 0.00836), but + the seam battery blocked the H5 on one cell with no support at all: + `dwp/uc_payment_dist/SINGLE_annual_payment_28_800_to_30_000@2025` (single, no children, monthly + award £2,400–2,500; target 1,654), initial estimate 0 against 763 at design weights on every + main-based arm and on every pensions arm. Traced: the cell is supported by one FRS household (a + disabled single council tenant in Yorkshire, design weight 760, with its incidence clone at 3.3); on + the stack its UC award is £678 a year lower, which moves it into the 27,600–28,800 band that the + uk-data#452 class already excludes. Nothing on the household's own inputs changed; its consumption, + energy, land and savings draws all moved at once, because the SPI income band donors now insert + 1,920 rows where 480 stood and every stage drawn after them that is not identity-keyed reads its + rows in a new order. The cell joins the measure exclusions beside its sibling (approved 2026-10-02, + expiring 2026-11-26 with the class, for her signature). +- Stack 2 (558325016, after review round 1): the spine passes; dense loss 0.00769; the battery blocked + on two further cells. `dwp/uc_payment_dist/COUPLE_NO_CHILDREN_annual_payment_22_800_to_24_000` + has two supporting households on every main-based spine (design weights 1,025 and 1,999 against + 2,761) and both awards moved out of the band when the stages drawn after the reworked WAS chain read + their rows in a new order; it joins the uk-data#452 measure exclusions. `hmrc.spi_region. + income_tax_by_region_12570_15000@E12000008@2025` (South East) sits on its target at design weights + is 11% short at design weights (69.6m against 78.3m; 62.4m on main, 58.3m on stack 1) and the + solver pulls it to +32.0%. The pull comes from the national employment-income bands: at design + weights the £12,570–15,000 and £15,000–20,000 employment-income bands are 20% and 31% short (the + earners gap the 2026-09-30 build found), the solver fills them by raising the weights of low-paid + employees everywhere (three-fold on the 200 households that carry most of this cell's gain), and + the South East income-tax cell, carried by the same households, overshoots while its total-income + sibling lands at +11.2% (main pulls the same cell from −20% to +22.6%). The cell is deferred four + weeks in the target-fit register (it was deferred on 2026-09-18 and retired on microcosm#1012 at + +24.5%); the fix is on the employment-income side, not on the draws. Separately, every regional + £12,570–15,000 income-tax cell moves 10–30% at design weights between spines that share the same + incomes (North West 63.0m, 44.5m, 52.0m; East 41.3m, 53.5m, 53.2m) because the salary-sacrifice and + pension draws after the SPI chain are positional; identity-keying those draws would make the + design values stable between spines but would not move where the solver lands. +- Stack 3 (2fe7d8bbb, the Table 3.6 amounts projected by HMRC's band growth): all 7 seam gates pass + and the H5 is written; dense loss 0.00713 (main 0.00836). The three lowest employment rows fit at + −3.9%, −11.9% and −10.0% (stack 2: −8.9%, −23.1%, −18.9%; main −9.9%, −23.2%, −19.6%). The South + East £12,570–15,000 income-tax cell lands at +30.3% under its deferral; the North West twin at + +14.4%. Folded weights: max-to-median 1,028 (main 609, stack 1 710, stack 2 1,116; bound 1,151), + ESS 4,953 (main 5,191, stack 1 5,222), median positive weight 40.0 (main 55.8), heaviest household + 41,085 (an East Midlands household in its thirties carrying 10–12% of that region's 30–39 + population, £20–30k taxpayers and band-B council tax rows; on stack 1 the heaviest was an 85–89 + Pension Credit unit at 29,690). The rise in the fold between stack 1 and stack 2 came with the WAS + mortgage rework and the first thin-cell exclusion; the margin to the bound is now thin and belongs + in the review. +- Final build (2fe7d8bbb): running at the time of writing; the readings follow. + +The South East cell, with the count side read: the frame has 0.278m South East taxpayers with total +income in £12,570–15,000 at design weights against HMRC's projected 0.364m (nationally 2.31m against +2.97m; North West 0.240m against 0.333m, London 0.235m against 0.340m, East 0.256m against 0.273m), +and the ones it has pay £250 of income tax each against HMRC's £215. The bound count rows make the +solver scale these households up by a third to hit the count (final error 0.000), the total-income +sibling overshoots to +11% and the tax cell to +30%. So root 2 is a deficit of taxpayers just above +the frozen allowance, concentrated in the South East, London and the North West, not an employment +problem: the fix is the lowest band's support in the frame (the SPI support channel's coverage of +that band, which the pension-age prior share of 0.2 reduced, and the FRS's small incomes around the +allowance), a spine change for the follow-up. + +Child Benefit children in payment (review item 5): the trial's 11.1m is at design weights on the +1 October spine, where the eligible-child base is 14.04m; the claim rate (86.7%) and the opted-out +family share (9.0%) match HMRC, and the opted-out families carry 1.82 children each, so claimed +children are 12.17m and 1.09m of them are in opted-out families. On the 2026-09-30 build's calibrated +weights the eligible base is 14.79m, which at the same rates gives 11.7m in payment against HMRC's +11.73m: the gap is the design-weight child base, not the rates or the opt-out ages. + +Child Benefit (c8, trial on the 1 October spine at design weights): 86.7% of eligible children claimed +for (HMRC 86.6%; the frame's age mix implies 87.1%), 9.0% of claiming families opted out, all from +families charged the whole benefit, 11.1m children in payment against 13.5m before. + +## The certifier rehearsal (R5, 2026-09-30 build, 15 passed / 5 failed of 20) + +- `uk_export_surface`: label `microcosm_uk_2024`; `household.household_weight` reported missing and + every id column reported as an extra (the certifier read every frame column); 49 columns outside the + enhanced-FRS surface without an allow-list entry; `incapacity_benefit_reported` to be dropped. +- `uk_degenerate_release_surface`: `incapacity_benefit_reported` and `is_in_approved_training` + all-zero. FRS 2024-25 (SN 9563) carries `TRAIN2` (codes 1–8 and 10 are schemes, 9 none of these) and + `TRAINEE` in place of `TRAIN`; 28 adults report a scheme, so the column carries signal once read. +- `uk_nonnegative_columns`: `housing_water_and_electricity_consumption` 250 rows below zero (minimum + −14,165), inherited from LCFS donor rows whose diary nets refunds below zero; the support clip took + its floor from the donor's realised range. +- `uk_input_mass_parity` against the enhanced FRS: `adult_ema` +786% (38.9m vs 4.4m), + `dfe_education_spending` +187,061% (£98.8bn vs £52.8m). +- `uk_qrf_tail_concentration`: `charitable_investment_gifts` top-100 rows carry 100% of the mass over + 168 carriers. + +c9 addresses each at its source (see the commit); the certifier is to be re-run on the final build. + +## The employment-income band shortfall (ruling 2026-10-02: fix, do not defer) + +The three lowest `hmrc/employment_income_income_band_*` rows are SPI 2023-24 Table 3.6 amounts: the +employment income received by taxpayers whose total income falls in the band, projected to 2025. The +frame's taxpayer counts and total income in those bands sit within a few percent of HMRC; what was +short was the share of the band's income that is pay, 20–31% at design weights. Two roots: + +- The registry projected the Table 3.6 amount rows with flat national indices (average earnings for + employment, mixed income for self-employment, the pension index for private pensions) while the + count, total-income and tax rows by band use HMRC's own band-specific growth (Income Tax + Liabilities Table 2.5). An amount held in a fixed nominal band moves with the band's membership: + HMRC projects the £12,570–15,000 band's total income +5.0% from 2023-24 to calendar 2025, the + £15,000–20,000 band's −2.3% and the £20,000–30,000 band's +0.3%, against the flat +10.7%. The + 39 amount references now use the band growth; the three lowest employment targets move from + £18.7bn, £53.8bn and £190.4bn to £17.7bn, £47.5bn and £172.5bn, and the frame's design-weight gaps + from −19.8%, −31.2% and −25.1% to −15.5%, −22.1% and −17.4%. Dividend and savings-interest amounts + keep their own indices (they move with rates, not membership). The SPI tape re-banded under the + engine's own indices agrees with HMRC's direction in every band. +- The remaining gap is the frame's composition of the low bands. Banding the frame on core income + only (pay, profits, private pensions, State Pension), its earners in the £12,570–15,000 band are + 1.36m, HMRC's 1.36m exactly, with £17.8bn of pay against the band-aware £17.7bn; adding the + non-employment leaves (savings interest, dividends, property, miscellaneous) lifts 0.14m of them and + £2.2bn of pay out of the band, and the engine's broader banding concept a further £0.6bn. In the + £15,000–20,000 band the core-only comparison still leaves pay 8.6% and earners 14% short, with State + Pensioners 7% above HMRC's count. The places to look, in order: the SPI stage-2 rewrite of dividend, + property and savings income onto FRS rows (`hmrc_spi_income_spine`; on the tape the low earners who + sit in these bands carry about £350 of such income, the frame's FRS-channel low earners show 7% with + dividends averaging £17,500), the total-income concept the band measure uses against SPI's, and the + SPI channel's age mix in the low bands (58% of the band on main, 43% on the stack). Spine work, a + follow-up. + +The South East £12,570–15,000 income-tax cell's +32% is this pull: the solver raised the weights of +the low-paid households that carry it (three-fold on the 200 that carry most of its gain) to fill the +under-projected employment rows; its own target is already band-aware. + +## Review round 1 (Vahid, 2026-10-02) + +- The five internal disability carriers leave the export allow-list and are dropped at the release + boundary with `incapacity_benefit_reported`; the input-mass evidence pin and the gate digests follow + the c9 register entries. +- WAS mortgages: only the main-residence mortgage (`TotMortR8`) is tenure-stratified; the mortgages + on other property (`HMortGR8` less `TotMortR8`) are drawn on every tenure without a stratum, and + `mortgage_debt` is derived as their sum, so a renter's or an outright owner's buy-to-let keeps its + mortgage beside the property. The chain runs eight segments; the coherence gate holds the + main-residence mortgage off the mortgaged tenure at zero and the mortgage identity on every row. +- The split's docstring and notes say what calibration binds: the two national residential rows. +- The plan-2 renewal stays on this PR (her ruling). + +## Residential split (c5b) + +Each liable gainer's residential probability `p` (the logistic solved to Table 8a's individuals-basis +count and gains, with the stock shift) is carried as weight: a household with one liable gainer becomes +a residential arm at `p·w` beside the incumbent at `(1 − p)·w`; with `k` gainers, `2^k` arms at +product weights (`k > 3` refused). At design weights the residential count and gains are identities on +Table 8a in every gain band (no draw, no seed, no error bound); the asset-type stage types the arms and +keeps the BADR claims and the Table 7 fit. The £522m copy (weight 20.8, `p` 0.2%) becomes a residential +arm at 0.04 households beside a non-residential arm at 20.76. Family folds, the geography identity +kernel, the export surface, national sampling, the student-loans receipt and the E8 identity receipt +carry the new layer. + +## Final build (2fe7d8bbb on main fe4e38dfa, 2026-10-02) + +Spine `spine-1063-2fe7d8bbb.h5` (sha256 ccda40bd…), national candidate `runs/uk-national-1063-2fe7d8bbb/ +microcosm_uk_2024_25.h5` (sha256 df3ca7cc…, staged), 7/7 seam gates, loss 0.00713 (1,090 targets; the +South East £12,570–15,000 income-tax cell +30.3% under its reviewed deferral). Evaluation against the +2026-09-30 incumbent PASSED: loss 0.00817 vs 0.327, 1,020 wins vs 30, 118 pruned. The release-cut +certifier stopped at its first part, `uk_ledger_compile_parity_incumbent_2025`: the signed receipt still +pinned the 39 Table 3.6 amount references at their flat-index values; re-signed against the pinned +ledger (2af5ad3e5, only those 39 rows move). E7 passed. E8 failed with three symptoms, all diagnosed on +the built spine: + +- `anchor_recompute` 55,622 households (max 286) and `support_split_selection_stored` extra 1 / + missing 2: the recompute banded the carriers on post-sacrifice pay. Since microcosm#1069 c9 (on main + as #1084) `salary_sacrifice` lowers a converted record's `employment_income` and zeroes its employee + contribution in place, after the support split (stage 30) and the anchor (33) banded on them; + 2,797 clone carriers carry salary sacrifice, 82 of them moved down a Table 3 band, so the anchor + derived loss 153,816 / sub-exempt 40,723 against the stage's 153,919 / 40,709 at the same liable mass, + and band 100,000 lost households 13421 and 2000006588 to band 0 while band 125,140 gained 6477. + The 30 September main spine predates the pay drop, which is why E8 was exact there. Fix: the stage + keeps a converted record's pre-conversion pay on `salary_sacrifice_pre_conversion_pay` (zero + elsewhere; dropped at the release boundary), and every receipt of a stage before `salary_sacrifice` + reverses the conversion from it (`_pre_salary_sacrifice_frame`). +- `residential_split_permutation`: the stored arms were reproduced to 3.4e-13, but two person orders + of the same frame differed by 1e-13 on 2,999 arm weights — the logistic solve's bisection sums are + pairwise float sums. Fix: the split walks the liable gainers in `person_id` order (the identity on + the spine's sorted tables) and breaks a tie for the exact-total correction row by household id; the + order test is now bitwise under a reversed and a random person order. + +The weight-shape warning stands (folded max-to-median 1,028 vs main's 609; bound 1,151). Both fixes +change the spine (a new carrier column; the arms bitwise unchanged), so the final build and the +certifier run again from the fixed head. + +## Final build, second run (64c460b66, 2026-10-03) + +Spine `spine-1063-64c460b66.h5` (sha256 b7202ef6…): bitwise identical to the first run's in every shared +column and every household weight; the only difference is the new carrier, set on exactly the 7,843 +converted records the stage reports. Candidate `runs/uk-national-1063-64c460b66/microcosm_uk_2024_25.h5` +(sha256 aa31bdf6…): the same calibration (loss 0.00713, 7/7 seam gates) and the same evaluation +(0.00817 vs 0.327; 1,020 wins vs 30). E7 and E8 pass; E8 reports the restored pre-sacrifice pay, anchor +targets equal to the stage's and the residential arms identical under permutation. The certifier +reaches all 20 gates: 17 pass, three fail. + +- `uk_nonnegative_columns` and `uk_qrf_tail_concentration` looked for declared stage outputs on the + release candidate, where the export now drops them (the five disability carriers and the + pre-conversion pay carrier). Both bindings now read an export-dropped declared column from the + spine frame the certifier supplies, at the spine's weights; without that frame the absence still + fails. +- `uk_input_mass_parity` against the uk-data 1.56.16 enhanced FRS (±452%): `access_fund` +535%, + `jsa_income_reported` +1,058%, `working_tax_credit_reported` +747%, and a stale `adult_ema` + exclusion (+74% now; entry retired). Measured on the candidate against the spine's design weights: + 98% of the access-fund mass is one FRS household (sernum 14443, South East) and its two CGT copies, + whose 19-year-old reports an access fund that the FRS weeklyises to £1,035.62 (`ACCSSAMT`, period + code 5) and the spine annualises to £54,037, eleven times the next largest value; the calibration + takes the household to its 10x bound (design 896, calibrated 8,927). The JSA and tax-credit masses sit + in the SPI synthetic channel (76 and 274 of the nonzero rows), where the stage-2 QRF imputes these + legacy-benefit leaves onto taxpayer rows at FRS rates, and in a handful of rows at or near the + bound (ratios 6–10). None of this is new: at design weights the stack spine carries less JSA mass + than the 30 September main spine (3.1e8 vs 4.7e8) and similar tax-credit and access-fund masses, + and the 30 September main candidate sat at +421% and +428% on the same two columns, just under the + tolerance, with the same 0.5% of households at the 10x bound. The three findings are the + calibration's weight concentration and the SPI channel's legacy-benefit leaves, not the stack's + stages; the rulings on them (an FRS outlier fence on `access_fund`, the SPI channel's treatment of + `jsa_income_reported` and `working_tax_credit_reported`, or reviewed exclusions) are hers. + +Certifier rerun from the gate-fix head (1e58ecb3a) on the same artifacts, 2026-10-03 00:23Z: 19 of 20 pass; +`uk_input_mass_parity` alone fails, on the three columns above (`final/certify2.err`, +`microcosm_uk_2024_25.release_cut_gates.rerun.json`). + +The access-fund record, checked against the FRS 2024-25 variable listing (review round 2, item 7): +`ACCSSPD` 5 is "Calendar month", and £1,035.62 a week is exactly £4,500 a calendar month +(1,035.616438 × 365/7 ÷ 12). A £4,500 access fund a month is not a plausible award; £4,500 a year sits +beside the next FRS values (£4,803, £4,654, £3,602). So the record is an annual or termly award under +the wrong period code, a data fix for the spine's period handling rather than an outlier fence, and +the household's weight is the calibration's: on the 30 September main build the same record sat at +design weight (ratio 1.00) because its incidence clone drew £2,355 of gains, under the exempt amount, +and the anchor returned the clone's mass; on this build the Table 3 redraw's positional stream, shifted +by the 1,920 inserted donor households, gives the clone £3,776, a liable South East gainer with a +residential arm, and the calibration takes the household and both copies to the 10x bound. + +Ruling 2026-10-04 (in session): the deferrals needed to build and publish the national dataset are +signed. `input_mass_reviewed_exclusions.json` gains `access_fund` (expiring 2027-01-04, with the +period rule owed), `jsa_income_reported` and `working_tax_credit_reported` (2027-04-04), each entry +carrying the diagnosis above; the certifier reruns on the same artifacts from the signed head. + +Certifier rerun from the signed head (427bec73f), 2026-10-04 16:23Z: all 20 gates pass. The run then +stopped composing the certificate: "spine: signature does not authenticate under the release signing +key". Not the key (every report, this run's own included, carries the same key fingerprint) but the +canonical bytes: the gate battery signs its reports over `json.dumps(sort_keys, compact)` while the +certifier verified the parts, and signed the certificate, over the logbook's #628 form, which renders +`1.0` as `1` and `1e-05` as `0.00001`. The two agree on the float-free synthetic fixtures and on no real +report, so the first certification to reach composition was the first to see it. The certifier now +verifies and signs over the battery's canonical bytes (`canonical_report_bytes`, the form the data +contract's verifier already uses), with a regression test that puts such floats in a part. A latent +defect on main, reached here for the first time. + +The engine-free lane's remaining failure, `test_uk_national_graph.py`'s seam-scope test, is the +committed target-fit register against the test's fixed review date: the South East deferral is +approved 2026-10-02 and the graph was reviewed at 2026-09-29, so the gate refused it as not yet in +force. Path-tiered CI had not selected the file before round 2 widened the diff. The test now reviews +at the latest approval across the committed registers, inside every live window. + +Certifier rerun from 5720b80dd on the same artifacts, 2026-10-04 16:55Z: all 20 gates pass and the signed +certification composes (`shippable: true`; spine, calibration-seam and release-cut parts, 31 entries; +candidate aa31bdf6…): `final/microcosm_uk_2024_25.release_certification.rerun.json`. + +## Still owed + +- The access-fund period rule in the spine (`ACCSSPD` 5 on sernum 14443), owed before the + `access_fund` exclusion expires on 2027-01-04. +- Follow-ups outside this PR: key the Table 3 redraw's within-band draws and the other positional + draws by identity, so an upstream row insertion no longer moves a thin UC cell or an incidence + clone across the exempt amount (review round 2, item 8); the SPI channel's treatment of the + legacy-benefit leaves (`jsa_income_reported`, `working_tax_credit_reported`); running + `salary_sacrifice` before the capital-gains stages, which would retire the pre-conversion pay + carrier. +- Her calls: Child Benefit opt-out pool order (fully charged first, then the taper, as built; or one + pool over every charged family); `other_residential_property_value` (WAS `DVHseVal`) excludes + buy-to-let (`DVBltVal`), which sits in the drawn remainder; the Table 8a rows stay fenced + (promotion to bound targets is the #970 leverage question, after the final build). +- Findings outside this PR: SPI-channel rows receive `child_benefit_reported` from the stage-2 QRF + without regard to children (1.13m weighted reporter units with no eligible child); the frame has + 0.67m reporter families with an adult above the £60,000 charge start against HMRC's 0.31m–0.44m + liable individuals; the E8 donor recompute takes the licensed tape (`--spi-tab`) rather than storing + the propensity table in the receipt (small cells). +- policyengine-uk: #1996 (VAT) filed; `is_renting` omitting `RENT_FROM_HA` and the engine ignoring + `child_benefit_opts_out` to file on her go. diff --git a/packages/microcosm-build/src/microcosm/build/gate_battery.py b/packages/microcosm-build/src/microcosm/build/gate_battery.py index c97070878..e476787a5 100644 --- a/packages/microcosm-build/src/microcosm/build/gate_battery.py +++ b/packages/microcosm-build/src/microcosm/build/gate_battery.py @@ -73,6 +73,7 @@ "GATE_BATTERY_PRODUCER", "GATE_BATTERY_SCHEMA_VERSION", "GATE_BATTERY_SIGNATURE_ALGORITHM", + "canonical_report_bytes", "GateBatteryBlockedError", "GateBatteryRun", "GateBinding", @@ -121,6 +122,19 @@ def _canonical_json_bytes(value: object) -> bytes: ).encode("utf-8") +def canonical_report_bytes(value: object) -> bytes: + """The bytes a gate-battery report is signed over. + + Sorted keys, no whitespace, Python's JSON number rendering. Every + verifier of a battery signature must canonicalise with this function + and no other: the logbook's #628 form renders ``1.0`` as ``1`` and + ``1e-05`` as ``0.00001``, so it agrees with this one only on payloads + without such floats. + """ + + return _canonical_json_bytes(value) + + def _canonical_sha256(value: object) -> str: return hashlib.sha256(_canonical_json_bytes(value)).hexdigest() diff --git a/packages/microcosm-build/src/microcosm/build/source_manifest.py b/packages/microcosm-build/src/microcosm/build/source_manifest.py index be741c594..4ee2065fc 100644 --- a/packages/microcosm-build/src/microcosm/build/source_manifest.py +++ b/packages/microcosm-build/src/microcosm/build/source_manifest.py @@ -51,10 +51,11 @@ "assign_bus_pass_eligibility", "assign_clipped_normal", "assign_component_take_up_residual", + "assign_family_claims_by_child_age", + "assign_opt_out_by_charge_income", "assign_period_constant", "assign_student_loan_plan_cohorts", "assign_main_asset_type", - "assign_residential_property_flag", "assign_trips_from_band_means", "assign_uniform_draw", "aggregate_person_to_benunit", @@ -168,6 +169,7 @@ "replace_sentinels", "split_component_by_share", "split_top_wealth_support_households", + "split_liable_gainers_by_residential_probability", "stack_income_band_donor_households", "stack_zero_weight_donors", "strict_read_private_table", diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json index eaf7c3993..57c738af1 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/sources.schema.json @@ -5559,70 +5559,6 @@ "source_vintage" ] }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "basis_assumption": { - "type": "string" - }, - "count_target": { - "type": "string" - }, - "gains_target": { - "type": "string" - }, - "kind": { - "const": "assign_residential_property_flag" - }, - "model": { - "type": "string" - }, - "output_column": { - "type": "string" - }, - "output_semantics": { - "type": "string" - }, - "parameter_solver": { - "type": "string" - }, - "population": { - "type": "string" - }, - "realization": { - "type": "string" - }, - "seed": { - "type": "number" - }, - "seed_mixing": { - "type": "string" - }, - "stock_log_odds": { - "type": "number" - }, - "stock_signal": { - "type": "string" - } - }, - "required": [ - "basis_assumption", - "count_target", - "gains_target", - "kind", - "model", - "output_column", - "output_semantics", - "parameter_solver", - "population", - "realization", - "seed", - "seed_mixing", - "stock_log_odds", - "stock_signal" - ] - }, { "type": "object", "additionalProperties": false, @@ -5710,6 +5646,9 @@ "type": "object", "additionalProperties": false, "properties": { + "basis_assumption": { + "type": "string" + }, "categories": { "type": "array", "items": { @@ -5725,6 +5664,9 @@ "claimant_restriction": { "type": "string" }, + "claimant_share_targets": { + "type": "string" + }, "diagnostic_only": { "const": true }, @@ -5737,6 +5679,9 @@ "log_sigma": { "type": "number" }, + "non_claimant_share_targets": { + "type": "string" + }, "none_semantics": { "type": "string" }, @@ -5749,12 +5694,6 @@ "realization": { "type": "string" }, - "seed": { - "type": "number" - }, - "seed_mixing": { - "type": "string" - }, "share_targets": { "type": "string" }, @@ -5787,19 +5726,20 @@ } }, "required": [ + "basis_assumption", "categories", "claimant_categories", "claimant_restriction", + "claimant_share_targets", "diagnostic_only", "kernel", "kind", "log_sigma", + "non_claimant_share_targets", "none_semantics", "output_column", "population", "realization", - "seed", - "seed_mixing", "share_targets", "stock_signals", "stock_type_floor", @@ -6492,6 +6432,130 @@ } } }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "output", + "draw", + "seed", + "anchor", + "eligibility", + "family_key", + "resource", + "rate_measure", + "rate_period", + "weight_mapping", + "rule" + ], + "properties": { + "kind": { + "const": "assign_family_claims_by_child_age" + }, + "output": { + "type": "string" + }, + "draw": { + "type": "string" + }, + "seed": { + "type": "integer" + }, + "anchor": { + "type": "string" + }, + "eligibility": { + "type": "string" + }, + "family_key": { + "type": "string" + }, + "resource": { + "type": "string" + }, + "rate_measure": { + "type": "string" + }, + "rate_period": { + "type": "string" + }, + "weight_mapping": { + "type": "string" + }, + "rule": { + "type": "string" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "output", + "draw", + "seed", + "resource", + "numerator_measure", + "denominator_measure", + "caseload_period", + "charge_income", + "charge_parameters", + "pools", + "weight_mapping", + "rule", + "payment_rule" + ], + "properties": { + "kind": { + "const": "assign_opt_out_by_charge_income" + }, + "output": { + "type": "string" + }, + "draw": { + "type": "string" + }, + "seed": { + "type": "integer" + }, + "resource": { + "type": "string" + }, + "numerator_measure": { + "type": "string" + }, + "denominator_measure": { + "type": "string" + }, + "caseload_period": { + "type": "string" + }, + "charge_income": { + "type": "string" + }, + "charge_parameters": { + "type": "string" + }, + "pools": { + "type": "array", + "minItems": 1, + "items": { + "type": "string" + } + }, + "weight_mapping": { + "type": "string" + }, + "rule": { + "type": "string" + }, + "payment_rule": { + "type": "string" + } + } + }, { "type": "object", "additionalProperties": false, @@ -6939,6 +7003,9 @@ "type": "string" } }, + "floor": { + "type": "number" + }, "logged_dropna_row_count": { "type": "boolean" }, @@ -8039,6 +8106,8 @@ "kind", "predictors", "chain_order", + "stratified_targets", + "derived_totals", "seed" ], "properties": { @@ -8069,12 +8138,32 @@ "type": "string" } }, - "debt_segment_predictors": { + "stratified_targets": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "internal_components": { "type": "array", "items": { "type": "string" } }, + "derived_totals": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "array", + "items": { + "type": "string" + } + } + }, "region_remap": { "type": "object", "minProperties": 1, @@ -8542,6 +8631,106 @@ } } }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "population", + "model", + "stock_log_odds", + "stock_signal", + "parameter_solver", + "count_target", + "gains_target", + "basis_assumption", + "realization", + "maximum_liable_gainers_per_household", + "gainer_order", + "id_remapping", + "flag_column", + "index_column", + "probability_column", + "output_column", + "output_semantics", + "values", + "weight_kind_out", + "conservation", + "declared_factor", + "reason" + ], + "properties": { + "kind": { + "const": "split_liable_gainers_by_residential_probability" + }, + "population": { + "type": "string" + }, + "model": { + "type": "string" + }, + "stock_log_odds": { + "type": "number" + }, + "stock_signal": { + "type": "string" + }, + "parameter_solver": { + "type": "string" + }, + "count_target": { + "type": "string" + }, + "gains_target": { + "type": "string" + }, + "basis_assumption": { + "type": "string" + }, + "realization": { + "type": "string" + }, + "maximum_liable_gainers_per_household": { + "type": "integer" + }, + "gainer_order": { + "type": "string" + }, + "id_remapping": { + "type": "string" + }, + "flag_column": { + "type": "string" + }, + "index_column": { + "type": "string" + }, + "probability_column": { + "type": "string" + }, + "output_column": { + "type": "string" + }, + "output_semantics": { + "type": "string" + }, + "values": { + "type": "string" + }, + "weight_kind_out": { + "type": "string" + }, + "conservation": { + "const": "exact_total" + }, + "declared_factor": { + "const": 1.0 + }, + "reason": { + "type": "string" + } + } + }, { "type": "object", "additionalProperties": false, @@ -8551,21 +8740,33 @@ "taxpayer_count_measure", "taxpayer_count_period", "band_lower_bounds", - "donors_per_band", + "minimum_donors_per_band", + "maximum_donor_weight", + "count_rule", + "weight_rule", "expected_band_count", - "expected_donor_count", "candidate_population", "candidate_order", "draw", "propensity", + "tape_preparation", + "tape_preparation_seed", "seed", + "salt", "clone_index", "flag_column", "band_column", "carrier_column", - "initial_weight", + "funding", + "funding_strata", + "maximum_reallocation_share", + "maximum_donor_household_share", + "seating_scale", + "mass_scale", "never_zero_weight", "weight_kind_out", + "conservation", + "declared_factor", "reason" ], "properties": { @@ -8588,13 +8789,21 @@ "type": "integer" } }, - "donors_per_band": { - "type": "integer" + "minimum_donors_per_band": { + "type": "integer", + "minimum": 1 }, - "expected_band_count": { - "type": "integer" + "maximum_donor_weight": { + "type": "number", + "exclusiveMinimum": 0 }, - "expected_donor_count": { + "count_rule": { + "type": "string" + }, + "weight_rule": { + "type": "string" + }, + "expected_band_count": { "type": "integer" }, "candidate_population": { @@ -8609,9 +8818,18 @@ "propensity": { "type": "string" }, + "tape_preparation": { + "type": "string" + }, + "tape_preparation_seed": { + "type": "integer" + }, "seed": { "type": "integer" }, + "salt": { + "type": "string" + }, "clone_index": { "type": "integer" }, @@ -8624,7 +8842,30 @@ "carrier_column": { "type": "string" }, - "initial_weight": { + "funding": { + "type": "string" + }, + "funding_strata": { + "type": "array", + "minItems": 1, + "items": { + "type": "string" + } + }, + "maximum_reallocation_share": { + "type": "number", + "exclusiveMinimum": 0, + "exclusiveMaximum": 1 + }, + "maximum_donor_household_share": { + "type": "number", + "exclusiveMinimum": 0, + "exclusiveMaximum": 1 + }, + "seating_scale": { + "type": "string" + }, + "mass_scale": { "type": "string" }, "never_zero_weight": { @@ -8633,6 +8874,12 @@ "weight_kind_out": { "type": "string" }, + "conservation": { + "const": "exact_total" + }, + "declared_factor": { + "const": 1.0 + }, "reason": { "type": "string" } @@ -8715,6 +8962,9 @@ "weighting", "with_replacement", "outputs", + "draw", + "draw_key", + "salt", "seed" ], "properties": { @@ -8746,6 +8996,15 @@ "with_replacement": { "type": "boolean" }, + "draw": { + "type": "string" + }, + "draw_key": { + "type": "string" + }, + "salt": { + "type": "string" + }, "outputs": { "type": "string" }, diff --git a/packages/microcosm-build/src/microcosm/build/uk/calibration_measure_exclusions.json b/packages/microcosm-build/src/microcosm/build/uk/calibration_measure_exclusions.json index 0842fc3c4..4db168e2b 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/calibration_measure_exclusions.json +++ b/packages/microcosm-build/src/microcosm/build/uk/calibration_measure_exclusions.json @@ -154,6 +154,24 @@ "approved_on": "2026-08-26", "expires_on": "2026-11-26" }, + { + "name": "dwp/uc_payment_dist/SINGLE_annual_payment_28_800_to_30_000", + "reason": "Single-row support: one FRS household (a disabled single council tenant, design weight 760 against the published 1,654) is the only row whose UC award falls in this band on every main-based spine, so the cell is a scaling of one row rather than a composition the spine carries. On the #1063 stack its award moved GBP 678 lower, out of the band and into the 27,600-28,800 band that is already excluded for the same class (uk-data#452), when the stages drawn after the SPI income band donors read their rows in a new order; the cell then sits at -100% with final estimate 0.0. The payment distribution needs element/taper composition support beyond the would_claim_uc take-up lever, as the sibling entries say.", + "tracking": "uk-data#452", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (the stack arm of 2026-10-02 on the review of #1089; same class as the 2026-08-26 microcosm#757 dispositions)", + "approved_on": "2026-10-02", + "expires_on": "2026-11-26" + }, + { + "name": "dwp/uc_payment_dist/COUPLE_NO_CHILDREN_annual_payment_22_800_to_24_000", + "reason": "Two-household support: two FRS households (design weights 1,025 and 1,999 against the published 2,761, with their incidence clones) are the only rows whose UC award falls in this band on every main-based spine. On the #1063 stack both awards moved out of the band when the stages drawn after the WAS wealth chain read their rows in a new order, and the cell sits at -100% with final estimate 0.0; the same class as its 2026-08-26 siblings (uk-data#452): the payment distribution needs element/taper composition support beyond the would_claim_uc take-up lever.", + "tracking": "uk-data#452", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (the stack arm of 2026-10-02 on the review of #1089; same class as the 2026-08-26 microcosm#757 dispositions)", + "approved_on": "2026-10-02", + "expires_on": "2026-11-26" + }, { "name": "dwp/uc_payment_dist/COUPLE_NO_CHILDREN_annual_payment_26_400_to_27_600", "reason": "One-sided undershoot: the annual-payment band composition inside the UC caseload is unreachable under the spine's current UC support (the #623-carried catalogue); every cell in this class undershoots on the rebind packet and two sit at -100% with final estimate 0.0. The payment distribution needs element/taper composition support beyond the would_claim_uc take-up lever.", @@ -339,18 +357,18 @@ "reason": "The spine carries 3.4M plan-2 borrowers liable against the SLC 9.7M (-65% at initialization on spine-m; -52% after the solve): student-loan borrower coverage in the frame. Unreachable by reweighting within the stretch bound: a frame-side gap, not a weight question; while bound it pulls the joint solve toward weight concentration (housing benefit alone triples the max/median ratio under grain_equal). One-month window: correct the spine or re-adjudicate.", "tracking": "microcosm#868", "approved_by": "juaristi22", - "adjudication": "microcosm#762 (A16, ruling 2026-09-03; receipts R8-R10 in experiments/762-uk-rowwise-candidate-receipts.md)", - "approved_on": "2026-09-03", - "expires_on": "2026-10-03" + "adjudication": "microcosm#762 (A16, ruling 2026-09-03; receipts R8-R10 in experiments/762-uk-rowwise-candidate-receipts.md); renewed one month on 2026-10-02 by juaristi22 in session (microcosm#1063 stack): the plan-2 borrower coverage gap stays with microcosm#868 and is not corrected by the spine follow-ups", + "approved_on": "2026-10-02", + "expires_on": "2026-11-03" }, { "name": "slc.borrowers.plan_2_above_threshold", "reason": "The spine carries 1.4M plan-2 borrowers above threshold against the SLC 4.5M (-69% at initialization on spine-m; -43% after the solve): student-loan borrower coverage in the frame. Unreachable by reweighting within the stretch bound: a frame-side gap, not a weight question; while bound it pulls the joint solve toward weight concentration (housing benefit alone triples the max/median ratio under grain_equal). One-month window: correct the spine or re-adjudicate.", "tracking": "microcosm#868", "approved_by": "juaristi22", - "adjudication": "microcosm#762 (A16, ruling 2026-09-03; receipts R8-R10 in experiments/762-uk-rowwise-candidate-receipts.md)", - "approved_on": "2026-09-03", - "expires_on": "2026-10-03" + "adjudication": "microcosm#762 (A16, ruling 2026-09-03; receipts R8-R10 in experiments/762-uk-rowwise-candidate-receipts.md); renewed one month on 2026-10-02 by juaristi22 in session (microcosm#1063 stack): the plan-2 borrower coverage gap stays with microcosm#868 and is not corrected by the spine follow-ups", + "approved_on": "2026-10-02", + "expires_on": "2026-11-03" }, { "name": "dwp.jsa_claimants", diff --git a/packages/microcosm-build/src/microcosm/build/uk/country_package.json b/packages/microcosm-build/src/microcosm/build/uk/country_package.json index 8a7199439..2e9ab85fa 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/country_package.json +++ b/packages/microcosm-build/src/microcosm/build/uk/country_package.json @@ -162,6 +162,11 @@ "kind": "legacy_json", "schema_id": "legacy_json" }, + { + "path": "hmrc_child_benefit_statistics.json", + "kind": "legacy_json", + "schema_id": "legacy_json" + }, { "path": "lcfs_consumption_support_bounds.json", "kind": "legacy_json", diff --git a/packages/microcosm-build/src/microcosm/build/uk/gates.json b/packages/microcosm-build/src/microcosm/build/uk/gates.json index 4cb1738dd..aeb01125e 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/gates.json +++ b/packages/microcosm-build/src/microcosm/build/uk/gates.json @@ -99,6 +99,7 @@ "num_vehicles", "other_residential_property_value", "owned_land", + "private_pension_wealth", "property_wealth", "savings", "stocks_and_shares_isa", @@ -116,6 +117,7 @@ "num_vehicles": 0, "other_residential_property_value": 0, "owned_land": 0, + "private_pension_wealth": 0, "property_wealth": 0, "savings": 0, "stocks_and_shares_isa": 0, @@ -133,13 +135,27 @@ "num_vehicles": 0, "other_residential_property_value": 0, "owned_land": 0, + "private_pension_wealth": 0, "property_wealth": 0, "savings": 0, "stocks_and_shares_isa": 0, "student_loan_balance": 0 } }, - "notes": "Stage-time support-clip health gate over the WAS wealth receipt; raw FRS mapping stages have no imputation semantics and intentionally have no empty health gate. Allowances pinned at the receipted spine-e baseline (2026-08-26): zero rows clipped on every non-exempt column, so any clipping is a real signal against a clean baseline; a missing allowance fails closed rather than skipping the comparison." + "notes": "Stage-time support-clip health gate over the WAS wealth receipt; raw FRS mapping stages have no imputation semantics and intentionally have no empty health gate. Allowances pinned at the receipted spine-e baseline (2026-08-26): zero rows clipped on every non-exempt column, so any clipping is a real signal against a clean baseline; a missing allowance fails closed rather than skipping the comparison. private_pension_wealth joined the gated columns with microcosm#1063 (it had been an output without an allowance); the derived totals are clipped like every other column, and a clipped total would also break its identity in the wealth_coherence gate." + }, + { + "id": "uk_stage_was_wealth_coherence", + "gate": "stage_health", + "phase": "transferred", + "criticality": "release_blocking", + "evidence_absent_blocks": true, + "parameters": { + "stage": "was_wealth", + "check": "wealth_coherence", + "maximum_owner_share_without_main_residence_excess": 0.005 + }, + "notes": "Stage-time coherence gate over the WAS wealth receipt (microcosm#1063). Zero rows may carry a main-residence mortgage off the mortgaged tenure or main_residence_value off an owner tenure: both columns are fitted and drawn inside their tenure stratum, so a non-zero count is a defect of the stage, not of a draw; mortgage_debt itself is the main-residence mortgage plus the mortgages on other property, held on any tenure. Every derived total must equal its components on every row (property_wealth, corporate_wealth, gross_financial_wealth, mortgage_debt, and net_financial_wealth as gross less consumer debt and the student loan balance); the stage derives them, so only a support clip of a total can open a gap, and the support gate allows none. The weighted share of owner households without a main-residence value may exceed the donor's own share by at most the reviewed excess (the donor's share is zero on the pinned round-8 tab, where every owner reports a value). The receipt's donor comparisons (positive shares by tenure, the mortgaged share owing more than the main-residence value, the mortgage mass held off the mortgaged tenure on each side) and the rows the donor-range cap adjusted are recorded in the details and carry no threshold." }, { "id": "uk_stage_was_lisa_support", @@ -229,6 +245,7 @@ "parameters": { "stage": "lcfs_consumption", "check": "support_clip", + "donor_floor": 0.0, "columns": [ "alcohol_and_tobacco_consumption", "bus_fare_spending", @@ -456,6 +473,24 @@ }, "notes": "Stage-time gate over the Pension Credit take-up receipt (microcosm#1069 R9): each component band (Guarantee Credit, Savings Credit only) realizes its DWP FYE 2024 caseload take-up within 0.05 at the stage's weights, or at least the rate where its reporters alone exceed it. The terminal take-up signal gate no longer measures would_claim_pc, whose rate is a share of the entitled units the engine finds, not of every benefit unit." }, + { + "id": "uk_stage_child_benefit_take_up", + "gate": "stage_health", + "phase": "transferred", + "criticality": "release_blocking", + "evidence_absent_blocks": true, + "population_fact_check": true, + "parameters": { + "stage": "child_benefit_take_up", + "check": "child_benefit_take_up", + "maximum_claim_rate_deviation": 0.02, + "maximum_age_claim_rate_deviation": 0.05, + "minimum_age_child_rows": 200, + "maximum_opt_out_share_deviation": 0.01, + "minimum_eligible_family_units": 1 + }, + "notes": "Stage-time gate over the Child Benefit take-up receipt (microcosm#1063): the share of eligible children claimed for is within 0.02 of HMRC's published claim rates by single year of age (Table 15, May 2025) at the frame's age mix, and within 0.05 at each age with at least 200 child rows whose solved rate was not clipped (clipped ages are reported); the opted-out share of claiming families is within 0.01 of the published opted-out share of registered families (August 2025), or below it only where the charged families are exhausted. The terminal take-up signal gate no longer measures would_claim_child_benefit or child_benefit_opts_out, whose rates are shares of eligible children and of claiming families, not of every benefit unit." + }, { "id": "uk_stage_cgt_incidence_clone_mass", "gate": "stage_health", @@ -494,7 +529,11 @@ "parameters": { "stage": "spi_income_band_donors", "check": "spi_income_band_donor_support", - "donors_per_band": 120, + "minimum_donors_per_band": 120, + "maximum_donor_weight": 250.0, + "minimum_funding_factor": 0.5, + "maximum_band_taxpayer_deviation": 0.5, + "maximum_relative_mass_deviation": 1e-09, "band_lower_bounds": [ 200000, 500000, @@ -502,7 +541,7 @@ 2000000 ] }, - "notes": "Stage-time support gate for spi_income_band_donors (PolicyEngine/chronicle#280 lane): every HMRC Table 2.5 band from GBP 200,000 carries its reserved donor households at positive band-exact weight with one carrier each, and the bands are exactly the declared four." + "notes": "Stage-time support gate for spi_income_band_donors (PolicyEngine/chronicle#280 lane; mass-conserving since microcosm#1063): every HMRC Table 2.5 band from GBP 200,000 carries reserved donor households with one carrier each at equal positive weight no greater than the declared maximum, and the bands are exactly the declared four. The donors' mass is reallocated from the households of each donor's region: total household mass is conserved to the declared deviation and no region's incumbents are scaled below the declared funding factor. At full scale every band seats its planned count (never fewer than the minimum) and carries its published taxpayers to the declared deviation; a full-sample build must be at full scale, and a small frame records its seating and mass scales instead." }, { "id": "uk_stage_hmrc_cgt_gains_spine_summary", @@ -517,6 +556,22 @@ }, "notes": "Stage-time CGT redraw summary gate wired through the existing UKCGTImputationSummary receipt rather than fabricating a terminal support verdict." }, + { + "id": "uk_stage_cgt_residential_split_mass", + "gate": "stage_health", + "phase": "transferred", + "criticality": "release_blocking", + "evidence_absent_blocks": true, + "parameters": { + "stage": "cgt_residential_split", + "check": "cgt_residential_split", + "maximum_relative_mass_deviation": 1e-09, + "maximum_solve_relative_error": 1e-06, + "maximum_identity_relative_error": 1e-09, + "maximum_liable_gainers_per_household": 3 + }, + "notes": "Stage-time gate over the CGT residential split receipt (microcosm#1063). The split carries the solved residential probability as weight: every household with liable gainers becomes arms whose weights are products of p and 1 - p, one arm per subset of residential gainers, the arm with no residential gainer keeping the household's ids. Total household mass is conserved to the declared deviation; the logistic solve reproduces the Table 8a 2024-25 totals on the individuals basis within the solve tolerance; the residential count and gains at design weights equal the solved expectations in every HMRC gain band to the identity tolerance (an arithmetic identity, not a realisation); every arm weight is its declared product; the arms created match the households by liable gainers; and no household carries more liable gainers than the declared maximum. The details carry the arm-weight distribution and the share of the residential gains on the largest arm stakes." + }, { "id": "uk_stage_hmrc_cgt_asset_type_spine_summary", "gate": "stage_health", @@ -526,11 +581,9 @@ "parameters": { "stage": "hmrc_cgt_asset_type_spine", "check": "cgt_asset_type_summary", - "maximum_relative_deviation": 0.05, - "maximum_gains_sigma": 3.0, "maximum_solve_relative_error": 1e-06 }, - "notes": "Stage-time gate over the CGT asset-type receipt (microcosm#725, #1014): the logistic solve reproduces the Table 8a 2024-25 totals on the individuals basis in expectation, the realised residential taxpayer count sits within one carrier row's weight of the expectation (the weighted systematic walk's own bound, max_liable_weight, about 3,300 people at full scale), and the realised residential gains sit within the wider of the reviewed relative band and three Bernoulli sigmas of the draw (an envelope that overstates a systematic draw's noise, so conservative), so a small frame is judged by its noise floor and a production frame by the band. Each Table 4.1 2024-25 BADR band is held to deterministic bounds only: its solve to the band's published claimant count and qualifying gains (the open top band to the published gains over the lifetime limit), its realised count within the band pool's largest weight, its realised qualifying gains within that weight times (2 max gain - min gain) of the band pool below the limit and exactly the limit times the realised count in the top band; every BADR invariant (claimants outside the non-residential liable pool, qualifying gains above the net gain or the limit or outside the band, residential overlap, sub-AEA claimants) must be zero and the restricted main-type fit must have converged. Every liable gainer carries an asset type and the Table 7 composition receipt is finite; the calibration binds the residential totals and the Table 4.1 bands." + "notes": "Stage-time gate over the CGT asset-type receipt (microcosm#725, #1014): the residential arms the cgt_residential_split stage left carry the Table 8a 2024-25 totals on the individuals basis at design weights within the solve tolerance (microcosm#1063 carries the solved residential probability as weight, so nothing is drawn here and no walk bound applies), and every gain band row of the residential receipt is finite and sums to the receipt. Each Table 4.1 2024-25 BADR band is held to deterministic bounds only: its solve to the band's published claimant count and qualifying gains (the open top band to the published gains over the lifetime limit), its realised count within the band pool's largest weight, its realised qualifying gains within that weight times (2 max gain - min gain) of the band pool below the limit and exactly the limit times the realised count in the top band; every BADR invariant (claimants outside the non-residential liable pool, qualifying gains above the net gain or the limit or outside the band, residential overlap, sub-AEA claimants) must be zero and the main-type fits (one per claimant status since microcosm#1063) must have converged. Every liable gainer carries an asset type and the Table 7 composition receipt is finite; the calibration binds the residential totals and the Table 4.1 bands." }, { "id": "uk_stage_cgt_incidence_anchor_composition", @@ -772,6 +825,8 @@ "household.cash_isa", "household.cgt_support_copies", "household.cgt_support_copy_index", + "household.household_is_cgt_residential_clone", + "household.cgt_residential_clone_index", "household.clone_index", "household.constituency_code_oa", "household.consumer_debt", @@ -812,6 +867,7 @@ "person.capital_gains_asset_type", "person.capital_gains_badr", "person.capital_gains_residential_property", + "person.cgt_residential_probability", "person.bus_in_london_trips", "person.bus_pass_eligible", "person.care_hours", @@ -851,7 +907,45 @@ "person.would_claim_carers_allowance", "person.would_claim_marriage_allowance", "person.would_claim_scp", - "person.person_is_spi_income_band_carrier" + "person.person_is_spi_income_band_carrier", + "benunit.benunit_source_id", + "benunit.benunit_support_channel", + "benunit.benunit_support_clone_index", + "benunit.dependent_children", + "household.council_tax_rebate", + "household.council_tax_reported", + "household.council_tax_single_adult_raw", + "household.household_source_id", + "household.household_support_channel", + "household.household_support_clone_index", + "household.num_bedrooms", + "household.source_household_id", + "household.source_household_key", + "household.source_year", + "household.subrent", + "person.disabled_students_allowance_eligible_expenses", + "person.free_school_breakfasts", + "person.hmrc_spi_assessable_income", + "person.hmrc_spi_employed_income", + "person.hmrc_spi_employment_benefits", + "person.hmrc_spi_employment_expenses", + "person.hmrc_spi_incapacity_benefit_income", + "person.hmrc_spi_miscellaneous_employment_income", + "person.hmrc_spi_other_income", + "person.hmrc_spi_other_social_security_income", + "person.hmrc_spi_pay", + "person.hmrc_spi_state_pension_income", + "person.hmrc_spi_taxable_termination_pay", + "person.hmrc_spi_total_earned_income", + "person.hmrc_spi_total_investment_income", + "person.hmrc_spi_unemployment_benefit_income", + "person.is_in_approved_training", + "person.ossben_identifiable_subset", + "person.other_investment_income", + "person.person_source_id", + "person.person_support_channel", + "person.person_support_clone_index", + "person.srp_regular_code5" ], "reviewed_exclusions": { "person.incapacity_benefit_reported": "The enhanced FRS stores this legacy reported-benefit input as an all-zero layer; the candidate must drop dead zero layers." @@ -867,7 +961,7 @@ "parameters": { "maximum_share_deviation": 0.05 }, - "notes": "Every contract-driven stochastic flag must be non-constant and have a weighted share within +/-0.05 of its contract target. would_claim_pc is measured by uk_stage_pension_credit_take_up instead (microcosm#1069 R9)." + "notes": "Every contract-driven stochastic flag must be non-constant and have a weighted share within +/-0.05 of its contract target. would_claim_pc is measured by uk_stage_pension_credit_take_up instead (microcosm#1069 R9), and would_claim_child_benefit and child_benefit_opts_out by uk_stage_child_benefit_take_up (microcosm#1063)." }, { "id": "uk_brma_enum_domain", diff --git a/packages/microcosm-build/src/microcosm/build/uk/hmrc_child_benefit_statistics.json b/packages/microcosm-build/src/microcosm/build/uk/hmrc_child_benefit_statistics.json new file mode 100644 index 000000000..febc0be20 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk/hmrc_child_benefit_statistics.json @@ -0,0 +1,3186 @@ +{ + "schema_version": 1, + "country": "uk", + "kind": "uk_vendored_ledger_facts", + "resource": "hmrc_child_benefit_statistics.json", + "purpose": "HMRC Child Benefit statistics, annual release at August 2025, United Kingdom: the percentage of eligible children for whom Child Benefit is claimed, opted-out families included, by single year of age and for all ages (Table 15, May 2025 reference month); families registered, families in payment and families opted out of payment because of the High Income Child Benefit Charge; children registered; children in payment and children in opted-out families by single year of age. Consumed by the child_benefit_take_up stage, which draws family claims to the claim rates by age and opt-outs to the opted-out share of registered families, and reports the frame's families and children in payment beside the published ones (microcosm#1063).", + "consumers": [ + "uk_runtime.child_benefit_take_up" + ], + "planned_consumers": [], + "policy": "Rows are copied verbatim from the pinned Chronicle consumer feed named in source_fact_feed; every value is a publisher-stated fact. Consumers declare their own arithmetic on these rows and never edit this file by hand: regenerate it with tools/vendor_uk_ledger_facts.py.", + "generator": { + "tool": "tools/vendor_uk_ledger_facts.py", + "version": 1 + }, + "source_fact_feed": { + "source_repo": "PolicyEngine/chronicle", + "source_commit": "825406f98913ab7324c834c3642804e9317a2256", + "build": "build-bundle --suite uk -> build-consumer-artifact", + "artifact_schema_version": "policyengine_ledger.consumer_artifact.v2", + "fact_row_count": 344402, + "facts_sha256": "28b7105761be48ff01244bd0e28f4b579bac103f2c4a98473a04e2cc4303eb3d", + "manifest_sha256": "6e04a43d7dca4d398e14c0ec59ec983477633ba3e9bef062a8843f554dcb7119" + }, + "selections": [ + { + "label": "claim_rate_of_eligible_children_by_age_may_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "source_measure_id": "claim_rate", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-05" + ] + }, + "row_count": 21 + }, + { + "label": "families_registered_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.families_registered_for_child_benefit", + "source_measure_id": "families_registered", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "row_count": 1 + }, + { + "label": "families_in_payment_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.families_in_receipt_of_child_benefit_payments", + "source_measure_id": "families_in_payment", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "row_count": 1 + }, + { + "label": "families_opted_out_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.families_opted_out_of_child_benefit_payment", + "source_measure_id": "families_opted_out_claimant_all", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "row_count": 1 + }, + { + "label": "children_registered_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.children_registered_for_child_benefit", + "source_measure_id": "children_registered", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "row_count": 1 + }, + { + "label": "children_in_payment_by_age_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "source_measure_id": "children_in_payment_united_kingdom", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "row_count": 21 + }, + { + "label": "children_in_opted_out_families_by_age_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "source_measure_id": "children_opted_out_united_kingdom", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "row_count": 21 + } + ], + "row_count": 67, + "rows": [ + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_79e409495cd4974828e3decc", + "semantic_fact_key": "ledger_semantic_fact_v2_45c6a0d680a425984964b172", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_0.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_0", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 0, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 446265 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_503a92a21e799d1ddac56e9d", + "semantic_fact_key": "ledger_semantic_fact_v2_c62466e8adab89161ed2f36f", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_1.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_1", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 1, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 499555 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_b2f8c74fdb5eadc8ae206a0a", + "semantic_fact_key": "ledger_semantic_fact_v2_7760aac3b961fdd2c90da778", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_10.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_10", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 10, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 673440 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_9fecb531149442b6982ad203", + "semantic_fact_key": "ledger_semantic_fact_v2_c6b9a910e1d157ad59c64d70", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_11.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_11", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 11, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 677790 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_07c80577f2ae68c9e5e89868", + "semantic_fact_key": "ledger_semantic_fact_v2_7aec3e1c88b8b0a8e298b08d", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_12.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_12", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 12, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 696090 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c54a020d12467bb0e6ab2c17", + "semantic_fact_key": "ledger_semantic_fact_v2_f0085aa8c63cf3555f5c1e8d", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_13.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_13", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 13, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 721200 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_1071283ee9985d7620a85604", + "semantic_fact_key": "ledger_semantic_fact_v2_7d403b33fe74e495de75b5f8", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_14.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_14", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 14, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 724780 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_9d63d491a1dbbbf9dced574e", + "semantic_fact_key": "ledger_semantic_fact_v2_a269829431f132879222d557", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_15.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_15", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 15, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 716575 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_db2d4704651f2ea949acfb34", + "semantic_fact_key": "ledger_semantic_fact_v2_0c38ba5ae9c8c0712943b2a5", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_16.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_16", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 16, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 704790 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_613486a04f7a006b88db0f9c", + "semantic_fact_key": "ledger_semantic_fact_v2_31c9be3a70ccd8b9ad1a3475", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_17.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_17", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 17, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 555640 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_a97e56df6ce74898eaf8f0b9", + "semantic_fact_key": "ledger_semantic_fact_v2_1b411d8c2f48a8ebac188de5", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_18.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_18", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 18, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 479675 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c619a9b259227285e4c1b0c6", + "semantic_fact_key": "ledger_semantic_fact_v2_04fe98b7e1f5e9c401627018", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_19.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_19", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 19, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 95335 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c35cfaa5ddc3f806c0f0dff6", + "semantic_fact_key": "ledger_semantic_fact_v2_ab0edf246beba50ed666b9f8", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_2.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_2", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 2, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 514875 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_64ffbb62ea267173c9b3fd50", + "semantic_fact_key": "ledger_semantic_fact_v2_e525560fd05573cdab1732ba", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_3.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_3", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 3, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 549135 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_578c59920b56b0dd9cc0f2ac", + "semantic_fact_key": "ledger_semantic_fact_v2_aca3c93f85f0aad0cffe5212", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_4.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_4", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 4, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 554760 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_5248846992a90d6c149b36fb", + "semantic_fact_key": "ledger_semantic_fact_v2_70030ca4c518bf7fd0eea82a", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_5.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_5", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 5, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 576155 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_1bec5dc07709aac653782a94", + "semantic_fact_key": "ledger_semantic_fact_v2_3bd63f079209dad8fcbc6c65", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_6.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_6", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 6, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 602215 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_9a030cd57855750abe07e71c", + "semantic_fact_key": "ledger_semantic_fact_v2_907d520618de5de8a4752068", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_7.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_7", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 7, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 623920 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_e789b41c9a6d7a8b88993cb0", + "semantic_fact_key": "ledger_semantic_fact_v2_b2e7ca01d65c653a348ebd0e", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_8.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_8", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 8, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 648010 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_ceff5759e25e209d9a5285d9", + "semantic_fact_key": "ledger_semantic_fact_v2_64a3c097dfa5a98652c05918", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.age_9.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_9", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 9, + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 670530 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_2a2115c74ce179bc2d1483ef", + "semantic_fact_key": "ledger_semantic_fact_v2_f4c8b43ae43b3523228e564c", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08.all_ages.children_in_payment_united_kingdom", + "concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "measure_id": "children_in_payment_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_payment_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "all_ages", + "table_record_kind": "total" + }, + "dimensions": { + "child_age": "All ages", + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 11730730 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_f4bebe196a7f7307e0c454be", + "semantic_fact_key": "ledger_semantic_fact_v2_f0a27057d7d3590b83b95a4e", + "source_record_id": "hmrc.child_benefit.aug2025.children_registered.month2025_08.united_kingdom.children_registered", + "concept": "hmrc.children_registered_for_child_benefit", + "measure_id": "children_registered", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_registered.month2025_08", + "groupby_dimension": "hmrc.child_benefit_status", + "groupby_value_id": "united_kingdom", + "table_record_kind": "total" + }, + "dimensions": { + "payment_status": "registered" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 12730410 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c77d8aae3c71630c14d57395", + "semantic_fact_key": "ledger_semantic_fact_v2_78719fb8899064d4b6073e90", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_0.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_0", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 0, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 9770 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_72363d43cca06d1f4ec616fc", + "semantic_fact_key": "ledger_semantic_fact_v2_b270e3fcf005561593dc22b7", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_1.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_1", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 1, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 16985 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_b63f36e8a573893543c604d4", + "semantic_fact_key": "ledger_semantic_fact_v2_3a3632616cada0d6a1db2e3b", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_10.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_10", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 10, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 63975 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_194a3f458865b8806195bc71", + "semantic_fact_key": "ledger_semantic_fact_v2_021dcf5bbaa90b06921fec79", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_11.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_11", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 11, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 65770 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_f8e6193a8fb9ecefc3c95ffb", + "semantic_fact_key": "ledger_semantic_fact_v2_2cdba01321c4777daecb347e", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_12.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_12", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 12, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 72930 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_5f4f001b7e5d7df453f0f91d", + "semantic_fact_key": "ledger_semantic_fact_v2_464fd7a715750d31ecb189a7", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_13.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_13", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 13, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 92660 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_6c57fbe3c3a965ad2c25c10b", + "semantic_fact_key": "ledger_semantic_fact_v2_3fbeea13d75accb625281f55", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_14.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_14", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 14, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 94905 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_ca1b3044e6ee4ad40ffedd73", + "semantic_fact_key": "ledger_semantic_fact_v2_66a7ce8fb0add45b9a23f0f8", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_15.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_15", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 15, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 93890 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_7c8456924d700de30802e9ba", + "semantic_fact_key": "ledger_semantic_fact_v2_cc7f53f2b1e75dab75aa8db9", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_16.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_16", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 16, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 92800 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_aec8480730b986d3f8d8c9ef", + "semantic_fact_key": "ledger_semantic_fact_v2_da3c511d41a0954e897dd956", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_17.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_17", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 17, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 24590 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_7f9e51b30f222e29a1eb2ad9", + "semantic_fact_key": "ledger_semantic_fact_v2_fa2675e5dabb9e0ea9f97a19", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_18.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_18", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 18, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 25735 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_cac141d5ca9c9a1d63fa9990", + "semantic_fact_key": "ledger_semantic_fact_v2_d2c4261aad02daf86a374524", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_19.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_19", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 19, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 1105 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_d011cf2fc0cb95a9ed1206c8", + "semantic_fact_key": "ledger_semantic_fact_v2_9f1fda5fd0c3cb6a49a8ee2c", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_2.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_2", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 2, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 21480 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_fc30316768b734512ea2a80c", + "semantic_fact_key": "ledger_semantic_fact_v2_33a1803623598570d4d31510", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_3.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_3", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 3, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 27150 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_0f5b302cf27dfee19ffc6d8a", + "semantic_fact_key": "ledger_semantic_fact_v2_616f56f9d0cc37ec70a95a5b", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_4.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_4", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 4, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 33930 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_5d20c1d8bcfbe35b88d81f41", + "semantic_fact_key": "ledger_semantic_fact_v2_84adba98f5e25cdcf69ad096", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_5.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_5", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 5, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 39995 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_a0a33724586363f7f28ecff6", + "semantic_fact_key": "ledger_semantic_fact_v2_1973ae2f8b47617327359702", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_6.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_6", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 6, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 48375 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_dfd4d217494d05b2733e359c", + "semantic_fact_key": "ledger_semantic_fact_v2_554e7f49069e2c5fd8341bfd", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_7.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_7", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 7, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 53275 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_0fd042e9f166f23b7d6d3237", + "semantic_fact_key": "ledger_semantic_fact_v2_23e9d94987660bf0113606fe", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_8.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_8", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 8, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 58445 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_884f5ad68a46caa7a98e3b03", + "semantic_fact_key": "ledger_semantic_fact_v2_2cde16dc993a00d9fd747883", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.age_9.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_9", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 9, + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 61915 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_452103727fef41a179e3b894", + "semantic_fact_key": "ledger_semantic_fact_v2_c63a36027d953f5bdd016d58", + "source_record_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08.all_ages.children_opted_out_united_kingdom", + "concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "measure_id": "children_opted_out_united_kingdom", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "person", + "role": "child_in_child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.children_in_opted_out_families_by_single_year_of_age.month2025_08", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "all_ages", + "table_record_kind": "total" + }, + "dimensions": { + "child_age": "All ages", + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 999680 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_27f26b9f974df083550d5b9c", + "semantic_fact_key": "ledger_semantic_fact_v2_4a1ef077ef4534a750d87eb7", + "source_record_id": "hmrc.child_benefit.aug2025.families_in_payment.month2025_08.united_kingdom.families_in_payment", + "concept": "hmrc.families_in_receipt_of_child_benefit_payments", + "measure_id": "families_in_payment", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "family", + "role": "child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.families_in_payment.month2025_08", + "groupby_dimension": "hmrc.child_benefit_status", + "groupby_value_id": "united_kingdom", + "table_record_kind": "total" + }, + "dimensions": { + "payment_status": "in_payment" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 6867695 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_0f562aa196b46cd26de2eb08", + "semantic_fact_key": "ledger_semantic_fact_v2_a1ba9be6269019713601a08d", + "source_record_id": "hmrc.child_benefit.aug2025.families_opted_out_by_claimant_sex.month2025_08.united_kingdom.families_opted_out_claimant_all", + "concept": "hmrc.families_opted_out_of_child_benefit_payment", + "measure_id": "families_opted_out_claimant_all", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "family", + "role": "child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.families_opted_out_by_claimant_sex.month2025_08", + "groupby_dimension": "hmrc.child_benefit_claimant_sex", + "groupby_value_id": "united_kingdom", + "table_record_kind": "total" + }, + "dimensions": { + "claimant_sex": "all", + "payment_status": "opted_out" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 684635 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_49f5f7c1dae7a183ce6989ac", + "semantic_fact_key": "ledger_semantic_fact_v2_6c91363f893bd078687a9723", + "source_record_id": "hmrc.child_benefit.aug2025.families_registered.month2025_08.united_kingdom.families_registered", + "concept": "hmrc.families_registered_for_child_benefit", + "measure_id": "families_registered", + "unit": "count", + "aggregation": "sum", + "entity": { + "name": "family", + "role": "child_benefit_family" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-08" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.families_registered.month2025_08", + "groupby_dimension": "hmrc.child_benefit_status", + "groupby_value_id": "united_kingdom", + "table_record_kind": "total" + }, + "dimensions": { + "payment_status": "registered" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 7552330 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_2fb8d5dbb3f0063e2aa86bb1", + "semantic_fact_key": "ledger_semantic_fact_v2_c3e0c9a6c3f1796cb2af083c", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_0.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_0", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 0 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 68.8 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_a9da327d00cd0c3f4d2d09c3", + "semantic_fact_key": "ledger_semantic_fact_v2_0ecf043e211635f2e169fc88", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_1.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_1", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 1 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 76.8 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_9463a511e71824b2b8109bf1", + "semantic_fact_key": "ledger_semantic_fact_v2_8fd4c5401d1c341e38fd757b", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_10.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_10", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 10 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 92.10000000000001 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_555cbd7ea49d368345a223cc", + "semantic_fact_key": "ledger_semantic_fact_v2_108474417d2760bc85cac3d5", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_11.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_11", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 11 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 92.7 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_a2f48fe342f4285e88829325", + "semantic_fact_key": "ledger_semantic_fact_v2_6699260893b6881d7800132e", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_12.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_12", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 12 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 94.5 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_bfba8866621d25aa7e2a25ec", + "semantic_fact_key": "ledger_semantic_fact_v2_5e9e387dc53a31a875c5709d", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_13.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_13", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 13 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 96.8 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_47b41c16162dff7c783900c8", + "semantic_fact_key": "ledger_semantic_fact_v2_8a376d9448e331c2699484e2", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_14.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_14", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 14 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 96.39999999999999 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_fe44b4a70745aaf0dc43ffb8", + "semantic_fact_key": "ledger_semantic_fact_v2_20c088e169b1f7e356687ce2", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_15.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_15", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 15 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 96.6 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_6d9574d7c3c2871936ee050c", + "semantic_fact_key": "ledger_semantic_fact_v2_c4574d61347e6d3cef42ec92", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_16.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_16", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 16 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 89.4 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_5fe0474042ba683982af7e17", + "semantic_fact_key": "ledger_semantic_fact_v2_b32de577d9286372f967dcfa", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_17.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_17", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 17 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 75.1 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c649ebbdb0edf114254d8014", + "semantic_fact_key": "ledger_semantic_fact_v2_4f9f1d19d6eb6ec29d89db59", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_18.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_18", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 18 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 76.9 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_146f43e1c72a0804c841e1c7", + "semantic_fact_key": "ledger_semantic_fact_v2_64c53054bc2a973b13ae1432", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_19.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_19", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 19 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 49.7 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_1af023442d69ea844c5a639a", + "semantic_fact_key": "ledger_semantic_fact_v2_9937677d28e67b855de2aadd", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_2.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_2", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 2 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 79 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_368ac4145abfdad71df67202", + "semantic_fact_key": "ledger_semantic_fact_v2_20e0e4d3c469b147887207cb", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_3.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_3", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 3 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 81 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_8d43570474001508e7d886f6", + "semantic_fact_key": "ledger_semantic_fact_v2_dde5f895e9255537e39b00e4", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_4.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_4", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 4 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 81.6 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c8ea4160e9db8486b6b8e5a0", + "semantic_fact_key": "ledger_semantic_fact_v2_ecd85f5c2a7e32e1a15dd297", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_5.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_5", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 5 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 84.39999999999999 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_7fd4c81d64cdb31ea4fb57ca", + "semantic_fact_key": "ledger_semantic_fact_v2_dda3bd5e79611b0f6d2f6a70", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_6.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_6", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 6 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 86.8 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_3e58f0b4e7e4fc1409262d32", + "semantic_fact_key": "ledger_semantic_fact_v2_0a2596bfd371bdf8751dc438", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_7.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_7", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 7 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 89.3 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_c2a67c2e0be7d5c49857180d", + "semantic_fact_key": "ledger_semantic_fact_v2_3b15d5abcf1df0010188930d", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_8.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_8", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 8 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 90.9 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_91bc2fb4cbdb14a39afc5ec4", + "semantic_fact_key": "ledger_semantic_fact_v2_ef8d1f0f5edd77423d9ed9a2", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.age_9.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "age_9", + "table_record_kind": "detail" + }, + "dimensions": { + "child_age": 9 + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 91.2 + }, + { + "aggregate_fact_key": "ledger_aggregate_fact_v2_3b25fa0f0fa99b6c60d824e0", + "semantic_fact_key": "ledger_semantic_fact_v2_142c1cc432d5cbe3d458a905", + "source_record_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05.all_ages.claim_rate", + "concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "measure_id": "claim_rate", + "unit": "percent", + "aggregation": "rate", + "entity": { + "name": "person", + "role": "child_eligible_for_child_benefit" + }, + "geography": { + "id": "K02000001", + "level": "country", + "vintage": "current" + }, + "period": { + "type": "month", + "value": "2025-05" + }, + "period_coverage": { + "start_date": null, + "end_date": null, + "basis": null + }, + "layout": { + "record_set_id": "hmrc.child_benefit.aug2025.claim_rate_of_eligible_children_by_age.month2025_05", + "groupby_dimension": "hmrc.child_benefit_child_age", + "groupby_value_id": "all_ages", + "table_record_kind": "total" + }, + "dimensions": { + "child_age": "All ages" + }, + "source": { + "source_name": "hmrc", + "source_table": "Child Benefit Statistics: Annual Release, data at August 2025, main tables", + "source_file": "Child_Benefit_August_2025_Main_Tables.ods", + "source_sha256": "c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d", + "vintage": "august_2025", + "raw_r2_key": "raw/uk/hmrc/hmrc-child-benefit-august-2025/2025/c610f9509901f21efc28b76d24fae01c43d281b6867a14f28b6847d0e16cc69d/Child_Benefit_August_2025_Main_Tables.ods" + }, + "value": 86.6 + } + ] +} diff --git a/packages/microcosm-build/src/microcosm/build/uk/input_mass_reviewed_exclusions.json b/packages/microcosm-build/src/microcosm/build/uk/input_mass_reviewed_exclusions.json index 0679de266..c3df7a683 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/input_mass_reviewed_exclusions.json +++ b/packages/microcosm-build/src/microcosm/build/uk/input_mass_reviewed_exclusions.json @@ -1,7 +1,36 @@ { - "schema_version": 3, - "description": "Reviewed input-mass parity exclusions for the UK terminal battery, scoped per named comparison reference (microcosm#630): an exclusion suppresses only when its reference is the armed one, so a column excluded on a channel-blind reference is compared normally against a channel-aware one with no register change. Each entry is a complete approval receipt (schema-2 fields per #610). The gate reports dormant entries, FAILS stale ones, and FAILS expired ones until the adjudication is renewed, so this register cannot rot.", - "references": { - "efrs-post-calibration": {} + "schema_version": 3, + "description": "Reviewed input-mass parity exclusions for the UK terminal battery, scoped per named comparison reference (microcosm#630): an exclusion suppresses only when its reference is the armed one, so a column excluded on a channel-blind reference is compared normally against a channel-aware one with no register change. Each entry is a complete approval receipt (schema-2 fields per #610). The gate reports dormant entries, FAILS stale ones, and FAILS expired ones until the adjudication is renewed, so this register cannot rot.", + "references": { + "efrs-post-calibration": { + "access_fund": { + "reason": "One FRS household carries 98% of the candidate's access-fund mass (GBP 1.01bn against the incumbent's GBP 160m, +535%): sernum 14443's 19-year-old reports an access fund of GBP 4,500 with FRS period code 5 (calendar month), which the FRS weeklyises to GBP 1,035.62 (ACCSSAMT) and the spine annualises to GBP 54,037, eleven times the next largest FRS value; the calibration takes the household and its two capital-gains copies to the 10x weight bound. The value is almost certainly an annual or termly award under the wrong period code; the input stays on the release surface until the period rule is settled in the spine, and the exclusion expires with it.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (PR #1089 release-cut certifier on the 2026-10-03 final build; ruling 2026-10-04 in session: the deferrals needed to build and publish the national dataset are signed, each with its diagnosis)", + "approved_on": "2026-10-04", + "expires_on": "2027-01-04" + }, + "dfe_education_spending": { + "reason": "The candidate carries DfE education spending through the build (GBP 98.8bn against the incumbent's GBP 52.8m, +187,061%): the enhanced-FRS reference carries almost none of this input, so the comparison measures the reference's omission, not the candidate's. The input stays on the release surface and in the coverage manifest.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 c9 (certifier rehearsal R5 on the 2026-09-30 build, 2026-10-01; ruling 2026-10-02 in session)", + "approved_on": "2026-10-02", + "expires_on": "2027-04-02" + }, + "jsa_income_reported": { + "reason": "The candidate carries GBP 272m of reported income-based JSA against the incumbent's GBP 23m (+1,058%): the enhanced-FRS reference holds only the FRS respondents' amounts, while the spine's SPI synthetic channel receives this legacy-benefit leaf from the stage-2 QRF at FRS rates (76 of the 196 nonzero rows and most of the mass) and a few rows sit at the calibration's weight bound. Neither is new to this build: the 2026-09-30 main candidate stood at +428% on the same column. The comparison measures the SPI channel's imputation of a legacy benefit, which is a design question for the channel, not a mass the candidate lost or invented.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (PR #1089 release-cut certifier on the 2026-10-03 final build; ruling 2026-10-04 in session: the deferrals needed to build and publish the national dataset are signed, each with its diagnosis)", + "approved_on": "2026-10-04", + "expires_on": "2027-04-04" + }, + "working_tax_credit_reported": { + "reason": "The candidate carries GBP 625m of reported Working Tax Credit against the incumbent's GBP 74m (+747%): the enhanced-FRS reference holds only the FRS respondents' amounts, while the spine's SPI synthetic channel receives this legacy-benefit leaf from the stage-2 QRF at FRS rates (274 of the 372 nonzero rows) and a few rows sit at the calibration's weight bound. The 2026-09-30 main candidate stood at +421% on the same column. Tax credits closed in April 2025, so the SPI channel's treatment of the leaf is a design question for the channel, not a mass the candidate lost or invented.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (PR #1089 release-cut certifier on the 2026-10-03 final build; ruling 2026-10-04 in session: the deferrals needed to build and publish the national dataset are signed, each with its diagnosis)", + "approved_on": "2026-10-04", + "expires_on": "2027-04-04" + } } + } } diff --git a/packages/microcosm-build/src/microcosm/build/uk/ledger_compile_parity_incumbent_2025_signed_differences.json b/packages/microcosm-build/src/microcosm/build/uk/ledger_compile_parity_incumbent_2025_signed_differences.json index 585358f3e..1ffbcc223 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/ledger_compile_parity_incumbent_2025_signed_differences.json +++ b/packages/microcosm-build/src/microcosm/build/uk/ledger_compile_parity_incumbent_2025_signed_differences.json @@ -6347,7 +6347,7 @@ { "fixture_value": 82300000000.0, "kind": "calibration_drift", - "ledger_value": 91081771559.55191, + "ledger_value": 101699285714.28572, "name": "hmrc/employment_income_income_band_100_000_to_150_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6355,7 +6355,7 @@ { "fixture_value": 16900000000.0, "kind": "calibration_drift", - "ledger_value": 18703304244.914062, + "ledger_value": 17745000000.0, "name": "hmrc/employment_income_income_band_12_570_to_15_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6363,7 +6363,7 @@ { "fixture_value": 39200000000.0, "kind": "calibration_drift", - "ledger_value": 43382812213.05511, + "ledger_value": 45429306930.69307, "name": "hmrc/employment_income_income_band_150_000_to_200_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6371,7 +6371,7 @@ { "fixture_value": 48600000000.0, "kind": "calibration_drift", - "ledger_value": 53785833509.04281, + "ledger_value": 47490871369.2946, "name": "hmrc/employment_income_income_band_15_000_to_20_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6379,7 +6379,7 @@ { "fixture_value": 35500000000.0, "kind": "calibration_drift", - "ledger_value": 39288005958.251434, + "ledger_value": 42457796852.64664, "name": "hmrc/employment_income_income_band_1_000_000_to_inf", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6387,7 +6387,7 @@ { "fixture_value": 36600000000.0, "kind": "calibration_drift", - "ledger_value": 40505380790.760635, + "ledger_value": 44366741321.38857, "name": "hmrc/employment_income_income_band_200_000_to_300_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6395,7 +6395,7 @@ { "fixture_value": 172000000000.0, "kind": "calibration_drift", - "ledger_value": 190353155628.71118, + "ledger_value": 172509881422.92493, "name": "hmrc/employment_income_income_band_20_000_to_30_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6403,7 +6403,7 @@ { "fixture_value": 29000000000.0, "kind": "calibration_drift", - "ledger_value": 32094427402.515255, + "ledger_value": 35153975363.941765, "name": "hmrc/employment_income_income_band_300_000_to_500_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6411,7 +6411,7 @@ { "fixture_value": 170000000000.0, "kind": "calibration_drift", - "ledger_value": 188139746842.3308, + "ledger_value": 190228365384.6154, "name": "hmrc/employment_income_income_band_30_000_to_40_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6419,7 +6419,7 @@ { "fixture_value": 139000000000.0, "kind": "calibration_drift", - "ledger_value": 153831910653.43518, + "ledger_value": 155539663461.53845, "name": "hmrc/employment_income_income_band_40_000_to_50_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6427,7 +6427,7 @@ { "fixture_value": 23100000000.0, "kind": "calibration_drift", - "ledger_value": 25564871482.693188, + "ledger_value": 28300690607.734806, "name": "hmrc/employment_income_income_band_500_000_to_1_000_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6435,7 +6435,7 @@ { "fixture_value": 174000000000.0, "kind": "calibration_drift", - "ledger_value": 192566564415.09152, + "ledger_value": 211351999999.99997, "name": "hmrc/employment_income_income_band_50_000_to_70_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6443,7 +6443,7 @@ { "fixture_value": 118000000000.0, "kind": "calibration_drift", - "ledger_value": 130591118396.44138, + "ledger_value": 143330666666.66666, "name": "hmrc/employment_income_income_band_70_000_to_100_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6555,7 +6555,7 @@ { "fixture_value": 5430000000.0, "kind": "calibration_drift", - "ledger_value": 5906164169.251498, + "ledger_value": 6709928571.428572, "name": "hmrc/private_pension_income_income_band_100_000_to_150_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6563,7 +6563,7 @@ { "fixture_value": 4350000000.0, "kind": "calibration_drift", - "ledger_value": 4731457483.654515, + "ledger_value": 4567500000.0, "name": "hmrc/private_pension_income_income_band_12_570_to_15_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6571,7 +6571,7 @@ { "fixture_value": 1810000000.0, "kind": "calibration_drift", - "ledger_value": 1968721389.7504995, + "ledger_value": 2097628712.8712869, "name": "hmrc/private_pension_income_income_band_150_000_to_200_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6579,7 +6579,7 @@ { "fixture_value": 13600000000.0, "kind": "calibration_drift", - "ledger_value": 14792602707.517565, + "ledger_value": 13289626556.016598, "name": "hmrc/private_pension_income_income_band_15_000_to_20_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6587,7 +6587,7 @@ { "fixture_value": 370000000.0, "kind": "calibration_drift", - "ledger_value": 402445808.954522, + "ledger_value": 442517882.68955654, "name": "hmrc/private_pension_income_income_band_1_000_000_to_inf", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6595,7 +6595,7 @@ { "fixture_value": 1570000000.0, "kind": "calibration_drift", - "ledger_value": 1707675459.6178367, + "ledger_value": 1903163493.8409853, "name": "hmrc/private_pension_income_income_band_200_000_to_300_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6603,7 +6603,7 @@ { "fixture_value": 30200000000.0, "kind": "calibration_drift", - "ledger_value": 32848279541.693417, + "ledger_value": 30289525691.699608, "name": "hmrc/private_pension_income_income_band_20_000_to_30_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6611,7 +6611,7 @@ { "fixture_value": 1120000000.0, "kind": "calibration_drift", - "ledger_value": 1218214340.6190937, + "ledger_value": 1357670772.6763716, "name": "hmrc/private_pension_income_income_band_300_000_to_500_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6619,7 +6619,7 @@ { "fixture_value": 24900000000.0, "kind": "calibration_drift", - "ledger_value": 27083515251.26378, + "ledger_value": 27862860576.923077, "name": "hmrc/private_pension_income_income_band_30_000_to_40_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6627,7 +6627,7 @@ { "fixture_value": 16900000000.0, "kind": "calibration_drift", - "ledger_value": 18381984246.841682, + "ledger_value": 18910937500.0, "name": "hmrc/private_pension_income_income_band_40_000_to_50_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6635,7 +6635,7 @@ { "fixture_value": 640000000.0, "kind": "calibration_drift", - "ledger_value": 696122480.3537678, + "ledger_value": 784088397.7900552, "name": "hmrc/private_pension_income_income_band_500_000_to_1_000_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6643,7 +6643,7 @@ { "fixture_value": 17400000000.0, "kind": "calibration_drift", - "ledger_value": 18925829934.61806, + "ledger_value": 21135200000.0, "name": "hmrc/private_pension_income_income_band_50_000_to_70_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6651,7 +6651,7 @@ { "fixture_value": 9470000000.0, "kind": "calibration_drift", - "ledger_value": 10300437326.484657, + "ledger_value": 11502893333.333332, "name": "hmrc/private_pension_income_income_band_70_000_to_100_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6867,7 +6867,7 @@ { "fixture_value": 4980000000.0, "kind": "calibration_drift", - "ledger_value": 5151235856.246702, + "ledger_value": 6153857142.857143, "name": "hmrc/self_employment_income_income_band_100_000_to_150_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6875,7 +6875,7 @@ { "fixture_value": 4810000000.0, "kind": "calibration_drift", - "ledger_value": 4975390455.531453, + "ledger_value": 5050500000.0, "name": "hmrc/self_employment_income_income_band_12_570_to_15_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6883,7 +6883,7 @@ { "fixture_value": 3650000000.0, "kind": "calibration_drift", - "ledger_value": 3775504191.827402, + "ledger_value": 4230024752.4752474, "name": "hmrc/self_employment_income_income_band_150_000_to_200_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6891,7 +6891,7 @@ { "fixture_value": 8030000000.0, "kind": "calibration_drift", - "ledger_value": 8306109222.020285, + "ledger_value": 7846742738.589211, "name": "hmrc/self_employment_income_income_band_15_000_to_20_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6899,7 +6899,7 @@ { "fixture_value": 19500000000.0, "kind": "calibration_drift", - "ledger_value": 20170501846.749134, + "ledger_value": 23321888412.01717, "name": "hmrc/self_employment_income_income_band_1_000_000_to_inf", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6907,7 +6907,7 @@ { "fixture_value": 4390000000.0, "kind": "calibration_drift", - "ledger_value": 4540948877.293779, + "ledger_value": 5321584546.472564, "name": "hmrc/self_employment_income_income_band_200_000_to_300_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6915,7 +6915,7 @@ { "fixture_value": 15700000000.0, "kind": "calibration_drift", - "ledger_value": 16239839948.408278, + "ledger_value": 15746541501.976286, "name": "hmrc/self_employment_income_income_band_20_000_to_30_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6923,7 +6923,7 @@ { "fixture_value": 4840000000.0, "kind": "calibration_drift", - "ledger_value": 5006421996.834145, + "ledger_value": 5867077267.637177, "name": "hmrc/self_employment_income_income_band_300_000_to_500_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6931,7 +6931,7 @@ { "fixture_value": 12600000000.0, "kind": "calibration_drift", - "ledger_value": 13033247347.13021, + "ledger_value": 14099278846.153845, "name": "hmrc/self_employment_income_income_band_30_000_to_40_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6939,7 +6939,7 @@ { "fixture_value": 8730000000.0, "kind": "calibration_drift", - "ledger_value": 9030178519.083075, + "ledger_value": 9768786057.692307, "name": "hmrc/self_employment_income_income_band_40_000_to_50_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6947,7 +6947,7 @@ { "fixture_value": 6670000000.0, "kind": "calibration_drift", - "ledger_value": 6899346016.298294, + "ledger_value": 8171671270.718231, "name": "hmrc/self_employment_income_income_band_500_000_to_1_000_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6955,7 +6955,7 @@ { "fixture_value": 8220000000.0, "kind": "calibration_drift", - "ledger_value": 8502642316.937327, + "ledger_value": 9984560000.0, "name": "hmrc/self_employment_income_income_band_50_000_to_70_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." @@ -6963,7 +6963,7 @@ { "fixture_value": 5360000000.0, "kind": "calibration_drift", - "ledger_value": 5544302046.080788, + "ledger_value": 6510613333.333333, "name": "hmrc/self_employment_income_income_band_70_000_to_100_000", "period": 2025, "reason": "Declared uprating (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-22): ours moves the SPI 2023-24 band value to the 2025 calibration year by the index the reference declares (amounts by the pinned engine's per-variable OBR and ONS indices between 1 January 2023 and 1 January 2025, the state pension by the new State Pension rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band as the calendar-2025 window); the frozen incumbent fixture holds the 2023-24 values it projects one year forward with its own income-projection ratios." diff --git a/packages/microcosm-build/src/microcosm/build/uk/ledger_fact_vendor_selections.json b/packages/microcosm-build/src/microcosm/build/uk/ledger_fact_vendor_selections.json index a404f29c3..95a83f4ef 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/ledger_fact_vendor_selections.json +++ b/packages/microcosm-build/src/microcosm/build/uk/ledger_fact_vendor_selections.json @@ -941,6 +941,114 @@ "expected_row_count": 13 } ] + }, + { + "resource": "hmrc_child_benefit_statistics.json", + "purpose": "HMRC Child Benefit statistics, annual release at August 2025, United Kingdom: the percentage of eligible children for whom Child Benefit is claimed, opted-out families included, by single year of age and for all ages (Table 15, May 2025 reference month); families registered, families in payment and families opted out of payment because of the High Income Child Benefit Charge; children registered; children in payment and children in opted-out families by single year of age. Consumed by the child_benefit_take_up stage, which draws family claims to the claim rates by age and opt-outs to the opted-out share of registered families, and reports the frame's families and children in payment beside the published ones (microcosm#1063).", + "consumers": [ + "uk_runtime.child_benefit_take_up" + ], + "planned_consumers": [], + "selections": [ + { + "label": "claim_rate_of_eligible_children_by_age_may_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.percentage_of_eligible_children_for_whom_child_benefit_is_claimed", + "source_measure_id": "claim_rate", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-05" + ] + }, + "expected_row_count": 21 + }, + { + "label": "families_registered_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.families_registered_for_child_benefit", + "source_measure_id": "families_registered", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "expected_row_count": 1 + }, + { + "label": "families_in_payment_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.families_in_receipt_of_child_benefit_payments", + "source_measure_id": "families_in_payment", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "expected_row_count": 1 + }, + { + "label": "families_opted_out_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.families_opted_out_of_child_benefit_payment", + "source_measure_id": "families_opted_out_claimant_all", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "expected_row_count": 1 + }, + { + "label": "children_registered_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.children_registered_for_child_benefit", + "source_measure_id": "children_registered", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "expected_row_count": 1 + }, + { + "label": "children_in_payment_by_age_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.children_for_whom_child_benefit_payment_is_received", + "source_measure_id": "children_in_payment_united_kingdom", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "expected_row_count": 21 + }, + { + "label": "children_in_opted_out_families_by_age_august_2025", + "selector": { + "source_name": "hmrc", + "source_concept": "hmrc.children_within_families_opted_out_of_child_benefit_payment", + "source_measure_id": "children_opted_out_united_kingdom", + "geography_id": "K02000001", + "period_type": "month", + "period_value": [ + "2025-08" + ] + }, + "expected_row_count": 21 + } + ] } ] } diff --git a/packages/microcosm-build/src/microcosm/build/uk/qrf_tail_reviewed_exclusions.json b/packages/microcosm-build/src/microcosm/build/uk/qrf_tail_reviewed_exclusions.json index 5150ca9e2..c95b8a07b 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/qrf_tail_reviewed_exclusions.json +++ b/packages/microcosm-build/src/microcosm/build/uk/qrf_tail_reviewed_exclusions.json @@ -1,5 +1,13 @@ { "schema_version": 2, "description": "Reviewed QRF tail-concentration exclusions for the UK terminal battery (#609). Each entry maps a declared fit_weighted_qrf_stage* output column to a non-empty reason naming the tracked defect or the genuinely concentrated instrument. The shared gate reports dormant entries (columns absent from the declared surface) and FAILS stale ones (columns now below the concentration threshold), so this register cannot rot. Empty until an incident or adjudication earns an entry.", - "exclusions": {} + "exclusions": { + "charitable_investment_gifts": { + "reason": "A genuinely concentrated instrument: the SPI stage-1 QRF leaf for gifts of investments to charity has 168 weighted carriers on the 2026-09-30 national build, so its top 100 weighted records carry the whole mass by construction (the published SPI population itself is a few thousand donors of listed shares and property). Not the microcosm#462 broadcast signature: the carriers are SPI-channel rows drawn from the tape's own sparse leaf, not a point mass copied across a donor tail. An earlier entry for this column was retired on 2026-09-21 when the first national candidate measured it inside the fence; the 2026-09-30 build measures it at the threshold again, and the gate fails this entry as stale the day a build brings it back inside.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 c9 (certifier rehearsal R5 on the 2026-09-30 build, 2026-10-01; ruling 2026-10-02 in session)", + "approved_on": "2026-10-02", + "expires_on": "2027-04-02" + } + } } diff --git a/packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json b/packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json index 1edc791c4..b98488cfe 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json +++ b/packages/microcosm-build/src/microcosm/build/uk/release_input_coverage_manifest.json @@ -468,7 +468,7 @@ "required_mass_change_reason": "Capital-gains incidence anchor moves the mass of non-liable clone households back to their paired originals until the sub-exempt and loss-making clone mass match the Advani-Summers reporter composition at the redrawn liable mass; every pair's mass and the total household mass are conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465", "survey": "Family Resources Survey 2024-25, SPI synthetic support, the HMRC Table 3 redrawn gains on the spine and Advani-Summers capital-gains incidence" @@ -491,7 +491,7 @@ "capital_gains" ], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465", "survey": "Family Resources Survey 2024-25, SPI synthetic support, and Advani-Summers capital-gains incidence" @@ -499,6 +499,29 @@ "stage": "cgt_incidence_clone", "status": "required_at_build" }, + "cgt_residential_split": { + "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833", + "base_candidate_tier": "frs", + "effective_mass_requirements": {}, + "mass_change_semantics": "mass_conserving", + "output_weight_kind": "importance", + "outputs": [ + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", + "cgt_residential_probability", + "capital_gains_residential_property" + ], + "required_mass_change_reason": "Capital-gains residential split divides every household carrying liable gainers into arms weighted by the solved residential probabilities, one arm per subset of residential gainers; every household's mass and the total household mass are conserved.", + "rewrites": [], + "source_manifest": "source_stages.json", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", + "source_vintages": { + "source": "https://www.gov.uk/government/statistics/capital-gains-tax-statistics", + "survey": "HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative) on the amounts the Table 3 redraw placed" + }, + "stage": "cgt_residential_split", + "status": "required_at_build" + }, "cgt_support_split": { "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833", "base_candidate_tier": "frs", @@ -513,7 +536,7 @@ "required_mass_change_reason": "Capital-gains support split divides the wealthiest households of each HMRC Table 3 income band into light copies at equal weight; every household's mass and the total household mass are conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "HMRC Capital Gains Tax statistics, July 2026 release, Table 3 for 2024-25, via the vendored conditioning facts", "survey": "Family Resources Survey 2024-25, SPI synthetic support and HMRC Capital Gains Tax statistics Table 3 (2024-25 individuals by size of gain and taxable income)" @@ -542,7 +565,7 @@ "required_mass_change_reason": "ETB public-services imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "UK Data Service SN 8856 Effects of Taxes and Benefits household tab, DfT rail fare index, and public NHS activity/cost table.", "survey": "Effects of Taxes and Benefits 1977-2024 and NHS age-gender public table" @@ -562,7 +585,7 @@ "required_mass_change_reason": "ETB VAT expenditure-rate imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "UK Data Service SN 8856 Effects of Taxes and Benefits household tab and cited VAT anchor resource.", "survey": "Effects of Taxes and Benefits 1977-2024" @@ -578,13 +601,12 @@ "output_weight_kind": "importance", "outputs": [ "capital_gains_asset_type", - "capital_gains_residential_property", "capital_gains_badr" ], "required_mass_change_reason": "CGT asset-type assignment on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "https://www.gov.uk/government/statistics/capital-gains-tax-statistics", "survey": "HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative), Table 4.1 (individuals claiming Business Asset Disposal Relief or Investors' Relief by band of qualifying gain, 2024-25, administrative) and Table 7 (disposals, proceeds and gains by asset type, 2023-24, sample-based), vendored from the pinned Chronicle feed" @@ -608,7 +630,7 @@ "capital_gains" ], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "conditioning_resource": "hmrc_cgt_conditioning_facts.json", "conditioning_resource_sha256": "067ba21c12aeb4e95e6a8a16141dbaa075e8787ed875a005f549cab6e91a6b56", @@ -697,7 +719,7 @@ "full_frs_tei_band_unavailable" ], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "hmrc_surface": "2023-24", "mapped_build_period": "2024", @@ -738,7 +760,7 @@ "required_mass_change_reason": "LCFS consumption imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "UK Data Service SN 9468 Living Costs and Food Survey 2023-24 household/person tabs and the vendored Chronicle facts for NEED 2023 mean kWh, the FY2024-25 Ofgem cap levels, road-fuel outturn (DESNZ, HMRC, ONS), the DfT VEH1103 licensed-car stock, NTS bus use and the DfT and devolved bus finance tables.", "survey": "Living Costs and Food Survey 2023-24" @@ -764,7 +786,7 @@ "required_mass_change_reason": "NTS bus-travel imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "Department for Transport National Travel Survey, UK Data Service SN 5340 (End User Licence, 19th edition, November 2025), DOI 10.5255/UKDA-SN-5340-19; local licensed tab files (household, individual, trip); England residents only from 2013.", "survey": "National Travel Survey 2002-2024" @@ -782,10 +804,13 @@ "required_mass_change_reason": "Regional property uprating on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [ "main_residence_value", - "property_wealth" + "property_wealth", + "owned_land", + "other_residential_property_value", + "non_residential_property_value" ], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "MHCLG dwellings and ONS UK House Price Index December 2025 regional average prices.", "survey": "Public regional property reference" @@ -802,7 +827,8 @@ "outputs": [ "pension_contributions_via_salary_sacrifice", "employee_pension_contributions", - "employment_income" + "employment_income", + "salary_sacrifice_pre_conversion_pay" ], "required_mass_change_reason": "Salary-sacrifice support stage rewrites pension columns and converted records' pay; household rows and typed household weights pass through and total household mass is conserved.", "rewrites": [ @@ -811,7 +837,7 @@ "employment_income" ], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "HMRC, Salary sacrifice reform for pension contributions effective from 6 April 2029", "survey": "Family Resources Survey 2024-25 salary-sacrifice respondents and HMRC salary-sacrifice reform analysis" @@ -819,6 +845,39 @@ "stage": "salary_sacrifice", "status": "required_at_build" }, + "spi_income_band_donors": { + "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833", + "base_candidate_tier": "frs", + "effective_mass_requirements": {}, + "mass_change_semantics": "mass_conserving", + "output_weight_kind": "importance", + "outputs": [ + "household_is_spi_income_band_donor", + "spi_income_band_donor_lower_bound", + "person_is_spi_income_band_carrier" + ], + "required_mass_change_reason": "Reallocate the published HMRC Table 2.5 taxpayer mass of each total-income band from GBP 200,000 from the households of each donor's region to its income band donor support rows; every region's household mass and the total national mass are conserved.", + "rewrites": [ + "household_is_spi_synthetic", + "person_support_channel", + "person_support_clone_index", + "person_source_id", + "benunit_support_channel", + "benunit_support_clone_index", + "benunit_source_id", + "household_support_channel", + "household_support_clone_index", + "household_source_id" + ], + "source_manifest": "source_stages.json", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", + "source_vintages": { + "source": "Reserved SPI support households for the HMRC Table 2.5 total-income bands from GBP 200,000, stacked at the published band taxpayer count over the donors per band (PolicyEngine/chronicle#280 lane).", + "survey": "Survey of Personal Incomes Public Use Tape 2022-23 and HMRC Income Tax liabilities statistics July 2026 Table 2.5" + }, + "stage": "spi_income_band_donors", + "status": "required_at_build" + }, "student_loans": { "base_candidate_sha256": "f17306ccb2aad7ff0130be3589b560afb2e2a12a943570911cd0c77f07934833", "base_candidate_tier": "frs", @@ -833,7 +892,7 @@ "student_loan_plan" ], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "Explore Education Statistics Table 6a, Higher education total", "survey": "Family Resources Survey 2024-25 and Student Loans Company borrower forecasts for England" @@ -855,7 +914,7 @@ "required_mass_change_reason": "WAS Lifetime ISA imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "Office for National Statistics Wealth and Assets Survey, UK Data Service SN 7215, DOI 10.5255/UKDA-SN-7215-20; local licensed round-8 person and household tabs (interviews April 2020 to March 2022, Great Britain).", "survey": "Wealth and Assets Survey round 8" @@ -890,7 +949,7 @@ "required_mass_change_reason": "WAS wealth imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", "rewrites": [], "source_manifest": "source_stages.json", - "source_manifest_sha256": "c461c338a92a48409c1537bd1c142852f750ab2529b302fa74bf08af32da4752", + "source_manifest_sha256": "90429a0d9fa3092257efc2809a35eb745e2ac8cf60bbed92c29da7fda07a05eb", "source_vintages": { "source": "Office for National Statistics Wealth and Assets Survey, UK Data Service SN 7215, DOI 10.5255/UKDA-SN-7215-20; local licensed 2006-22 household tab.", "survey": "Wealth and Assets Survey round 8" diff --git a/packages/microcosm-build/src/microcosm/build/uk/source_stages.json b/packages/microcosm-build/src/microcosm/build/uk/source_stages.json index 27840af3a..73ea7e2c7 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/source_stages.json +++ b/packages/microcosm-build/src/microcosm/build/uk/source_stages.json @@ -1497,22 +1497,36 @@ 1000000, 2000000 ], - "donors_per_band": 120, + "minimum_donors_per_band": 120, + "maximum_donor_weight": 250.0, + "count_rule": "max(minimum_donors_per_band, ceil(published band taxpayers / maximum_donor_weight))", + "weight_rule": "published band taxpayers / band donor count, equal within a band", "expected_band_count": 4, - "expected_donor_count": 480, "candidate_population": "raw FRS base channel adults aged 16 and over", - "candidate_order": "person_id ascending", - "draw": "weighted_without_replacement", + "candidate_order": "person_id ascending breaks equal draw keys", + "draw": "weighted_without_replacement_identity_keyed: Efraimidis-Spirakis keys log(u) / (household weight x propensity) on stable_identity_uniforms(person_id, seed, salt:band); the largest keys are seated, one carrier per household, the highest band first", "propensity": "SPI 2022-23 FACT-weighted band share by region, sex and age band", + "tape_preparation": "as hmrc_spi_income_spine prepares the tape: its stage-1 seed and draw_spi_donor_ages_by_population", + "tape_preparation_seed": 42, "seed": 3, + "salt": "spi_income_band_donor_draw", "clone_index": 2, "flag_column": "household_is_spi_income_band_donor", "band_column": "spi_income_band_donor_lower_bound", "carrier_column": "person_is_spi_income_band_carrier", - "initial_weight": "published band taxpayers / donors_per_band", + "funding": "stratum reallocation: every incumbent household of the stratum a donor's source household belongs to is scaled by 1 - donor mass / stratum mass", + "funding_strata": [ + "region" + ], + "maximum_reallocation_share": 0.5, + "maximum_donor_household_share": 0.15, + "seating_scale": "min(1, maximum_donor_household_share x eligible base households / expected donor count); band counts floored, at least one per band", + "mass_scale": "min(1, min over funding strata of maximum_reallocation_share x stratum mass / seated donor mass); every band's weight scaled alike", "never_zero_weight": true, "weight_kind_out": "importance", - "reason": "SPI income band donors stack positive-weight support households per HMRC Table 2.5 total-income band from GBP 200,000, one carrier adult each; the published band taxpayer mass is added explicitly and the base and synthetic channels keep their weights." + "conservation": "exact_total", + "declared_factor": 1.0, + "reason": "Reallocate the published HMRC Table 2.5 taxpayer mass of each total-income band from GBP 200,000 from the households of each donor's region to its income band donor support rows; every region's household mass and the total national mass are conserved." } ], "outputs": [ @@ -1535,7 +1549,7 @@ "nonnegative_outputs": [ "spi_income_band_donor_lower_bound" ], - "notes": "Reserves 120 SPI-channel households per Table 2.5 band from GBP 200,000 (480 in all) so the top of the income distribution has support before calibration. Carriers are FRS adults drawn by the tape's own band propensity given region, sex and age band, their households are copied whole at clone index 2 with the synthetic flag, and each starts at the build tax year's published taxpayers over 120. The income stage gives every carrier a band-conditional draw from the tape (resample_band_donor_leaves)." + "notes": "Reserves SPI-channel households for every Table 2.5 band from GBP 200,000 so the top of the income distribution has support before calibration. A band seats enough donors that none starts above the maximum weight and never fewer than the minimum, at equal weights of the build tax year's published taxpayers over the band's count. Carriers are FRS adults seated by identity-keyed draws on household weight times the tape's own band propensity given region, sex and age band; their households are copied whole at clone index 2 with the synthetic flag. The mass the donors carry is taken in proportion from the households of each donor's region, so the national total and every region's mass are unchanged (microcosm#1063). The income stage gives every carrier a band-conditional draw from the tape (resample_band_donor_leaves)." }, { "stage": "hmrc_spi_income_spine", @@ -1651,6 +1665,29 @@ { "kind": "fit_weighted_qrf_stage1", "training_artifact_role": "qrf_donor", + "recipient_minimum_age": 16, + "rebase_income_to_build_period": true, + "donor_income_period": 2022, + "income_uprating_variables": { + "self_employment_income": "self_employment_income", + "savings_interest_income": "savings_interest_income", + "dividend_income": "dividend_income", + "private_pension_income": "private_pension_income", + "property_income": "property_income", + "other_investment_income": "other_investment_income", + "gift_aid": null, + "charitable_investment_gifts": null, + "hmrc_spi_pay": "employment_income_before_lsr", + "hmrc_spi_employment_benefits": "employment_income_before_lsr", + "hmrc_spi_employment_expenses": "employment_income_before_lsr", + "hmrc_spi_incapacity_benefit_income": null, + "hmrc_spi_other_social_security_income": null, + "hmrc_spi_taxable_termination_pay": "employment_income_before_lsr", + "hmrc_spi_unemployment_benefit_income": null, + "hmrc_spi_miscellaneous_employment_income": "employment_income_before_lsr", + "hmrc_spi_other_income": "miscellaneous_income", + "hmrc_spi_state_pension_income": null + }, "predictors": [ "age", "gender", @@ -1802,30 +1839,7 @@ "charitable_investment_gifts": 0.0 }, "n_estimators": 100, - "seed": 42, - "donor_income_period": 2022, - "income_uprating_variables": { - "charitable_investment_gifts": null, - "dividend_income": "dividend_income", - "gift_aid": null, - "hmrc_spi_employment_benefits": "employment_income_before_lsr", - "hmrc_spi_employment_expenses": "employment_income_before_lsr", - "hmrc_spi_incapacity_benefit_income": null, - "hmrc_spi_miscellaneous_employment_income": "employment_income_before_lsr", - "hmrc_spi_other_income": "miscellaneous_income", - "hmrc_spi_other_social_security_income": null, - "hmrc_spi_pay": "employment_income_before_lsr", - "hmrc_spi_state_pension_income": null, - "hmrc_spi_taxable_termination_pay": "employment_income_before_lsr", - "hmrc_spi_unemployment_benefit_income": null, - "other_investment_income": "other_investment_income", - "private_pension_income": "private_pension_income", - "property_income": "property_income", - "savings_interest_income": "savings_interest_income", - "self_employment_income": "self_employment_income" - }, - "rebase_income_to_build_period": true, - "recipient_minimum_age": 16 + "seed": 42 }, { "kind": "resample_band_donor_leaves", @@ -1842,6 +1856,9 @@ "age_pool_minimum": 20, "weighting": "FACT", "with_replacement": true, + "draw": "identity_keyed_inverse_cdf: stable_identity_uniforms(draw_key, seed, salt) over the pool's cumulative FACT in tape order", + "draw_key": "person_source_id", + "salt": "spi_income_band_donor_leaf_draw", "outputs": "stage-1 outputs, uprated as the stage-1 draws", "seed": 44 }, @@ -2288,12 +2305,41 @@ "notes": "Runs the SPI-trained income QRFs on the raw-spine support channel, initializes FRS charity columns to zero, trains FRS-only stage 2 before redrawing base-channel dividends, and emits a sidecar-only 208-fact replay report for the spine path." }, { - "artifacts": [], + "stage": "spi_housing_shell", + "survey": "Family Resources Survey 2024-25", + "source": "Family Resources Survey 2024-25 base households' housing (tenure, dwelling type, bedrooms, council tax, housing costs and the housing-linked benefit reports), imputed onto the SPI support households from their own region, composition and post-SPI incomes.", "grain": "household", - "notes": "Runs after the SPI income chain and before the income-conditioned donor stages, the household counterpart of the SPI stage-2 FRS-only fill. The support copies and income band donors are whole FRS households, so without this stage each kept its FRS parent's housing, drawn for the parent's income. Trained on the FRS base households (positive household weight) and drawn for every SPI-channel household from engine-free predictors (region, household type, adults, children, single-adult status, reference-person age and household income components). Tenure, dwelling type, bedrooms and council-tax band are drawn in that order by weighted multiclass classifiers with an inverse-CDF draw; the housing amounts and the head benefit unit's Housing Benefit and the reference person's council tax benefit by one chained regime-gated QRF conditioned on the drawn categories, clipped to the training range. Every draw reads uniforms keyed on household_id. An amount is zero wherever the training households have no positive value for its tenure or region; mortgage payments need a mortgaged tenure (true of every FRS household); Housing Benefit needs a rented tenure (the engine pays it to renting reporters only); council tax benefit is capped at council tax and the household council_tax_rebate equals it, both on the reference person as the FRS holds them. Region, single-adult status, BRMA (region and the recipient's own LHA category), the flat-rate household draws and composition are kept; FRS base rows are byte-unchanged.", + "artifacts": [], "operations": [ { - "amount_fit": "regime_gated_qrf", + "kind": "impute_spi_housing_shell", + "training_rows": "base_frs_channel_positive_weight", + "rows": "spi_channel", + "identity": "household_id", + "predictors": [ + "region", + "ons_household_type", + "num_adults", + "num_children", + "single_adult", + "hrp_age", + "employment_income", + "self_employment_income", + "private_pension_income", + "investment_income", + "state_pension_reported" + ], + "categorical_predictors": [ + "region", + "ons_household_type" + ], + "categorical_targets": [ + "tenure_type", + "accommodation_type", + "num_bedrooms", + "council_tax_band" + ], + "categorical_fit": "weighted_hist_gradient_boosting_classifier", "amount_targets": [ "council_tax", "council_tax_reported", @@ -2310,48 +2356,21 @@ "housing_benefit_reported", "council_tax_benefit_reported" ], - "categorical_fit": "weighted_hist_gradient_boosting_classifier", - "categorical_predictors": [ - "region", - "ons_household_type" - ], - "categorical_targets": [ + "amount_fit": "regime_gated_qrf", + "support_clip": "donor_range", + "structural_zero_keys": [ "tenure_type", - "accommodation_type", - "num_bedrooms", - "council_tax_band" - ], - "identity": "household_id", - "kind": "impute_spi_housing_shell", - "landing": "household_reference_person", - "n_estimators": 100, - "predictors": [ - "region", - "ons_household_type", - "num_adults", - "num_children", - "single_adult", - "hrp_age", - "employment_income", - "self_employment_income", - "private_pension_income", - "investment_income", - "state_pension_reported" + "region" ], - "rows": "spi_channel", "rules": [ "mortgage_requires_mortgaged_tenure", "housing_benefit_requires_rented_tenure", "council_tax_benefit_capped_at_council_tax", "council_tax_rebate_equals_hrp_council_tax_benefit" ], - "seed": 0, - "structural_zero_keys": [ - "tenure_type", - "region" - ], - "support_clip": "donor_range", - "training_rows": "base_frs_channel_positive_weight" + "landing": "household_reference_person", + "n_estimators": 100, + "seed": 0 } ], "outputs": [], @@ -2374,9 +2393,7 @@ "housing_benefit_reported", "council_tax_benefit_reported" ], - "source": "Family Resources Survey 2024-25 base households' housing (tenure, dwelling type, bedrooms, council tax, housing costs and the housing-linked benefit reports), imputed onto the SPI support households from their own region, composition and post-SPI incomes.", - "stage": "spi_housing_shell", - "survey": "Family Resources Survey 2024-25" + "notes": "Runs after the SPI income chain and before the income-conditioned donor stages, the household counterpart of the SPI stage-2 FRS-only fill. The support copies and income band donors are whole FRS households, so without this stage each kept its FRS parent's housing, drawn for the parent's income. Trained on the FRS base households (positive household weight) and drawn for every SPI-channel household from engine-free predictors (region, household type, adults, children, single-adult status, reference-person age and household income components). Tenure, dwelling type, bedrooms and council-tax band are drawn in that order by weighted multiclass classifiers with an inverse-CDF draw; the housing amounts and the head benefit unit's Housing Benefit and the reference person's council tax benefit by one chained regime-gated QRF conditioned on the drawn categories, clipped to the training range. Every draw reads uniforms keyed on household_id. An amount is zero wherever the training households have no positive value for its tenure or region; mortgage payments need a mortgaged tenure (true of every FRS household); Housing Benefit needs a rented tenure (the engine pays it to renting reporters only); council tax benefit is capped at council tax and the household council_tax_rebate equals it, both on the reference person as the FRS holds them. Region, single-adult status, BRMA (region and the recipient's own LHA category), the flat-rate household draws and composition are kept; FRS base rows are byte-unchanged." }, { "stage": "was_wealth", @@ -2415,7 +2432,10 @@ "net_financial_wealth": "HFINWNTR8_Sum", "net_financial_wealth_exsl": "HFINWNTR8_exSLC_Sum", "mortgage_debt": "HMortGR8", + "main_residence_mortgage": "TotMortR8", + "other_property_mortgage": "OthMortR8_sum", "tenure_code": "Ten1R8", + "private_rent_code": "DVPriRntR8", "main_residence_value": "DVhvalueR8", "other_residential_property_value": "DVHseValR8_sum", "non_residential_property_value": "DVBlDValR8_sum", @@ -2430,7 +2450,9 @@ "cash_isa": "DVCISAVR8_aggr", "student_loan_balance": "Tot_LosR8_aggr - Tot_los_exc_SLCR8_aggr", "consumer_debt": "max(HFINWR8_SUM - HFINWNTR8_exSLC_Sum, 0)", - "has_mortgage_tenure": "Ten1R8 in {2, 3}", + "tenure_category": "Ten1R8 1 owned_outright; 2 or 3 owned_with_mortgage; 4 or 5 private_rent when DVPriRntR8 == 1, otherwise social_rent; any other code private_rent, counted in the receipt", + "other_property_value": "DVPropertyR8 - (DVhvalueR8 + DVHseValR8_sum + DVBlDValR8_sum + DVLUKValR8_sum), refused when negative", + "other_financial_assets": "HFINWR8_SUM - (DVSaValR8_aggr + DVCISAVR8_aggr + corporate_wealth), refused when negative", "is_renting": "DVPriRntR8 == 1", "region": "GORR8 via incumbent REGIONS map" } @@ -2444,8 +2466,7 @@ "private_pension_income", "employment_income", "self_employment_income", - "capital_income", - "is_renting" + "capital_income" ], "consumed_only": true }, @@ -2464,12 +2485,16 @@ "capital_income", "num_bedrooms", "council_tax", - "is_renting", + "tenure_owned_with_mortgage", + "tenure_private_rent", + "tenure_social_rent", "region" ], "categorical_predictors": [ "region", - "is_renting" + "tenure_owned_with_mortgage", + "tenure_private_rent", + "tenure_social_rent" ], "region_remap": { "NORTHERN_IRELAND": "WALES" @@ -2477,27 +2502,70 @@ "integer_outputs": [ "num_vehicles" ], - "debt_segment_predictors": [ - "has_mortgage_tenure" + "stratified_targets": { + "main_residence_value": [ + "owned_outright", + "owned_with_mortgage" + ], + "main_residence_mortgage": [ + "owned_with_mortgage" + ] + }, + "internal_components": [ + "other_property_value", + "other_financial_assets", + "other_mortgage" ], + "derived_totals": { + "property_wealth": [ + "owned_land", + "main_residence_value", + "other_residential_property_value", + "non_residential_property_value", + "other_property_value" + ], + "corporate_wealth": [ + "corporate_wealth_excl_isa", + "stocks_and_shares_isa" + ], + "gross_financial_wealth": [ + "savings", + "cash_isa", + "corporate_wealth", + "other_financial_assets" + ], + "net_financial_wealth": [ + "gross_financial_wealth", + "-consumer_debt", + "-student_loan_balance" + ], + "mortgage_debt": [ + "main_residence_mortgage", + "other_mortgage" + ] + }, "chain_order": [ "owned_land", + "main_residence_value", + "other_residential_property_value", + "non_residential_property_value", + "other_property_value", "property_wealth", "private_pension_wealth", "corporate_wealth_excl_isa", "stocks_and_shares_isa", "corporate_wealth", - "gross_financial_wealth", - "net_financial_wealth", - "main_residence_value", - "other_residential_property_value", - "non_residential_property_value", "savings", + "cash_isa", + "other_financial_assets", "num_vehicles", "student_loan_balance", - "cash_isa", + "gross_financial_wealth", + "main_residence_mortgage", + "other_mortgage", "mortgage_debt", - "consumer_debt" + "consumer_debt", + "net_financial_wealth" ] }, { @@ -2572,7 +2640,7 @@ "mortgage_debt", "consumer_debt" ], - "notes": "Ports incumbent WAS round-8 wealth imputation with signed E5 differences: exact lower-case column matching replaces the fuzzy r/w fallback; cash ISA uses DVCISAVR8_aggr and stocks-and-shares ISA uses DVIISAVR8_aggr; private pension wealth (WAS total private pension wealth less the current defined-benefit component; DB-type components at the SCAPE rate, money-purchase components at reported fund values) is drawn as its own output private_pension_wealth and no longer folded into corporate_wealth, which keeps only the share-like holdings - pension rights are disregarded capital in every means test (UC Regs 2013 Sch 10 para 10 and the HB/JSA/ESA/IS/SPC parallels; uk-data#452) - and the downstream chain segment conditions on both components; each chain segment starts its own fit and draw RNG streams from a child seed derived deterministically from the declared seed (start_chain respawns from the model seed on every call, so one shared model coupled the k-th target of every segment); corporate_wealth folds stocks-and-shares ISA after drawing corporate_wealth_excl_isa; recipient Northern Ireland regions are mapped to Wales for prediction only; student_loan_balance is allocated by household id rather than the incumbent positional off-by-one. The debt segment (fourth chain segment, fourth child seed) reads WAS round-8 derived variables HMortGR8 (total outstanding mortgage debt), HFINWNTR8_exSLC_Sum (net financial wealth excluding student loans; gross less it is the non-student-loan financial liabilities) and Ten1R8 (tenure; codes 2 and 3 are buying with a mortgage and part own/part rent, confirmed by the I1 audit of the pinned tab); UKDS SN 7215 documentation is the codebook. Engine predictors materialize at their native entity and person/benunit values are summed to household, reproducing the incumbent map_to=household semantics; region is one-hot encoded jointly across donor and recipient (the incumbent's dummy encoding), with unmapped donor GOR codes becoming all-zero dummy rows. The WAS and FRS predictor definitions are not fully like-for-like and are ported as-is; raw WAS missing values are blanket-filled with zero; UKDS negative sentinel codes (-9/-8/-7/-6) are recoded to zero for the nonnegative-domain columns the licensed audit found carrying them (vcarnr8: 2 rows; HBedRmR8: 95.8 percent - the bedrooms question is effectively unasked in the WAS household file, predictor-quality revisit registered on microcosm#145) - a signed difference vs the incumbent, which trains on raw sentinels; DVPriRntR8's -9 is structural not-applicable so the is_renting mapping is unchanged; genuinely negative domains are never recoded." + "notes": "Ports incumbent WAS round-8 wealth imputation with signed E5 differences: exact lower-case column matching replaces the fuzzy r/w fallback; cash ISA uses DVCISAVR8_aggr and stocks-and-shares ISA uses DVIISAVR8_aggr; private pension wealth (WAS total private pension wealth less the current defined-benefit component; DB-type components at the SCAPE rate, money-purchase components at reported fund values) is drawn as its own output private_pension_wealth and no longer folded into corporate_wealth, which keeps only the share-like holdings - pension rights are disregarded capital in every means test (UC Regs 2013 Sch 10 para 10 and the HB/JSA/ESA/IS/SPC parallels; uk-data#452) - and the downstream chain segment conditions on both components; each chain segment starts its own fit and draw RNG streams from a child seed derived deterministically from the declared seed (start_chain respawns from the model seed on every call, so one shared model coupled the k-th target of every segment); corporate_wealth folds stocks-and-shares ISA after drawing corporate_wealth_excl_isa; recipient Northern Ireland regions are mapped to Wales for prediction only; student_loan_balance is allocated by household id rather than the incumbent positional off-by-one. Tenure enters every chain segment as one four-way category built the same way on both sides (microcosm#1063): the donor's from Ten1R8 and DVPriRntR8 (owned outright; buying with a mortgage or part rent, part mortgage; rented or rent-free from a private landlord; rented from a council or housing association), the recipient's from the frame's tenure_type (council and housing association together), so the stage reads no engine tenure formula. Two columns are structural in tenure and are fitted and drawn inside their stratum only, zero by rule elsewhere: main_residence_value on the owner tenures (every WAS owner has a positive value and no renter has one) and main_residence_mortgage (TotMortR8) on the mortgaged tenure. mortgage_debt is HMortGR8, all of the household's mortgages, and is derived as the main-residence mortgage plus the mortgages on other property (other_mortgage, HMortGR8 less TotMortR8 on the donor: buy-to-let, second homes and the rest), which are held on every tenure and drawn without a stratum, so an outright owner's or a renter's buy-to-let keeps its mortgage beside the property it is secured on (OthMortR8_sum is read for the receipt only); the donor's conditioning copy carries the same rule, so consumer_debt is fitted on values recipients can hold. Totals are not drawn: property_wealth is the sum of land, the main residence, other houses, buildings and a drawn remainder (buy-to-let, overseas and other property), gross_financial_wealth is savings, the cash ISA, the share-like holdings and a drawn remainder, and net_financial_wealth is gross less consumer debt and the student loan balance - identities the pinned tab satisfies on every row and the cleaning step refuses a donor that breaks. A sum of donor-valued components that leaves the donor's range for the total gives up the excess from the remainder first, and the receipt counts the rows. Eight chain segments, each with its own child seed: land; the main residence; the other property components; pension and share-like wealth; the financial components, vehicles and the student loan balance; the main-residence mortgage; the other mortgages; consumer debt. HFINWNTR8_exSLC_Sum is net financial wealth excluding student loans, so gross less it is the non-student-loan financial liabilities; UKDS SN 7215 documentation is the codebook. Engine predictors materialize at their native entity and person/benunit values are summed to household, reproducing the incumbent map_to=household semantics; region is one-hot encoded jointly across donor and recipient (the incumbent's dummy encoding), with unmapped donor GOR codes becoming all-zero dummy rows. The WAS and FRS predictor definitions are not fully like-for-like and are ported as-is; raw WAS missing values are blanket-filled with zero; UKDS negative sentinel codes (-9/-8/-7/-6) are recoded to zero for the nonnegative-domain columns the licensed audit found carrying them (vcarnr8: 2 rows; HBedRmR8: 95.8 percent - the bedrooms question is effectively unasked in the WAS household file, predictor-quality revisit registered on microcosm#145) - a signed difference vs the incumbent, which trains on raw sentinels; DVPriRntR8's -9 is structural not-applicable so the private-renter reading is unchanged; genuinely negative domains are never recoded." }, { "stage": "was_lisa", @@ -2627,7 +2695,7 @@ "lifetime_isa_balance": "DVFLISAvR8" }, "household_aggregate": "DVFLISAVR8_aggr", - "aggregate_tolerance_gbp": 1, + "aggregate_tolerance_gbp": 1.0, "non_dependent_code": 2, "female_code": 2, "age_band_edges": [ @@ -2669,7 +2737,7 @@ ], "credibility_rule": { "maximum_holder_age_band_lower": 45, - "balance_ceiling_gbp": 40000, + "balance_ceiling_gbp": 40000.0, "basis": "LISAs have been opened only by people aged 18 to 39 since 6 April 2017, so no holder was older than 44 by the end of round-8 fieldwork (March 2022); a holder recorded in an age band starting at 45 or above is a misreported ISA type and is recoded to a non-holder. About GBP 30,500 of contributions, government bonus and the one-off 2017-18 Help to Buy ISA transfer was attainable before returns, so a holder above GBP 40,000 stays a holder but leaves the balance fit." } }, @@ -2682,8 +2750,7 @@ "private_pension_income", "employment_income", "self_employment_income", - "capital_income", - "is_renting" + "capital_income" ], "consumed_only": true }, @@ -2693,7 +2760,7 @@ "seed": 0, "salt": "was_lisa:has_lifetime_isa", "model": "weighted_ridge_logistic", - "penalty_c": 1, + "penalty_c": 1.0, "solver": "lbfgs", "max_iter": 5000, "standardise": "donor_unweighted_mean_sd", @@ -2767,7 +2834,7 @@ "kind": "record_mass_conservation_receipt", "entity": "household", "reason": "WAS Lifetime ISA imputation on the source spine: household weights pass through unchanged and total household mass is conserved.", - "declared_factor": 1, + "declared_factor": 1.0, "gate_coupling": "The terminal family gate requires a valid mass-conserving MassChangeRecord carrying exactly this stage-specific reason." } ], @@ -2820,9 +2887,12 @@ "outputs": [], "rewrites": [ "main_residence_value", - "property_wealth" + "property_wealth", + "owned_land", + "other_residential_property_value", + "non_residential_property_value" ], - "notes": "Deterministically rescales owner rows so regional unweighted owner means match the public house-price reference. Northern Ireland has no reference row and is never scaled; empty and nonpositive regions are skipped. The unweighted mean follows the incumbent behavior." + "notes": "Deterministically rescales owner rows so regional unweighted owner means match the public house-price reference. Northern Ireland has no reference row and is never scaled; empty and nonpositive regions are skipped. The unweighted mean follows the incumbent behavior. The factor is read off the main residence and has always scaled the property total; since microcosm#1063 it scales the total's other components in the frame as well (land, other houses, buildings), so an owner's property_wealth stays the sum of its components after the uprating. Mortgage debt is never uprated." }, { "stage": "nts_bus_travel", @@ -3339,7 +3409,8 @@ "LCFS tenure 4 and 8 -> RENT_PRIVATELY", "LCFS accommodation 4 and 5 -> FLAT" ], - "logged_dropna_row_count": true + "logged_dropna_row_count": true, + "floor": 0.0 }, { "kind": "uprate_donor_columns", @@ -4496,6 +4567,78 @@ ], "notes": "Runs after the SPI income chain and UC capital coherence, before CGT cloning (microcosm#1069 R9). One temporary engine materialization gives each benefit unit's pre-take-up Pension Credit entitlement (guarantee_credit, savings_credit, is_pension_credit_eligible; consumed only). Entitled units fall into DWP's take-up components, Guarantee Credit (with or without Savings Credit) and Savings Credit only; within a component a reporter claims and the other entitled units claim at r = (t * E - R) / (E - R), t being DWP's financial year ending 2024 caseload take-up (69% and 37%), E the component's entitled units and R its reporters, weighted at the stage's household weights. A unit with no entitlement claims only if it reports. The uniform draw is frs_take_up's would_claim_pc identity stream, whose early draw this rewrite replaces; units with every adult at or over State Pension age are the only ones the engine finds eligible." }, + { + "stage": "child_benefit_take_up", + "survey": "Family Resources Survey 2024-25", + "source": "HMRC Child Benefit statistics, annual release at August 2025 (percentage of eligible children for whom Child Benefit is claimed by single year of age; families registered and families opted out of payment), applied to PolicyEngine-UK Child Benefit eligibility and adjusted net income after the SPI income chain.", + "grain": "benunit", + "artifacts": [ + { + "role": "child_benefit_statistics", + "kind": "public_aggregate_reference", + "resource": "hmrc_child_benefit_statistics.json", + "format": "json", + "runtime_sha256_required": true + } + ], + "operations": [ + { + "kind": "materialize_rules_engine_predictors", + "predictors": [ + "is_child_or_qualifying_young_person_for_child_benefit", + "adjusted_net_income" + ], + "consumed_only": true + }, + { + "kind": "aggregate_person_to_benunit", + "method": "any_positive", + "consumed_only": true, + "aggregates": { + "child_benefit_reported_anchor": "child_benefit_reported" + } + }, + { + "kind": "assign_family_claims_by_child_age", + "output": "would_claim_child_benefit", + "draw": "would_claim_child_benefit", + "seed": 0, + "anchor": "child_benefit_reported_anchor", + "eligibility": "is_child_or_qualifying_young_person_for_child_benefit", + "family_key": "age of the benefit unit's eldest eligible child", + "resource": "hmrc_child_benefit_statistics.json", + "rate_measure": "claim_rate", + "rate_period": "2025-05", + "weight_mapping": "household_to_benunit", + "rule": "a reporter claims; a non-reporting family whose eldest eligible child is aged g claims at r_g = (t_g * C_g - R_g - sum over k > g of N_gk * r_k) / N_gg, weighted, solved from the eldest age down and clipped to [0, 1]" + }, + { + "kind": "assign_opt_out_by_charge_income", + "output": "child_benefit_opts_out", + "draw": "child_benefit_opts_out", + "seed": 0, + "resource": "hmrc_child_benefit_statistics.json", + "numerator_measure": "families_opted_out_claimant_all", + "denominator_measure": "families_registered", + "caseload_period": "2025-08", + "charge_income": "adjusted_net_income", + "charge_parameters": "gov.hmrc.income_tax.charges.CB_HITC", + "pools": [ + "fully_charged", + "taper" + ], + "weight_mapping": "household_to_benunit", + "rule": "published opted-out share of registered families, of the claiming families; drawn among claiming non-reporting families with the highest adjusted net income at or above the full-charge income, then, for any remainder, among those inside the taper", + "payment_rule": "would_claim_child_benefit = claims and not opted out on families with an eligible child; a benefit unit without one keeps its early draw" + } + ], + "outputs": [], + "rewrites": [ + "would_claim_child_benefit", + "child_benefit_opts_out" + ], + "notes": "Runs after the SPI income chain, UC capital coherence and the Pension Credit redraw, before CGT cloning (microcosm#1063). frs_take_up draws would_claim_child_benefit early at one flat rate over every benefit unit with the reporters on top, and child_benefit_opts_out at a flat rate whatever the unit's income; this stage redraws both. One temporary engine materialization gives each person's eligibility as a child or qualifying young person and each person's adjusted net income (consumed only). A family claims for all its children or none, so the draw is by family, keyed on the age of its eldest eligible child: a reporter claims, and the other families claim at rates solved from the eldest age down so that the claimed share of the eligible children of each single year of age is HMRC's published one (Table 15, May 2025, opted-out families included), weighted at the stage's household weights. The published share of registered families that opted out of payment (August 2025) of the claiming families then opts out, drawn among claiming families that do not report receipt and whose highest adjusted net income is at or above the income at which the High Income Child Benefit Charge takes the whole benefit, then inside the taper for any remainder. The engine pays on would_claim_child_benefit alone, so the stage stores claims-and-not-opted-out there and the opt-out in child_benefit_opts_out; a benefit unit with no eligible child keeps its early draw and is never opted out." + }, { "stage": "uc_deduction_attributes", "survey": "DWP Universal Credit deductions statistics, March 2025 to February 2026", @@ -4925,6 +5068,172 @@ ], "notes": "Spine manifest projection of the merged HMRC Table 3 amounts stage. It deliberately omits base_candidate and verify_certified_candidate, which belong only to the certified-H5 path. The allocation targets are raked to the 2024-25 Table 6 age and Table 5 region margins from the vendored conditioning resource and walked inside income x age x region cells, so large gains land on the kind of person who holds them (microcosm#725)." }, + { + "stage": "cgt_incidence_anchor", + "survey": "Family Resources Survey 2024-25, SPI synthetic support, the HMRC Table 3 redrawn gains on the spine and Advani-Summers capital-gains incidence", + "source": "Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465", + "grain": "household", + "artifacts": [ + { + "role": "capital_gains_incidence_and_quantiles", + "kind": "public_aggregate_reference", + "resource": "advani_summers_capital_gains_distribution.json", + "format": "json", + "runtime_sha256_required": true + }, + { + "role": "policy_parameters", + "kind": "versioned_parameter_tree", + "dependency": "policyengine-uk>=2.98.0 via microcosm-build[uk]", + "parameters": [ + "gov.hmrc.cgt.annual_exempt_amount" + ], + "instant_rule": "raw dated parameter files evaluated at 1 June of the build period's tax year", + "runtime_sha256_required": false, + "dependency_discipline": "deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time" + } + ], + "operations": [ + { + "kind": "pair_clone_households_to_originals", + "clone_flag_column": "household_is_capital_gains_clone", + "id_remapping": "id_multiplier_for_values", + "pairing": "clone household_id minus the clone stage's id multiplier, recomputed from the pre-clone rows (clone flag false; the support split's copies are pre-clone rows with their own clones)", + "requirement": "bijection between clone and original households; an unpaired clone or original fails the build" + }, + { + "kind": "derive_reporter_composition_from_banded_quantiles", + "resource": "advani_summers_capital_gains_distribution.json", + "income_proxy_components": [ + "employment_income", + "self_employment_income", + "state_pension_reported", + "private_pension_income", + "property_income", + "savings_interest_income", + "dividend_income", + "miscellaneous_income" + ], + "allowance_subtraction": false, + "carrier": "oldest adult; person_id ascending breaks age ties", + "adult_minimum_age": 16, + "quantile_points": [ + 0.05, + 0.1, + 0.25, + 0.5, + 0.75, + 0.9, + 0.95 + ], + "spline_degree": 1, + "extrapolation": "ext=0", + "crossings": "each band's degree-1 quantile spline inverted at zero and at the build period's annual exempt amount", + "aggregation": "crossings averaged over clone carriers weighted by household weight times the band's percent_with_gains", + "liable_mass": "weighted persons with capital_gains above the annual exempt amount on every household after the Table 3 redraw", + "targets": "sub_exempt = liable_mass * (exempt_quantile - zero_quantile) / (1 - exempt_quantile); loss = liable_mass * zero_quantile / (1 - exempt_quantile)" + }, + { + "kind": "transfer_clone_mass_to_originals", + "population": "clone households paired to their originals; originals are outside the anchor, so the groups, targets and masses are clone-side quantities, not population counts", + "sub_exempt_group": "clone households whose carrier's capital_gains lie in (0, annual exempt amount]; a carrier with exactly zero gains is no reporter and is left untouched", + "loss_group": "clone households whose carrier's capital_gains are negative", + "liable_clones": "untouched", + "factor": "min(maximum_factor, scale * percent_with_gains) with scale solved per group by bisection so the group's mass equals its target; a group at or below its target is untouched", + "maximum_factor": 1.0, + "transfer": "each clone's removed mass is added to its paired original, so every pair's mass is conserved to rounding; no global exact-total correction is applied, so no household outside a trimmed pair moves", + "weight_kind_out": "importance", + "conservation": "household_total_to_rounding", + "pair_conservation": "to_rounding", + "declared_factor": 1.0, + "reason": "Capital-gains incidence anchor moves the mass of non-liable clone households back to their paired originals until the sub-exempt and loss-making clone mass match the Advani-Summers reporter composition at the redrawn liable mass; every pair's mass and the total household mass are conserved." + } + ], + "outputs": [], + "rewrites": [], + "notes": "Spine-only weight anchor after the amounts and asset-type stages (microcosm#970). The equal-mass clone leaves half of all household mass on gainer households, so after the Table 3 redraw the clones below the exempt amount carry millions of weighted persons that HMRC publishes no count for; the anchor derives their level from the redrawn liable mass and the Advani-Summers reporter composition and moves the excess back to each clone's paired original, which is identical in every cell but the gains. No cell is written: amounts, flags, liable clones, zero-gain clone carriers are untouched; the groups, targets and receipt are clone-side quantities (originals are outside the anchor), every pair's mass and the household total are conserved, and the receipt is gated at stage time." + }, + { + "stage": "cgt_residential_split", + "survey": "HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative) on the amounts the Table 3 redraw placed", + "source": "https://www.gov.uk/government/statistics/capital-gains-tax-statistics", + "grain": "household", + "artifacts": [ + { + "role": "policy_parameters", + "kind": "versioned_parameter_tree", + "dependency": "policyengine-uk>=2.100.0 via microcosm-build[uk]", + "parameters": [ + "gov.hmrc.cgt.annual_exempt_amount" + ], + "instant_rule": "raw dated parameter files evaluated at 1 June of the build period's tax year", + "runtime_sha256_required": false, + "dependency_discipline": "deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time" + }, + { + "role": "cgt_asset_type_facts", + "kind": "public_aggregate_reference", + "resource": "hmrc_cgt_asset_type_facts.json", + "format": "json", + "runtime_sha256_required": true + } + ], + "operations": [ + { + "kind": "verify_vendored_fact_resource", + "artifact_role": "cgt_asset_type_facts", + "resource": "hmrc_cgt_asset_type_facts.json", + "feed_pin": "chronicle_feed.json", + "record_sets": [ + "hmrc.cgt_table8_2026.table8a.ty2024", + "hmrc.cgt_table8_2026.table8b.ty2024" + ], + "source_vintage": "2024-25", + "mapped_build_period": 2024, + "period_mapping": "published_tax_year_equals_build_period", + "require_before_source_read": true, + "runtime_sha256_required": true, + "fail_on_mismatch": true + }, + { + "kind": "split_liable_gainers_by_residential_probability", + "population": "persons with net capital gains above the annual exempt amount (the national taxpayer proxy)", + "model": "logistic probability in centred log gains with a stock shift, p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean log gain of the population, s the stock log-odds and h one where the stock signal holds, zero otherwise", + "stock_log_odds": 1.6, + "stock_signal": "household other residential property value above zero, or the person's property income above zero", + "parameter_solver": "nested bisection; the intercept matches the expected weighted count at each trial slope, the slope matches the expected weighted gains", + "count_target": "Table 8a 2024-25 total taxpayers reporting residential property disposals times the Table 8b 2024-25 individuals/all taxpayer share on the UK Property service", + "gains_target": "Table 8a 2024-25 total residential property gains times the Table 8b 2024-25 individuals/all gains share on the UK Property service", + "basis_assumption": "trusts hold the same share of the Self Assessment component as of the UK Property service, a residential gainer's whole net gain is attributed to residential property, and the gainers of one household are residential independently of each other", + "realization": "probability carried as weight, no draw: a household carrying k liable gainers becomes 2^k arms, one per subset of residential gainers, at the household's weight times the product of p over the residential gainers and 1 - p over the others; the arm with no residential gainer keeps the household's ids; at design weights the residential count and gains equal the solved expectations sum(p w) and sum(p w g) in every gain band", + "maximum_liable_gainers_per_household": 3, + "gainer_order": "person_id ascending within the household", + "id_remapping": "id_multiplier_for_values; arm j takes every entity id + j x multiplier, bit m of j marking the household's m-th liable gainer residential", + "flag_column": "household_is_cgt_residential_clone", + "index_column": "cgt_residential_clone_index", + "probability_column": "cgt_residential_probability", + "output_column": "capital_gains_residential_property", + "output_semantics": "the person's capital_gains on an arm where the person is residential, 0 otherwise", + "values": "every other column copied unchanged; no draw, no seed, no salt", + "weight_kind_out": "importance", + "conservation": "exact_total", + "declared_factor": 1.0, + "reason": "Capital-gains residential split divides every household carrying liable gainers into arms weighted by the solved residential probabilities, one arm per subset of residential gainers; every household's mass and the total household mass are conserved." + } + ], + "outputs": [ + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", + "cgt_residential_probability", + "capital_gains_residential_property" + ], + "rewrites": [], + "nonnegative_outputs": [ + "cgt_residential_probability", + "capital_gains_residential_property" + ], + "notes": "Structural stage after the incidence anchor and before the asset-type stage (microcosm#1063). The residential-property flag used to be realised by a weighted systematic walk, one whole flag per row, whose gains error is bounded by one row's stake: at the top of the gains distribution one stake is several times a band's expected residential gains, so the realised gains sat short of expectation by the lottery expectation of the giant stakes or over it by one of them. This stage solves the same logistic (intercept to the Table 8a 2024-25 taxpayers on the individuals basis, slope to the gains, with the stock shift) and carries the probability as weight instead: every household with liable gainers becomes 2^k arms at product weights, one per subset of residential gainers, the arm with no residential gainer keeping the household's ids and arm j taking every entity id plus j times the id multiplier, the lineage rule the executor, the geography identity kernel and the identity tool read for the support split and the clone. At design weights the residential count and gains equal the solved expectations in every gain band, with no draw, no offset and no seed; the calibration, which binds hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains, decides between the arms afterwards as it does between the incidence clone's halves. Every household's mass and composition and the total household mass are conserved; the receipt is gated at stage time." + }, { "stage": "hmrc_cgt_asset_type_spine", "survey": "HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative), Table 4.1 (individuals claiming Business Asset Disposal Relief or Investors' Relief by band of qualifying gain, 2024-25, administrative) and Table 7 (disposals, proceeds and gains by asset type, 2023-24, sample-based), vendored from the pinned Chronicle feed", @@ -4976,25 +5285,9 @@ "runtime_sha256_required": true, "fail_on_mismatch": true }, - { - "kind": "assign_residential_property_flag", - "population": "persons with net capital gains above the annual exempt amount (the national taxpayer proxy)", - "model": "logistic probability in centred log gains with a stock shift, p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean log gain of the population, s the stock log-odds and h one where the stock signal holds, zero otherwise", - "stock_log_odds": 1.6, - "stock_signal": "household other residential property value above zero, or the person's property income above zero", - "parameter_solver": "nested bisection; the intercept matches the expected weighted count at each trial slope, the slope matches the expected weighted gains", - "count_target": "Table 8a 2024-25 total taxpayers reporting residential property disposals times the Table 8b 2024-25 individuals/all taxpayer share on the UK Property service", - "gains_target": "Table 8a 2024-25 total residential property gains times the Table 8b 2024-25 individuals/all gains share on the UK Property service", - "basis_assumption": "trusts hold the same share of the Self Assessment component as of the UK Property service, and a flagged person's whole net gain is attributed to residential property", - "realization": "weighted systematic sampling in ascending gain order with one seeded offset: a person is flagged when the expected weight owed so far reaches their own weight, so the flagged weight tracks the expected weight within one person's weight along the gains axis and the realised count and gains sit close to their expectations; the Bernoulli sigma of each is reported as the envelope", - "seed": 553, - "seed_mixing": "seed combined with the build period", - "output_column": "capital_gains_residential_property", - "output_semantics": "the person's capital_gains where flagged residential, 0 otherwise" - }, { "kind": "assign_badr_qualifying_gains", - "population": "liable gainers not flagged residential", + "population": "liable gainers whose arm is not residential (capital_gains_residential_property == 0, cgt_residential_split)", "bands": "HMRC Table 4.1 2024-25 individuals claiming Business Asset Disposal Relief or Investors' Relief by band of qualifying gain (lower limits 0, 10,000, 25,000, 50,000, 100,000, 250,000, 500,000 and 1,000,000)", "band_pool": "the population with net gains inside the band's range; the open top band takes net gains at or above the lifetime limit", "model": "per band below the lifetime limit, a logistic probability in centred log gains with the stock shift, solved like the residential flag: the intercept matches the band's published claimant count and the slope its published qualifying gains, with zero slope where the band pool's own mean already matches; the open top band has zero slope, so its probability varies only with the stock shift, and its intercept matches its count", @@ -5040,12 +5333,13 @@ "other_financial_assets": "household gross financial wealth above zero" }, "share_targets": "Table 7 2023-24 gains by asset type excluding residential land and buildings, as shares", - "weight_fitting": "multiplicative updates of the type weights, with claimants restricted to the claimant categories, until the household-weighted expected gains shares by type match the share targets; the fit refuses if it has not converged within the iteration limit, and refuses up front when claimants hold more of the gains than the claimant categories' share", + "claimant_share_targets": "the share targets among the claimant categories, each over their sum", + "non_claimant_share_targets": "(share target - k x claimant share target) / (1 - k) with k the claimants' share of the non-residential gains, the composition that restores the share targets over claimants and non-claimants together", + "basis_assumption": "Table 7 does not split gains by relief, so claimants are given its composition among the claimant categories", + "weight_fitting": "one fit per claimant status: multiplicative updates of the type weights until the household-weighted expected gains shares by type match that status's share targets; a fit refuses if it has not converged within the iteration limit, and the stage refuses up front when claimants hold more of the gains than the claimant categories' share", "weight_fitting_iterations": 200, "weight_fitting_tolerance": 1e-06, - "realization": "one seeded uniform per person against the cumulative type probabilities, consumed in person_id order", - "seed": 554, - "seed_mixing": "seed combined with the build period", + "realization": "error diffusion per claimant status in descending (gain, person_id) order: each person adds weight x gain x type probability to a running balance per type and takes the type it may hold with the largest balance, which is drawn down by the whole weight x gain, so the realised gains by type track the fitted composition instead of following the heaviest rows; deterministic, no seed", "output_column": "capital_gains_asset_type", "value_domain": [ "none", @@ -5081,99 +5375,12 @@ ], "outputs": [ "capital_gains_asset_type", - "capital_gains_residential_property", "capital_gains_badr" ], "nonnegative_outputs": [ - "capital_gains_residential_property", "capital_gains_badr" ], - "notes": "Spine stage after the amounts redraw: every liable gainer carries one main asset type; where flagged residential, its net gain is restated in capital_gains_residential_property; where drawn as a BADR or Investors' Relief claimant, its qualifying gain is written to capital_gains_badr and its main type is one the reliefs apply to. The residential flag is solved to the Table 8a 2024-25 totals on the individuals basis, which the hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains targets bind; the BADR claims are solved band by band to Table 4.1 2024-25, whose individuals band rows the hmrc.cgt.badr_ir_taxpayers_by_band and hmrc.cgt.badr_ir_qualifying_gains_by_band targets bind (microcosm#1014); the non-residential composition follows the fenced Table 7 2023-24 prior and is diagnostic only. Carried interest (Table 9) is not modelled (microcosm#725)." - }, - { - "stage": "cgt_incidence_anchor", - "survey": "Family Resources Survey 2024-25, SPI synthetic support, the HMRC Table 3 redrawn gains on the spine and Advani-Summers capital-gains incidence", - "source": "Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465", - "grain": "household", - "artifacts": [ - { - "role": "capital_gains_incidence_and_quantiles", - "kind": "public_aggregate_reference", - "resource": "advani_summers_capital_gains_distribution.json", - "format": "json", - "runtime_sha256_required": true - }, - { - "role": "policy_parameters", - "kind": "versioned_parameter_tree", - "dependency": "policyengine-uk>=2.98.0 via microcosm-build[uk]", - "parameters": [ - "gov.hmrc.cgt.annual_exempt_amount" - ], - "instant_rule": "raw dated parameter files evaluated at 1 June of the build period's tax year", - "runtime_sha256_required": false, - "dependency_discipline": "deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time" - } - ], - "operations": [ - { - "kind": "pair_clone_households_to_originals", - "clone_flag_column": "household_is_capital_gains_clone", - "id_remapping": "id_multiplier_for_values", - "pairing": "clone household_id minus the clone stage's id multiplier, recomputed from the pre-clone rows (clone flag false; the support split's copies are pre-clone rows with their own clones)", - "requirement": "bijection between clone and original households; an unpaired clone or original fails the build" - }, - { - "kind": "derive_reporter_composition_from_banded_quantiles", - "resource": "advani_summers_capital_gains_distribution.json", - "income_proxy_components": [ - "employment_income", - "self_employment_income", - "state_pension_reported", - "private_pension_income", - "property_income", - "savings_interest_income", - "dividend_income", - "miscellaneous_income" - ], - "allowance_subtraction": false, - "carrier": "oldest adult; person_id ascending breaks age ties", - "adult_minimum_age": 16, - "quantile_points": [ - 0.05, - 0.1, - 0.25, - 0.5, - 0.75, - 0.9, - 0.95 - ], - "spline_degree": 1, - "extrapolation": "ext=0", - "crossings": "each band's degree-1 quantile spline inverted at zero and at the build period's annual exempt amount", - "aggregation": "crossings averaged over clone carriers weighted by household weight times the band's percent_with_gains", - "liable_mass": "weighted persons with capital_gains above the annual exempt amount on every household after the Table 3 redraw", - "targets": "sub_exempt = liable_mass * (exempt_quantile - zero_quantile) / (1 - exempt_quantile); loss = liable_mass * zero_quantile / (1 - exempt_quantile)" - }, - { - "kind": "transfer_clone_mass_to_originals", - "population": "clone households paired to their originals; originals are outside the anchor, so the groups, targets and masses are clone-side quantities, not population counts", - "sub_exempt_group": "clone households whose carrier's capital_gains lie in (0, annual exempt amount]; a carrier with exactly zero gains is no reporter and is left untouched", - "loss_group": "clone households whose carrier's capital_gains are negative", - "liable_clones": "untouched", - "factor": "min(maximum_factor, scale * percent_with_gains) with scale solved per group by bisection so the group's mass equals its target; a group at or below its target is untouched", - "maximum_factor": 1.0, - "transfer": "each clone's removed mass is added to its paired original, so every pair's mass is conserved to rounding; no global exact-total correction is applied, so no household outside a trimmed pair moves", - "weight_kind_out": "importance", - "conservation": "household_total_to_rounding", - "pair_conservation": "to_rounding", - "declared_factor": 1.0, - "reason": "Capital-gains incidence anchor moves the mass of non-liable clone households back to their paired originals until the sub-exempt and loss-making clone mass match the Advani-Summers reporter composition at the redrawn liable mass; every pair's mass and the total household mass are conserved." - } - ], - "outputs": [], - "rewrites": [], - "notes": "Spine-only weight anchor after the amounts and asset-type stages (microcosm#970). The equal-mass clone leaves half of all household mass on gainer households, so after the Table 3 redraw the clones below the exempt amount carry millions of weighted persons that HMRC publishes no count for; the anchor derives their level from the redrawn liable mass and the Advani-Summers reporter composition and moves the excess back to each clone's paired original, which is identical in every cell but the gains. No cell is written: amounts, flags, liable clones, zero-gain clone carriers are untouched; the groups, targets and receipt are clone-side quantities (originals are outside the anchor), every pair's mass and the household total are conserved, and the receipt is gated at stage time." + "notes": "Spine stage after the amounts redraw, the incidence anchor and the residential split: every liable gainer carries one main asset type; a residential arm (capital_gains_residential_property above zero, written by cgt_residential_split since microcosm#1063) is typed residential and its net gain is already restated there; where drawn as a BADR or Investors' Relief claimant, its qualifying gain is written to capital_gains_badr and its main type is one the reliefs apply to. The residential arms carry the Table 8a 2024-25 totals on the individuals basis at design weights, which the hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains targets bind; the BADR claims are solved band by band to Table 4.1 2024-25, whose individuals band rows the hmrc.cgt.badr_ir_taxpayers_by_band and hmrc.cgt.badr_ir_qualifying_gains_by_band targets bind (microcosm#1014); the non-residential composition follows the fenced Table 7 2023-24 prior and is diagnostic only: since microcosm#1063 the type is fitted once per claimant status (claimants to Table 7's composition among the types the reliefs apply to, non-claimants to the composition that restores Table 7 overall) and realised by a deterministic walk that keeps the realised gains by type on the fit. Carried interest (Table 9) is not modelled (microcosm#725)." }, { "stage": "salary_sacrifice", @@ -5193,7 +5400,7 @@ { "kind": "fit_weighted_qrf", "training_population": "support_channel == frs and not capital-gains clone and not CGT support copy and salary_sacrifice_asked == 1", - "target_population": "salary_sacrifice_asked != 1 frame-wide", + "target_population": "salary_sacrifice_asked != 1 and employment_income > 0, frame-wide; people not asked who have no employment income are set to zero", "predictors": [ "age", "employment_income" @@ -5215,7 +5422,7 @@ "target": 5400000, "donor_pool": "employee_pension_contributions > 0 and pension_contributions_via_salary_sacrifice == 0 and employment_income > 0", "rate_cap": 0.5, - "move": "full employee_pension_contributions to pension_contributions_via_salary_sacrifice; source zeroed; employment_income lowered by the amount sacrificed", + "move": "full employee_pension_contributions to pension_contributions_via_salary_sacrifice; source zeroed; employment_income lowered by the amount sacrificed; the converted record's pay before the conversion kept on salary_sacrifice_pre_conversion_pay (zero elsewhere)", "seed": 2024, "salt": "salary_sacrifice_conversion", "receipt": "weighted_headcount", @@ -5225,12 +5432,14 @@ "outputs": [ "pension_contributions_via_salary_sacrifice", "employee_pension_contributions", - "employment_income" + "employment_income", + "salary_sacrifice_pre_conversion_pay" ], "nonnegative_outputs": [ "pension_contributions_via_salary_sacrifice", "employee_pension_contributions", - "employment_income" + "employment_income", + "salary_sacrifice_pre_conversion_pay" ], "rewrites": [ "pension_contributions_via_salary_sacrifice", diff --git a/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml b/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml index 3442ecfe4..83b46e874 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml +++ b/packages/microcosm-build/src/microcosm/build/uk/spec/sources.yaml @@ -1296,22 +1296,35 @@ stages: - 500000 - 1000000 - 2000000 - donors_per_band: 120 + minimum_donors_per_band: 120 + maximum_donor_weight: 250.0 + count_rule: max(minimum_donors_per_band, ceil(published band taxpayers / maximum_donor_weight)) + weight_rule: published band taxpayers / band donor count, equal within a band expected_band_count: 4 - expected_donor_count: 480 candidate_population: raw FRS base channel adults aged 16 and over - candidate_order: person_id ascending - draw: weighted_without_replacement + candidate_order: person_id ascending breaks equal draw keys + draw: 'weighted_without_replacement_identity_keyed: Efraimidis-Spirakis keys log(u) / (household weight x propensity) on stable_identity_uniforms(person_id, seed, salt:band); the largest keys are seated, one carrier per household, the highest band first' propensity: SPI 2022-23 FACT-weighted band share by region, sex and age band + tape_preparation: 'as hmrc_spi_income_spine prepares the tape: its stage-1 seed and draw_spi_donor_ages_by_population' + tape_preparation_seed: 42 seed: 3 + salt: spi_income_band_donor_draw clone_index: 2 flag_column: household_is_spi_income_band_donor band_column: spi_income_band_donor_lower_bound carrier_column: person_is_spi_income_band_carrier - initial_weight: published band taxpayers / donors_per_band + funding: 'stratum reallocation: every incumbent household of the stratum a donor''s source household belongs to is scaled by 1 - donor mass / stratum mass' + funding_strata: + - region + maximum_reallocation_share: 0.5 + maximum_donor_household_share: 0.15 + seating_scale: min(1, maximum_donor_household_share x eligible base households / expected donor count); band counts floored, at least one per band + mass_scale: min(1, min over funding strata of maximum_reallocation_share x stratum mass / seated donor mass); every band's weight scaled alike never_zero_weight: true weight_kind_out: importance - reason: SPI income band donors stack positive-weight support households per HMRC Table 2.5 total-income band from GBP 200,000, one carrier adult each; the published band taxpayer mass is added explicitly and the base and synthetic channels keep their weights. + conservation: exact_total + declared_factor: 1.0 + reason: Reallocate the published HMRC Table 2.5 taxpayer mass of each total-income band from GBP 200,000 from the households of each donor's region to its income band donor support rows; every region's household mass and the total national mass are conserved. outputs: - household_is_spi_income_band_donor - spi_income_band_donor_lower_bound @@ -1329,7 +1342,7 @@ stages: - household_source_id nonnegative_outputs: - spi_income_band_donor_lower_bound - notes: Reserves 120 SPI-channel households per Table 2.5 band from GBP 200,000 (480 in all) so the top of the income distribution has support before calibration. Carriers are FRS adults drawn by the tape's own band propensity given region, sex and age band, their households are copied whole at clone index 2 with the synthetic flag, and each starts at the build tax year's published taxpayers over 120. The income stage gives every carrier a band-conditional draw from the tape (resample_band_donor_leaves). + notes: Reserves SPI-channel households for every Table 2.5 band from GBP 200,000 so the top of the income distribution has support before calibration. A band seats enough donors that none starts above the maximum weight and never fewer than the minimum, at equal weights of the build tax year's published taxpayers over the band's count. Carriers are FRS adults seated by identity-keyed draws on household weight times the tape's own band propensity given region, sex and age band; their households are copied whole at clone index 2 with the synthetic flag. The mass the donors carry is taken in proportion from the households of each donor's region, so the national total and every region's mass are unchanged (microcosm#1063). The income stage gives every carrier a band-conditional draw from the tape (resample_band_donor_leaves). - stage: hmrc_spi_income_spine survey: Survey of Personal Incomes Public Use Tape 2022-23 and HMRC Personal Incomes Tables 3.6/3.7 2023-24 source: https://assets.publishing.service.gov.uk/media/69f1f12d2fae53a03709682f/Collated_Tables_3_1_to_3_11_2324.ods @@ -1580,6 +1593,9 @@ stages: age_pool_minimum: 20 weighting: FACT with_replacement: true + draw: 'identity_keyed_inverse_cdf: stable_identity_uniforms(draw_key, seed, salt) over the pool''s cumulative FACT in tape order' + draw_key: person_source_id + salt: spi_income_band_donor_leaf_draw outputs: stage-1 outputs, uprated as the stage-1 draws seed: 44 - kind: fit_weighted_qrf_stage2 @@ -2058,7 +2074,10 @@ stages: net_financial_wealth: HFINWNTR8_Sum net_financial_wealth_exsl: HFINWNTR8_exSLC_Sum mortgage_debt: HMortGR8 + main_residence_mortgage: TotMortR8 + other_property_mortgage: OthMortR8_sum tenure_code: Ten1R8 + private_rent_code: DVPriRntR8 main_residence_value: DVhvalueR8 other_residential_property_value: DVHseValR8_sum non_residential_property_value: DVBlDValR8_sum @@ -2072,7 +2091,9 @@ stages: cash_isa: DVCISAVR8_aggr student_loan_balance: Tot_LosR8_aggr - Tot_los_exc_SLCR8_aggr consumer_debt: max(HFINWR8_SUM - HFINWNTR8_exSLC_Sum, 0) - has_mortgage_tenure: Ten1R8 in {2, 3} + tenure_category: 'Ten1R8 1 owned_outright; 2 or 3 owned_with_mortgage; 4 or 5 private_rent when DVPriRntR8 == 1, otherwise social_rent; any other code private_rent, counted in the receipt' + other_property_value: DVPropertyR8 - (DVhvalueR8 + DVHseValR8_sum + DVBlDValR8_sum + DVLUKValR8_sum), refused when negative + other_financial_assets: HFINWR8_SUM - (DVSaValR8_aggr + DVCISAVR8_aggr + corporate_wealth), refused when negative is_renting: DVPriRntR8 == 1 region: GORR8 via incumbent REGIONS map - kind: materialize_rules_engine_predictors @@ -2084,7 +2105,6 @@ stages: - employment_income - self_employment_income - capital_income - - is_renting consumed_only: true - kind: fit_weighted_qrf_chain weights: explicit @@ -2100,35 +2120,73 @@ stages: - capital_income - num_bedrooms - council_tax - - is_renting + - tenure_owned_with_mortgage + - tenure_private_rent + - tenure_social_rent - region categorical_predictors: - region - - is_renting + - tenure_owned_with_mortgage + - tenure_private_rent + - tenure_social_rent region_remap: NORTHERN_IRELAND: WALES integer_outputs: - num_vehicles - debt_segment_predictors: - - has_mortgage_tenure + stratified_targets: + main_residence_value: + - owned_outright + - owned_with_mortgage + main_residence_mortgage: + - owned_with_mortgage + internal_components: + - other_property_value + - other_financial_assets + - other_mortgage + derived_totals: + property_wealth: + - owned_land + - main_residence_value + - other_residential_property_value + - non_residential_property_value + - other_property_value + corporate_wealth: + - corporate_wealth_excl_isa + - stocks_and_shares_isa + gross_financial_wealth: + - savings + - cash_isa + - corporate_wealth + - other_financial_assets + net_financial_wealth: + - gross_financial_wealth + - -consumer_debt + - -student_loan_balance + mortgage_debt: + - main_residence_mortgage + - other_mortgage chain_order: - owned_land + - main_residence_value + - other_residential_property_value + - non_residential_property_value + - other_property_value - property_wealth - private_pension_wealth - corporate_wealth_excl_isa - stocks_and_shares_isa - corporate_wealth - - gross_financial_wealth - - net_financial_wealth - - main_residence_value - - other_residential_property_value - - non_residential_property_value - savings + - cash_isa + - other_financial_assets - num_vehicles - student_loan_balance - - cash_isa + - gross_financial_wealth + - main_residence_mortgage + - other_mortgage - mortgage_debt - consumer_debt + - net_financial_wealth - kind: fold_into output: corporate_wealth inputs: @@ -2187,8 +2245,8 @@ stages: - student_loan_balance - mortgage_debt - consumer_debt - notes: 'Ports incumbent WAS round-8 wealth imputation with signed E5 differences: exact lower-case column matching replaces the fuzzy r/w fallback; cash ISA uses DVCISAVR8_aggr and stocks-and-shares ISA uses DVIISAVR8_aggr; private pension wealth (WAS total private pension wealth less the current defined-benefit component; DB-type components at the SCAPE rate, money-purchase components at reported fund values) is drawn as its own output private_pension_wealth and no longer folded into corporate_wealth, which keeps only the share-like holdings - pension rights are disregarded capital in every means test (UC Regs 2013 Sch 10 para 10 and the HB/JSA/ESA/IS/SPC parallels; uk-data#452) - and the downstream chain segment conditions on both components; each chain segment starts its own fit and draw RNG streams from a child seed derived deterministically from the declared seed (start_chain respawns from the model seed on every call, so one shared model coupled the k-th target of every segment); corporate_wealth folds stocks-and-shares ISA after drawing corporate_wealth_excl_isa; recipient Northern Ireland regions are mapped to Wales for prediction only; student_loan_balance is allocated by household id rather than the incumbent positional off-by-one. The debt segment (fourth chain segment, fourth child seed) reads WAS round-8 derived variables HMortGR8 (total outstanding mortgage debt), HFINWNTR8_exSLC_Sum (net financial wealth excluding student loans; gross less it is the non-student-loan financial liabilities) and Ten1R8 (tenure; codes 2 and 3 are buying with a mortgage and part own/part rent, confirmed by the I1 audit of the pinned tab); UKDS SN 7215 documentation is the codebook. Engine predictors materialize at their native entity and person/benunit values are summed to household, reproducing the incumbent map_to=household semantics; region is one-hot encoded jointly across donor and recipient (the incumbent''s dummy encoding), with unmapped donor GOR codes becoming all-zero dummy rows. The WAS and FRS predictor definitions are not fully like-for-like and are ported as-is; raw WAS missing values are blanket-filled with zero; UKDS negative sentinel codes (-9/-8/-7/-6) are recoded to zero - for the nonnegative-domain columns the licensed audit found carrying them (vcarnr8: 2 rows; HBedRmR8: 95.8 percent - the bedrooms question is effectively unasked in the WAS household file, predictor-quality revisit registered on microcosm#145) - a signed difference vs the incumbent, which trains on raw sentinels; DVPriRntR8''s -9 is structural not-applicable so the is_renting mapping is unchanged; genuinely negative domains are never recoded.' + notes: 'Ports incumbent WAS round-8 wealth imputation with signed E5 differences: exact lower-case column matching replaces the fuzzy r/w fallback; cash ISA uses DVCISAVR8_aggr and stocks-and-shares ISA uses DVIISAVR8_aggr; private pension wealth (WAS total private pension wealth less the current defined-benefit component; DB-type components at the SCAPE rate, money-purchase components at reported fund values) is drawn as its own output private_pension_wealth and no longer folded into corporate_wealth, which keeps only the share-like holdings - pension rights are disregarded capital in every means test (UC Regs 2013 Sch 10 para 10 and the HB/JSA/ESA/IS/SPC parallels; uk-data#452) - and the downstream chain segment conditions on both components; each chain segment starts its own fit and draw RNG streams from a child seed derived deterministically from the declared seed (start_chain respawns from the model seed on every call, so one shared model coupled the k-th target of every segment); corporate_wealth folds stocks-and-shares ISA after drawing corporate_wealth_excl_isa; recipient Northern Ireland regions are mapped to Wales for prediction only; student_loan_balance is allocated by household id rather than the incumbent positional off-by-one. Tenure enters every chain segment as one four-way category built the same way on both sides (microcosm#1063): the donor''s from Ten1R8 and DVPriRntR8 (owned outright; buying with a mortgage or part rent, part mortgage; rented or rent-free from a private landlord; rented from a council or housing association), the recipient''s from the frame''s tenure_type (council and housing association together), so the stage reads no engine tenure formula. Two columns are structural in tenure and are fitted and drawn inside their stratum only, zero by rule elsewhere: main_residence_value on the owner tenures (every WAS owner has a positive value and no renter has one) and main_residence_mortgage (TotMortR8) on the mortgaged tenure. mortgage_debt is HMortGR8, all of the household''s mortgages, and is derived as the main-residence mortgage plus the mortgages on other property (other_mortgage, HMortGR8 less TotMortR8 on the donor: buy-to-let, second homes and the rest), which are held on every tenure and drawn without a stratum, so an outright owner''s or a renter''s buy-to-let keeps its mortgage beside the property it is secured on (OthMortR8_sum is read for the receipt only); the donor''s conditioning copy carries the same rule, so consumer_debt is fitted on values recipients can hold. Totals are not drawn: property_wealth is the sum of land, the main residence, other houses, buildings and a drawn remainder (buy-to-let, overseas and other property), gross_financial_wealth is savings, the cash ISA, the share-like holdings and a drawn remainder, and net_financial_wealth is gross less consumer debt and the student loan balance - identities the pinned tab satisfies on every row and the cleaning step refuses a donor that breaks. A sum of donor-valued components that leaves the donor''s range for the total gives up the excess from the remainder first, and the receipt counts the rows. Eight chain segments, each with its own child seed: land; the main residence; the other property components; pension and share-like wealth; the financial components, vehicles and the student loan balance; the main-residence mortgage; the other mortgages; consumer debt. HFINWNTR8_exSLC_Sum is net financial wealth excluding student loans, so gross less it is the non-student-loan financial liabilities; UKDS SN 7215 documentation is the codebook. Engine predictors materialize at their native entity and person/benunit values are summed to household, reproducing the incumbent map_to=household semantics; region is one-hot encoded jointly across donor and recipient (the incumbent''s dummy encoding), with unmapped donor GOR codes becoming all-zero dummy rows. The WAS and FRS predictor definitions are not fully like-for-like and are ported as-is; raw WAS missing values are blanket-filled with zero; UKDS negative sentinel codes (-9/-8/-7/-6) are recoded to zero + for the nonnegative-domain columns the licensed audit found carrying them (vcarnr8: 2 rows; HBedRmR8: 95.8 percent - the bedrooms question is effectively unasked in the WAS household file, predictor-quality revisit registered on microcosm#145) - a signed difference vs the incumbent, which trains on raw sentinels; DVPriRntR8''s -9 is structural not-applicable so the private-renter reading is unchanged; genuinely negative domains are never recoded.' - stage: was_lisa survey: Wealth and Assets Survey round 8 source: Office for National Statistics Wealth and Assets Survey, UK Data Service SN 7215, DOI 10.5255/UKDA-SN-7215-20; local licensed round-8 person and household tabs (interviews April 2020 to March 2022, Great Britain). @@ -2262,7 +2320,6 @@ stages: - employment_income - self_employment_income - capital_income - - is_renting consumed_only: true - kind: impute_lifetime_isa_ownership output: has_lifetime_isa @@ -2360,7 +2417,10 @@ stages: rewrites: - main_residence_value - property_wealth - notes: Deterministically rescales owner rows so regional unweighted owner means match the public house-price reference. Northern Ireland has no reference row and is never scaled; empty and nonpositive regions are skipped. The unweighted mean follows the incumbent behavior. + - owned_land + - other_residential_property_value + - non_residential_property_value + notes: Deterministically rescales owner rows so regional unweighted owner means match the public house-price reference. Northern Ireland has no reference row and is never scaled; empty and nonpositive regions are skipped. The unweighted mean follows the incumbent behavior. The factor is read off the main residence and has always scaled the property total; since microcosm#1063 it scales the total's other components in the frame as well (land, other houses, buildings), so an owner's property_wealth stays the sum of its components after the uprating. Mortgage debt is never uprated. - stage: nts_bus_travel survey: National Travel Survey 2002-2024 source: Department for Transport National Travel Survey, UK Data Service SN 5340 (End User Licence, 19th edition, November 2025), DOI 10.5255/UKDA-SN-5340-19; local licensed tab files (household, individual, trip); England residents only from 2013. @@ -2759,6 +2819,7 @@ stages: - LCFS tenure 4 and 8 -> RENT_PRIVATELY - LCFS accommodation 4 and 5 -> FLAT logged_dropna_row_count: true + floor: 0.0 - kind: uprate_donor_columns from_period: 2023 to_period: 2024 @@ -3637,6 +3698,60 @@ stages: rewrites: - would_claim_pc notes: Runs after the SPI income chain and UC capital coherence, before CGT cloning (microcosm#1069 R9). One temporary engine materialization gives each benefit unit's pre-take-up Pension Credit entitlement (guarantee_credit, savings_credit, is_pension_credit_eligible; consumed only). Entitled units fall into DWP's take-up components, Guarantee Credit (with or without Savings Credit) and Savings Credit only; within a component a reporter claims and the other entitled units claim at r = (t * E - R) / (E - R), t being DWP's financial year ending 2024 caseload take-up (69% and 37%), E the component's entitled units and R its reporters, weighted at the stage's household weights. A unit with no entitlement claims only if it reports. The uniform draw is frs_take_up's would_claim_pc identity stream, whose early draw this rewrite replaces; units with every adult at or over State Pension age are the only ones the engine finds eligible. +- stage: child_benefit_take_up + survey: Family Resources Survey 2024-25 + source: HMRC Child Benefit statistics, annual release at August 2025 (percentage of eligible children for whom Child Benefit is claimed by single year of age; families registered and families opted out of payment), applied to PolicyEngine-UK Child Benefit eligibility and adjusted net income after the SPI income chain. + grain: benunit + artifacts: + - role: child_benefit_statistics + kind: public_aggregate_reference + resource: hmrc_child_benefit_statistics.json + format: json + runtime_sha256_required: true + operations: + - kind: materialize_rules_engine_predictors + predictors: + - is_child_or_qualifying_young_person_for_child_benefit + - adjusted_net_income + consumed_only: true + - kind: aggregate_person_to_benunit + method: any_positive + consumed_only: true + aggregates: + child_benefit_reported_anchor: child_benefit_reported + - kind: assign_family_claims_by_child_age + output: would_claim_child_benefit + draw: would_claim_child_benefit + seed: 0 + anchor: child_benefit_reported_anchor + eligibility: is_child_or_qualifying_young_person_for_child_benefit + family_key: age of the benefit unit's eldest eligible child + resource: hmrc_child_benefit_statistics.json + rate_measure: claim_rate + rate_period: 2025-05 + weight_mapping: household_to_benunit + rule: a reporter claims; a non-reporting family whose eldest eligible child is aged g claims at r_g = (t_g * C_g - R_g - sum over k > g of N_gk * r_k) / N_gg, weighted, solved from the eldest age down and clipped to [0, 1] + - kind: assign_opt_out_by_charge_income + output: child_benefit_opts_out + draw: child_benefit_opts_out + seed: 0 + resource: hmrc_child_benefit_statistics.json + numerator_measure: families_opted_out_claimant_all + denominator_measure: families_registered + caseload_period: 2025-08 + charge_income: adjusted_net_income + charge_parameters: gov.hmrc.income_tax.charges.CB_HITC + pools: + - fully_charged + - taper + weight_mapping: household_to_benunit + rule: published opted-out share of registered families, of the claiming families; drawn among claiming non-reporting families with the highest adjusted net income at or above the full-charge income, then, for any remainder, among those inside the taper + payment_rule: would_claim_child_benefit = claims and not opted out on families with an eligible child; a benefit unit without one keeps its early draw + outputs: [] + rewrites: + - would_claim_child_benefit + - child_benefit_opts_out + notes: 'Runs after the SPI income chain, UC capital coherence and the Pension Credit redraw, before CGT cloning (microcosm#1063). frs_take_up draws would_claim_child_benefit early at one flat rate over every benefit unit with the reporters on top, and child_benefit_opts_out at a flat rate whatever the unit''s income; this stage redraws both. One temporary engine materialization gives each person''s eligibility as a child or qualifying young person and each person''s adjusted net income (consumed only). A family claims for all its children or none, so the draw is by family, keyed on the age of its eldest eligible child: a reporter claims, and the other families claim at rates solved from the eldest age down so that the claimed share of the eligible children of each single year of age is HMRC''s published one (Table 15, May 2025, opted-out families included), weighted at the stage''s household weights. The published share of registered families that opted out of payment (August 2025) of the claiming families then opts out, drawn among claiming families that do not report receipt and whose highest adjusted net income is at or above the income at which the High Income Child Benefit Charge takes the whole benefit, then inside the taper for any remainder. The engine pays on would_claim_child_benefit alone, so the stage stores claims-and-not-opted-out there and the opt-out in child_benefit_opts_out; a benefit unit with no eligible child keeps its early draw and is never opted out.' - stage: uc_deduction_attributes survey: DWP Universal Credit deductions statistics, March 2025 to February 2026 source: Department for Work and Pensions supplementary Tables 2, 5 and 6, sha-pinned by Chronicle package dwp-uc-deductions-march-2025-february-2026. @@ -3983,6 +4098,139 @@ stages: rewrites: - capital_gains notes: Spine manifest projection of the merged HMRC Table 3 amounts stage. It deliberately omits base_candidate and verify_certified_candidate, which belong only to the certified-H5 path. The allocation targets are raked to the 2024-25 Table 6 age and Table 5 region margins from the vendored conditioning resource and walked inside income x age x region cells, so large gains land on the kind of person who holds them (microcosm#725). +- stage: cgt_incidence_anchor + survey: Family Resources Survey 2024-25, SPI synthetic support, the HMRC Table 3 redrawn gains on the spine and Advani-Summers capital-gains incidence + source: Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465 + grain: household + artifacts: + - role: capital_gains_incidence_and_quantiles + kind: public_aggregate_reference + resource: advani_summers_capital_gains_distribution.json + format: json + runtime_sha256_required: true + - role: policy_parameters + kind: versioned_parameter_tree + dependency: policyengine-uk>=2.98.0 via microcosm-build[uk] + parameters: + - gov.hmrc.cgt.annual_exempt_amount + instant_rule: raw dated parameter files evaluated at 1 June of the build period's tax year + runtime_sha256_required: false + dependency_discipline: deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time + operations: + - kind: pair_clone_households_to_originals + clone_flag_column: household_is_capital_gains_clone + id_remapping: id_multiplier_for_values + pairing: clone household_id minus the clone stage's id multiplier, recomputed from the pre-clone rows (clone flag false; the support split's copies are pre-clone rows with their own clones) + requirement: bijection between clone and original households; an unpaired clone or original fails the build + - kind: derive_reporter_composition_from_banded_quantiles + resource: advani_summers_capital_gains_distribution.json + income_proxy_components: + - employment_income + - self_employment_income + - state_pension_reported + - private_pension_income + - property_income + - savings_interest_income + - dividend_income + - miscellaneous_income + allowance_subtraction: false + carrier: oldest adult; person_id ascending breaks age ties + adult_minimum_age: 16 + quantile_points: + - 0.05 + - 0.1 + - 0.25 + - 0.5 + - 0.75 + - 0.9 + - 0.95 + spline_degree: 1 + extrapolation: ext=0 + crossings: each band's degree-1 quantile spline inverted at zero and at the build period's annual exempt amount + aggregation: crossings averaged over clone carriers weighted by household weight times the band's percent_with_gains + liable_mass: weighted persons with capital_gains above the annual exempt amount on every household after the Table 3 redraw + targets: sub_exempt = liable_mass * (exempt_quantile - zero_quantile) / (1 - exempt_quantile); loss = liable_mass * zero_quantile / (1 - exempt_quantile) + - kind: transfer_clone_mass_to_originals + population: clone households paired to their originals; originals are outside the anchor, so the groups, targets and masses are clone-side quantities, not population counts + sub_exempt_group: clone households whose carrier's capital_gains lie in (0, annual exempt amount]; a carrier with exactly zero gains is no reporter and is left untouched + loss_group: clone households whose carrier's capital_gains are negative + liable_clones: untouched + factor: min(maximum_factor, scale * percent_with_gains) with scale solved per group by bisection so the group's mass equals its target; a group at or below its target is untouched + maximum_factor: 1.0 + transfer: each clone's removed mass is added to its paired original, so every pair's mass is conserved to rounding; no global exact-total correction is applied, so no household outside a trimmed pair moves + weight_kind_out: importance + conservation: household_total_to_rounding + pair_conservation: to_rounding + declared_factor: 1.0 + reason: Capital-gains incidence anchor moves the mass of non-liable clone households back to their paired originals until the sub-exempt and loss-making clone mass match the Advani-Summers reporter composition at the redrawn liable mass; every pair's mass and the total household mass are conserved. + outputs: [] + rewrites: [] + notes: 'Spine-only weight anchor after the amounts and asset-type stages (microcosm#970). The equal-mass clone leaves half of all household mass on gainer households, so after the Table 3 redraw the clones below the exempt amount carry millions of weighted persons that HMRC publishes no count for; the anchor derives their level from the redrawn liable mass and the Advani-Summers reporter composition and moves the excess back to each clone''s paired original, which is identical in every cell but the gains. No cell is written: amounts, flags, liable clones, zero-gain clone carriers are untouched; the groups, targets and receipt are clone-side quantities (originals are outside the anchor), every pair''s mass and the household total are conserved, and the receipt is gated at stage time.' +- stage: cgt_residential_split + survey: HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative) on the amounts the Table 3 redraw placed + source: https://www.gov.uk/government/statistics/capital-gains-tax-statistics + grain: household + artifacts: + - role: policy_parameters + kind: versioned_parameter_tree + dependency: policyengine-uk>=2.100.0 via microcosm-build[uk] + parameters: + - gov.hmrc.cgt.annual_exempt_amount + instant_rule: raw dated parameter files evaluated at 1 June of the build period's tax year + runtime_sha256_required: false + dependency_discipline: deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time + - role: cgt_asset_type_facts + kind: public_aggregate_reference + resource: hmrc_cgt_asset_type_facts.json + format: json + runtime_sha256_required: true + operations: + - kind: verify_vendored_fact_resource + artifact_role: cgt_asset_type_facts + resource: hmrc_cgt_asset_type_facts.json + feed_pin: chronicle_feed.json + record_sets: + - hmrc.cgt_table8_2026.table8a.ty2024 + - hmrc.cgt_table8_2026.table8b.ty2024 + source_vintage: 2024-25 + mapped_build_period: 2024 + period_mapping: published_tax_year_equals_build_period + require_before_source_read: true + runtime_sha256_required: true + fail_on_mismatch: true + - kind: split_liable_gainers_by_residential_probability + population: persons with net capital gains above the annual exempt amount (the national taxpayer proxy) + model: logistic probability in centred log gains with a stock shift, p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean log gain of the population, s the stock log-odds and h one where the stock signal holds, zero otherwise + stock_log_odds: 1.6 + stock_signal: household other residential property value above zero, or the person's property income above zero + parameter_solver: nested bisection; the intercept matches the expected weighted count at each trial slope, the slope matches the expected weighted gains + count_target: Table 8a 2024-25 total taxpayers reporting residential property disposals times the Table 8b 2024-25 individuals/all taxpayer share on the UK Property service + gains_target: Table 8a 2024-25 total residential property gains times the Table 8b 2024-25 individuals/all gains share on the UK Property service + basis_assumption: trusts hold the same share of the Self Assessment component as of the UK Property service, a residential gainer's whole net gain is attributed to residential property, and the gainers of one household are residential independently of each other + realization: 'probability carried as weight, no draw: a household carrying k liable gainers becomes 2^k arms, one per subset of residential gainers, at the household''s weight times the product of p over the residential gainers and 1 - p over the others; the arm with no residential gainer keeps the household''s ids; at design weights the residential count and gains equal the solved expectations sum(p w) and sum(p w g) in every gain band' + maximum_liable_gainers_per_household: 3 + gainer_order: person_id ascending within the household + id_remapping: id_multiplier_for_values; arm j takes every entity id + j x multiplier, bit m of j marking the household's m-th liable gainer residential + flag_column: household_is_cgt_residential_clone + index_column: cgt_residential_clone_index + probability_column: cgt_residential_probability + output_column: capital_gains_residential_property + output_semantics: the person's capital_gains on an arm where the person is residential, 0 otherwise + values: every other column copied unchanged; no draw, no seed, no salt + weight_kind_out: importance + conservation: exact_total + declared_factor: 1.0 + reason: Capital-gains residential split divides every household carrying liable gainers into arms weighted by the solved residential probabilities, one arm per subset of residential gainers; every household's mass and the total household mass are conserved. + outputs: + - household_is_cgt_residential_clone + - cgt_residential_clone_index + - cgt_residential_probability + - capital_gains_residential_property + rewrites: [] + nonnegative_outputs: + - cgt_residential_probability + - capital_gains_residential_property + notes: 'Structural stage after the incidence anchor and before the asset-type stage (microcosm#1063). The residential-property flag used to be realised by a weighted systematic walk, one whole flag per row, whose gains error is bounded by one row''s stake: at the top of the gains distribution one stake is several times a band''s expected residential gains, so the realised gains sat short of expectation by the lottery expectation of the giant stakes or over it by one of them. This stage solves the same logistic (intercept to the Table 8a 2024-25 taxpayers on the individuals basis, slope to the gains, with the stock shift) and carries the probability as weight instead: every household with liable gainers becomes 2^k arms at product weights, one per subset of residential gainers, the arm with no residential gainer keeping the household''s ids and arm j taking every entity id plus j times the id multiplier, the lineage rule the executor, the geography identity kernel and the identity tool read for the support split and the clone. At design weights the residential count and gains equal the solved expectations in every gain band, with no draw, no offset and no seed; the calibration, which binds hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains, decides between the arms afterwards as it does between the incidence clone''s halves. Every household''s mass and composition and the total household mass are conserved; the receipt is gated at stage time.' - stage: hmrc_cgt_asset_type_spine survey: HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative), Table 4.1 (individuals claiming Business Asset Disposal Relief or Investors' Relief by band of qualifying gain, 2024-25, administrative) and Table 7 (disposals, proceeds and gains by asset type, 2023-24, sample-based), vendored from the pinned Chronicle feed source: https://www.gov.uk/government/statistics/capital-gains-tax-statistics @@ -4024,22 +4272,8 @@ stages: require_before_source_read: true runtime_sha256_required: true fail_on_mismatch: true - - kind: assign_residential_property_flag - population: persons with net capital gains above the annual exempt amount (the national taxpayer proxy) - model: logistic probability in centred log gains with a stock shift, p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean log gain of the population, s the stock log-odds and h one where the stock signal holds, zero otherwise - stock_log_odds: 1.6 - stock_signal: household other residential property value above zero, or the person's property income above zero - parameter_solver: nested bisection; the intercept matches the expected weighted count at each trial slope, the slope matches the expected weighted gains - count_target: Table 8a 2024-25 total taxpayers reporting residential property disposals times the Table 8b 2024-25 individuals/all taxpayer share on the UK Property service - gains_target: Table 8a 2024-25 total residential property gains times the Table 8b 2024-25 individuals/all gains share on the UK Property service - basis_assumption: trusts hold the same share of the Self Assessment component as of the UK Property service, and a flagged person's whole net gain is attributed to residential property - realization: 'weighted systematic sampling in ascending gain order with one seeded offset: a person is flagged when the expected weight owed so far reaches their own weight, so the flagged weight tracks the expected weight within one person''s weight along the gains axis and the realised count and gains sit close to their expectations; the Bernoulli sigma of each is reported as the envelope' - seed: 553 - seed_mixing: seed combined with the build period - output_column: capital_gains_residential_property - output_semantics: the person's capital_gains where flagged residential, 0 otherwise - kind: assign_badr_qualifying_gains - population: liable gainers not flagged residential + population: liable gainers whose arm is not residential (capital_gains_residential_property == 0, cgt_residential_split) bands: HMRC Table 4.1 2024-25 individuals claiming Business Asset Disposal Relief or Investors' Relief by band of qualifying gain (lower limits 0, 10,000, 25,000, 50,000, 100,000, 250,000, 500,000 and 1,000,000) band_pool: the population with net gains inside the band's range; the open top band takes net gains at or above the lifetime limit model: 'per band below the lifetime limit, a logistic probability in centred log gains with the stock shift, solved like the residential flag: the intercept matches the band''s published claimant count and the slope its published qualifying gains, with zero slope where the band pool''s own mean already matches; the open top band has zero slope, so its probability varies only with the stock shift, and its intercept matches its count' @@ -4079,12 +4313,13 @@ stages: unlisted_shares: household corporate wealth above zero, or the person's self-employment income above zero other_financial_assets: household gross financial wealth above zero share_targets: Table 7 2023-24 gains by asset type excluding residential land and buildings, as shares - weight_fitting: multiplicative updates of the type weights, with claimants restricted to the claimant categories, until the household-weighted expected gains shares by type match the share targets; the fit refuses if it has not converged within the iteration limit, and refuses up front when claimants hold more of the gains than the claimant categories' share + claimant_share_targets: the share targets among the claimant categories, each over their sum + non_claimant_share_targets: (share target - k x claimant share target) / (1 - k) with k the claimants' share of the non-residential gains, the composition that restores the share targets over claimants and non-claimants together + basis_assumption: Table 7 does not split gains by relief, so claimants are given its composition among the claimant categories + weight_fitting: 'one fit per claimant status: multiplicative updates of the type weights until the household-weighted expected gains shares by type match that status''s share targets; a fit refuses if it has not converged within the iteration limit, and the stage refuses up front when claimants hold more of the gains than the claimant categories'' share' weight_fitting_iterations: 200 weight_fitting_tolerance: 1.0e-06 - realization: one seeded uniform per person against the cumulative type probabilities, consumed in person_id order - seed: 554 - seed_mixing: seed combined with the build period + realization: 'error diffusion per claimant status in descending (gain, person_id) order: each person adds weight x gain x type probability to a running balance per type and takes the type it may hold with the largest balance, which is drawn down by the whole weight x gain, so the realised gains by type track the fitted composition instead of following the heaviest rows; deterministic, no seed' output_column: capital_gains_asset_type value_domain: - none @@ -4113,80 +4348,10 @@ stages: adjudication: https://github.com/PolicyEngine/microcosm/issues/1014 outputs: - capital_gains_asset_type - - capital_gains_residential_property - capital_gains_badr nonnegative_outputs: - - capital_gains_residential_property - capital_gains_badr - notes: 'Spine stage after the amounts redraw: every liable gainer carries one main asset type; where flagged residential, its net gain is restated in capital_gains_residential_property; where drawn as a BADR or Investors'' Relief claimant, its qualifying gain is written to capital_gains_badr and its main type is one the reliefs apply to. The residential flag is solved to the Table 8a 2024-25 totals on the individuals basis, which the hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains targets bind; the BADR claims are solved band by band to Table 4.1 2024-25, whose individuals band rows the hmrc.cgt.badr_ir_taxpayers_by_band and hmrc.cgt.badr_ir_qualifying_gains_by_band targets bind (microcosm#1014); the non-residential composition follows the fenced Table 7 2023-24 prior and is diagnostic only. Carried interest (Table 9) is not modelled (microcosm#725).' -- stage: cgt_incidence_anchor - survey: Family Resources Survey 2024-25, SPI synthetic support, the HMRC Table 3 redrawn gains on the spine and Advani-Summers capital-gains incidence - source: Advani and Summers (2020), Capital Gains and UK Inequality, CAGE Working Paper 465 - grain: household - artifacts: - - role: capital_gains_incidence_and_quantiles - kind: public_aggregate_reference - resource: advani_summers_capital_gains_distribution.json - format: json - runtime_sha256_required: true - - role: policy_parameters - kind: versioned_parameter_tree - dependency: policyengine-uk>=2.98.0 via microcosm-build[uk] - parameters: - - gov.hmrc.cgt.annual_exempt_amount - instant_rule: raw dated parameter files evaluated at 1 June of the build period's tax year - runtime_sha256_required: false - dependency_discipline: deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time - operations: - - kind: pair_clone_households_to_originals - clone_flag_column: household_is_capital_gains_clone - id_remapping: id_multiplier_for_values - pairing: clone household_id minus the clone stage's id multiplier, recomputed from the pre-clone rows (clone flag false; the support split's copies are pre-clone rows with their own clones) - requirement: bijection between clone and original households; an unpaired clone or original fails the build - - kind: derive_reporter_composition_from_banded_quantiles - resource: advani_summers_capital_gains_distribution.json - income_proxy_components: - - employment_income - - self_employment_income - - state_pension_reported - - private_pension_income - - property_income - - savings_interest_income - - dividend_income - - miscellaneous_income - allowance_subtraction: false - carrier: oldest adult; person_id ascending breaks age ties - adult_minimum_age: 16 - quantile_points: - - 0.05 - - 0.1 - - 0.25 - - 0.5 - - 0.75 - - 0.9 - - 0.95 - spline_degree: 1 - extrapolation: ext=0 - crossings: each band's degree-1 quantile spline inverted at zero and at the build period's annual exempt amount - aggregation: crossings averaged over clone carriers weighted by household weight times the band's percent_with_gains - liable_mass: weighted persons with capital_gains above the annual exempt amount on every household after the Table 3 redraw - targets: sub_exempt = liable_mass * (exempt_quantile - zero_quantile) / (1 - exempt_quantile); loss = liable_mass * zero_quantile / (1 - exempt_quantile) - - kind: transfer_clone_mass_to_originals - population: clone households paired to their originals; originals are outside the anchor, so the groups, targets and masses are clone-side quantities, not population counts - sub_exempt_group: clone households whose carrier's capital_gains lie in (0, annual exempt amount]; a carrier with exactly zero gains is no reporter and is left untouched - loss_group: clone households whose carrier's capital_gains are negative - liable_clones: untouched - factor: min(maximum_factor, scale * percent_with_gains) with scale solved per group by bisection so the group's mass equals its target; a group at or below its target is untouched - maximum_factor: 1.0 - transfer: each clone's removed mass is added to its paired original, so every pair's mass is conserved to rounding; no global exact-total correction is applied, so no household outside a trimmed pair moves - weight_kind_out: importance - conservation: household_total_to_rounding - pair_conservation: to_rounding - declared_factor: 1.0 - reason: Capital-gains incidence anchor moves the mass of non-liable clone households back to their paired originals until the sub-exempt and loss-making clone mass match the Advani-Summers reporter composition at the redrawn liable mass; every pair's mass and the total household mass are conserved. - outputs: [] - rewrites: [] - notes: 'Spine-only weight anchor after the amounts and asset-type stages (microcosm#970). The equal-mass clone leaves half of all household mass on gainer households, so after the Table 3 redraw the clones below the exempt amount carry millions of weighted persons that HMRC publishes no count for; the anchor derives their level from the redrawn liable mass and the Advani-Summers reporter composition and moves the excess back to each clone''s paired original, which is identical in every cell but the gains. No cell is written: amounts, flags, liable clones, zero-gain clone carriers are untouched; the groups, targets and receipt are clone-side quantities (originals are outside the anchor), every pair''s mass and the household total are conserved, and the receipt is gated at stage time.' + notes: 'Spine stage after the amounts redraw, the incidence anchor and the residential split: every liable gainer carries one main asset type; a residential arm (capital_gains_residential_property above zero, written by cgt_residential_split since microcosm#1063) is typed residential and its net gain is already restated there; where drawn as a BADR or Investors'' Relief claimant, its qualifying gain is written to capital_gains_badr and its main type is one the reliefs apply to. The residential arms carry the Table 8a 2024-25 totals on the individuals basis at design weights, which the hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains targets bind; the BADR claims are solved band by band to Table 4.1 2024-25, whose individuals band rows the hmrc.cgt.badr_ir_taxpayers_by_band and hmrc.cgt.badr_ir_qualifying_gains_by_band targets bind (microcosm#1014); the non-residential composition follows the fenced Table 7 2023-24 prior and is diagnostic only: since microcosm#1063 the type is fitted once per claimant status (claimants to Table 7''s composition among the types the reliefs apply to, non-claimants to the composition that restores Table 7 overall) and realised by a deterministic walk that keeps the realised gains by type on the fit. Carried interest (Table 9) is not modelled (microcosm#725).' - stage: salary_sacrifice survey: Family Resources Survey 2024-25 salary-sacrifice respondents and HMRC salary-sacrifice reform analysis source: HMRC, Salary sacrifice reform for pension contributions effective from 6 April 2029 @@ -4200,7 +4365,7 @@ stages: operations: - kind: fit_weighted_qrf training_population: support_channel == frs and not capital-gains clone and not CGT support copy and salary_sacrifice_asked == 1 - target_population: salary_sacrifice_asked != 1 frame-wide + target_population: salary_sacrifice_asked != 1 and employment_income > 0, frame-wide; people not asked who have no employment income are set to zero predictors: - age - employment_income @@ -4218,7 +4383,7 @@ stages: target: 5400000 donor_pool: employee_pension_contributions > 0 and pension_contributions_via_salary_sacrifice == 0 and employment_income > 0 rate_cap: 0.5 - move: full employee_pension_contributions to pension_contributions_via_salary_sacrifice; source zeroed; employment_income lowered by the amount sacrificed + move: full employee_pension_contributions to pension_contributions_via_salary_sacrifice; source zeroed; employment_income lowered by the amount sacrificed; the converted record's pay before the conversion kept on salary_sacrifice_pre_conversion_pay (zero elsewhere) seed: 2024 salt: salary_sacrifice_conversion receipt: weighted_headcount @@ -4227,10 +4392,12 @@ stages: - pension_contributions_via_salary_sacrifice - employee_pension_contributions - employment_income + - salary_sacrifice_pre_conversion_pay nonnegative_outputs: - pension_contributions_via_salary_sacrifice - employee_pension_contributions - employment_income + - salary_sacrifice_pre_conversion_pay rewrites: - pension_contributions_via_salary_sacrifice - employee_pension_contributions diff --git a/packages/microcosm-build/src/microcosm/build/uk/take_up_contract.json b/packages/microcosm-build/src/microcosm/build/uk/take_up_contract.json index fc2cbcdbb..2cd674dec 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/take_up_contract.json +++ b/packages/microcosm-build/src/microcosm/build/uk/take_up_contract.json @@ -24,7 +24,7 @@ "source": { "source": "Child Benefit statistics: 2022 annual release", "agency": "HMRC", - "citation": "Child Benefit statistics annual release, August 2022: take-up fell to 89% in 2022.", + "citation": "Child Benefit statistics annual release, August 2022: take-up fell to 89% in 2022. Since microcosm#1063 this is only frs_take_up's early draw, which the intermediate engine simulations read; child_benefit_take_up replaces it after the SPI income chain with HMRC's August 2025 claim rates of eligible children by single year of age (hmrc_child_benefit_statistics.json).", "url": "https://www.gov.uk/government/statistics/child-benefit-statistics-annual-release-august-2022/child-benefit-statistics-annual-release-data-at-august-2022", "status": "sourced_administrative" } @@ -39,7 +39,7 @@ "source": { "source": "Child Benefit Statistics: Annual Release, August 2022", "agency": "HMRC", - "citation": "Incumbent derivation: 683k opt-out families over roughly 3m fully HICBC-liable families in the 2023 FRS.", + "citation": "Incumbent derivation: 683k opt-out families over roughly 3m fully HICBC-liable families in the 2023 FRS. Since microcosm#1063 this is only frs_take_up's early draw over every benefit unit; child_benefit_take_up replaces it after the SPI income chain with the published opted-out share of registered families (August 2025), drawn among claiming families charged the whole benefit.", "url": "https://www.gov.uk/government/statistics/child-benefit-statistics-annual-release-august-2022/child-benefit-statistics-annual-release-data-at-august-2022", "status": "sourced_administrative_derived" } diff --git a/packages/microcosm-build/src/microcosm/build/uk/target_fit_reviewed_exclusions.json b/packages/microcosm-build/src/microcosm/build/uk/target_fit_reviewed_exclusions.json index 1e9d92c1d..d7f534547 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/target_fit_reviewed_exclusions.json +++ b/packages/microcosm-build/src/microcosm/build/uk/target_fit_reviewed_exclusions.json @@ -1,5 +1,13 @@ { - "schema_version": 2, - "description": "Reviewed target-fit exclusions for the UK terminal battery (schema 2 per #610). Each entry defers one calibration target whose |relative error| breaches the 25% release bound, with the defect diagnosed, tracked, and owned elsewhere; the receipt records the reasoning, approver, adjudication reference, approval date, and expiry. The gate keeps the target BOUND in calibration (exclusion applies to the release fence only, never the solve), reports dormant entries, FAILS stale ones (targets back inside the bound), and FAILS expired ones until the adjudication is renewed, so this register cannot rot.", - "exclusions": {} + "schema_version": 2, + "description": "Reviewed target-fit exclusions for the UK terminal battery (schema 2 per #610). Each entry defers one calibration target whose |relative error| breaches the 25% release bound, with the defect diagnosed, tracked, and owned elsewhere; the receipt records the reasoning, approver, adjudication reference, approval date, and expiry. The gate keeps the target BOUND in calibration (exclusion applies to the release fence only, never the solve), reports dormant entries, FAILS stale ones (targets back inside the bound), and FAILS expired ones until the adjudication is renewed, so this register cannot rot.", + "exclusions": { + "hmrc.spi_region.income_tax_by_region_12570_15000@E12000008@2025": { + "reason": "The frame carries too few taxpayers just above the personal allowance: 0.278m South East taxpayers with total income in GBP 12,570-15,000 at design weights against HMRC's projected 0.364m (nationally 2.31m against 2.97m; the North West and London are 28-31% short, the East 6%), and the ones it carries pay GBP 250 of income tax each against HMRC's GBP 215 (they sit higher in the band). The bound taxpayer-count rows make the solver scale these households up by a third to hit the count exactly, and the income-tax cell, carried by the same households, lands at +30.3% (its total-income sibling at +11.2%); main pulls the same cell from -20% to +22.6% by the same route. With the Table 3.6 amount rows projected by HMRC's band growth (microcosm#1063, 2026-10-02) the national employment rows fit within 4-12% and the cell still breaches, so the pull is the count gap, not the employment rows. The row was deferred on 2026-09-18 and retired on microcosm#1012 at +24.5%; deferred four weeks again while the lowest taxpayer band's support is corrected in the frame (the SPI support channel's coverage of the band and the FRS's small incomes around the frozen allowance), which is where the gap lives.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (the stack arms of 2026-10-02 on the review of #1089)", + "approved_on": "2026-10-02", + "expires_on": "2026-11-02" + } + } } diff --git a/packages/microcosm-build/src/microcosm/build/uk/target_references.json b/packages/microcosm-build/src/microcosm/build/uk/target_references.json index 4af44a21d..202b924c3 100644 --- a/packages/microcosm-build/src/microcosm/build/uk/target_references.json +++ b/packages/microcosm-build/src/microcosm/build/uk/target_references.json @@ -577,20 +577,20 @@ "dwp.attendance_allowance.recipients_england": "Attendance Allowance recipients, England", "dwp.attendance_allowance.recipients_wales": "Attendance Allowance recipients, Wales", "dwp.benefit_cap.capped_households": "Capped households", - "dwp.benefit_cap.capped_households_1000_01_to_1100": "Capped households, \u00a31,000.01 to \u00a31,100 a month", - "dwp.benefit_cap.capped_households_100_01_to_200": "Capped households, \u00a3100.01 to \u00a3200 a month", - "dwp.benefit_cap.capped_households_1100_01_to_1200": "Capped households, \u00a31,100.01 to \u00a31,200 a month", - "dwp.benefit_cap.capped_households_1200_01_to_1300": "Capped households, \u00a31,200.01 to \u00a31,300 a month", - "dwp.benefit_cap.capped_households_1300_01_and_above": "Capped households, \u00a31,300.01 and above a month", - "dwp.benefit_cap.capped_households_200_01_to_300": "Capped households, \u00a3200.01 to \u00a3300 a month", - "dwp.benefit_cap.capped_households_300_01_to_400": "Capped households, \u00a3300.01 to \u00a3400 a month", - "dwp.benefit_cap.capped_households_400_01_to_500": "Capped households, \u00a3400.01 to \u00a3500 a month", - "dwp.benefit_cap.capped_households_500_01_to_600": "Capped households, \u00a3500.01 to \u00a3600 a month", - "dwp.benefit_cap.capped_households_600_01_to_700": "Capped households, \u00a3600.01 to \u00a3700 a month", - "dwp.benefit_cap.capped_households_700_01_to_800": "Capped households, \u00a3700.01 to \u00a3800 a month", - "dwp.benefit_cap.capped_households_800_01_to_900": "Capped households, \u00a3800.01 to \u00a3900 a month", - "dwp.benefit_cap.capped_households_900_01_to_1000": "Capped households, \u00a3900.01 to \u00a31,000 a month", - "dwp.benefit_cap.capped_households_up_to_100": "Capped households, Up to \u00a3100 a month", + "dwp.benefit_cap.capped_households_1000_01_to_1100": "Capped households, £1,000.01 to £1,100 a month", + "dwp.benefit_cap.capped_households_100_01_to_200": "Capped households, £100.01 to £200 a month", + "dwp.benefit_cap.capped_households_1100_01_to_1200": "Capped households, £1,100.01 to £1,200 a month", + "dwp.benefit_cap.capped_households_1200_01_to_1300": "Capped households, £1,200.01 to £1,300 a month", + "dwp.benefit_cap.capped_households_1300_01_and_above": "Capped households, £1,300.01 and above a month", + "dwp.benefit_cap.capped_households_200_01_to_300": "Capped households, £200.01 to £300 a month", + "dwp.benefit_cap.capped_households_300_01_to_400": "Capped households, £300.01 to £400 a month", + "dwp.benefit_cap.capped_households_400_01_to_500": "Capped households, £400.01 to £500 a month", + "dwp.benefit_cap.capped_households_500_01_to_600": "Capped households, £500.01 to £600 a month", + "dwp.benefit_cap.capped_households_600_01_to_700": "Capped households, £600.01 to £700 a month", + "dwp.benefit_cap.capped_households_700_01_to_800": "Capped households, £700.01 to £800 a month", + "dwp.benefit_cap.capped_households_800_01_to_900": "Capped households, £800.01 to £900 a month", + "dwp.benefit_cap.capped_households_900_01_to_1000": "Capped households, £900.01 to £1,000 a month", + "dwp.benefit_cap.capped_households_up_to_100": "Capped households, Up to £100 a month", "dwp.esa_claimants": "Employment and Support Allowance claimants", "dwp.esa_contrib_claimants": "Employment and Support Allowance contrib claimants", "dwp.esa_income_claimants": "Employment and Support Allowance income claimants", @@ -600,7 +600,7 @@ "dwp.hb.households_pension_age_social_rented": "Housing Benefit benefit units, pension age, social rented sector", "dwp.hb.households_private_rented": "Housing Benefit claimants, private rented", "dwp.hb.households_social_rented": "Housing Benefit claimants, social rented", - "dwp.jsa_claimants": "Jobseeker\u2019s Allowance claimants", + "dwp.jsa_claimants": "Jobseeker’s Allowance claimants", "dwp.pension_credit.benefit_units": "Pension Credit benefit units, Great Britain", "dwp.pension_credit.benefit_units_65_69": "Pension Credit benefit units, claimant aged 65 to 69", "dwp.pension_credit.benefit_units_70_74": "Pension Credit benefit units, claimant aged 70 to 74", @@ -1527,7 +1527,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1552,7 +1552,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1577,7 +1577,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1602,7 +1602,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1627,7 +1627,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1652,7 +1652,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1677,7 +1677,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1702,7 +1702,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1727,7 +1727,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1752,7 +1752,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1777,7 +1777,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1802,7 +1802,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -1827,7 +1827,7 @@ "contract_target_id": "hmrc.spi.employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.average_earnings", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2164,7 +2164,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2189,7 +2189,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2214,7 +2214,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2239,7 +2239,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2264,7 +2264,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2289,7 +2289,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2314,7 +2314,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2339,7 +2339,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2364,7 +2364,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2389,7 +2389,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2414,7 +2414,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2439,7 +2439,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -2464,7 +2464,7 @@ "contract_target_id": "hmrc.spi.self_employment_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.per_capita.mixed_income", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3750,7 +3750,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3775,7 +3775,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3800,7 +3800,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3825,7 +3825,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3850,7 +3850,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3875,7 +3875,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3900,7 +3900,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3925,7 +3925,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3950,7 +3950,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -3975,7 +3975,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -4000,7 +4000,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -4025,7 +4025,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -4050,7 +4050,7 @@ "contract_target_id": "hmrc.spi.private_pension_income.amount_by_total_income_band", "measure_kind": "prepared_column" }, - "uprating_index": "policyengine_uk_parameter:gov.economic_assumptions.indices.obr.private_pension_index", + "uprating_index": "hmrc.itl_2026.total_income_growth_by_total_income_band", "uprating_from_period": "2023", "uprating_to_period": 2025 }, @@ -5778,7 +5778,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a30.01 to \u00a3100.00" + "monthly_award_amount_bands": "£0.01 to £100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5808,7 +5808,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3100.01 to \u00a3200.00" + "monthly_award_amount_bands": "£100.01 to £200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5838,7 +5838,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31000.01 to \u00a31100.00" + "monthly_award_amount_bands": "£1000.01 to £1100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5868,7 +5868,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31100.01 to \u00a31200.00" + "monthly_award_amount_bands": "£1100.01 to £1200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5898,7 +5898,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31200.01 to \u00a31300.00" + "monthly_award_amount_bands": "£1200.01 to £1300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5928,7 +5928,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31300.01 to \u00a31400.00" + "monthly_award_amount_bands": "£1300.01 to £1400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5958,7 +5958,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31400.01 to \u00a31500.00" + "monthly_award_amount_bands": "£1400.01 to £1500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -5988,7 +5988,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31500.01 to \u00a31600.00" + "monthly_award_amount_bands": "£1500.01 to £1600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6018,7 +6018,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31600.01 to \u00a31700.00" + "monthly_award_amount_bands": "£1600.01 to £1700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6048,7 +6048,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31700.01 to \u00a31800.00" + "monthly_award_amount_bands": "£1700.01 to £1800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6078,7 +6078,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31800.01 to \u00a31900.00" + "monthly_award_amount_bands": "£1800.01 to £1900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6108,7 +6108,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a31900.01 to \u00a32000.00" + "monthly_award_amount_bands": "£1900.01 to £2000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6138,7 +6138,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3200.01 to \u00a3300.00" + "monthly_award_amount_bands": "£200.01 to £300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6168,7 +6168,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a32000.01 to \u00a32100.00" + "monthly_award_amount_bands": "£2000.01 to £2100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6198,7 +6198,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a32100.01 to \u00a32200.00" + "monthly_award_amount_bands": "£2100.01 to £2200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6228,7 +6228,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a32200.01 to \u00a32300.00" + "monthly_award_amount_bands": "£2200.01 to £2300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6258,7 +6258,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a32300.01 to \u00a32400.00" + "monthly_award_amount_bands": "£2300.01 to £2400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6288,7 +6288,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a32400.01 to \u00a32500.00" + "monthly_award_amount_bands": "£2400.01 to £2500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6318,7 +6318,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3300.01 to \u00a3400.00" + "monthly_award_amount_bands": "£300.01 to £400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6348,7 +6348,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3400.01 to \u00a3500.00" + "monthly_award_amount_bands": "£400.01 to £500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6378,7 +6378,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3500.01 to \u00a3600.00" + "monthly_award_amount_bands": "£500.01 to £600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6408,7 +6408,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3600.01 to \u00a3700.00" + "monthly_award_amount_bands": "£600.01 to £700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6438,7 +6438,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3700.01 to \u00a3800.00" + "monthly_award_amount_bands": "£700.01 to £800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6468,7 +6468,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3800.01 to \u00a3900.00" + "monthly_award_amount_bands": "£800.01 to £900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6498,7 +6498,7 @@ ], "dimension_values": { "family_type": "Single, no children", - "monthly_award_amount_bands": "\u00a3900.01 to \u00a31000.00" + "monthly_award_amount_bands": "£900.01 to £1000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6528,7 +6528,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a30.01 to \u00a3100.00" + "monthly_award_amount_bands": "£0.01 to £100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6558,7 +6558,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3100.01 to \u00a3200.00" + "monthly_award_amount_bands": "£100.01 to £200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6588,7 +6588,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31000.01 to \u00a31100.00" + "monthly_award_amount_bands": "£1000.01 to £1100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6618,7 +6618,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31100.01 to \u00a31200.00" + "monthly_award_amount_bands": "£1100.01 to £1200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6648,7 +6648,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31200.01 to \u00a31300.00" + "monthly_award_amount_bands": "£1200.01 to £1300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6678,7 +6678,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31300.01 to \u00a31400.00" + "monthly_award_amount_bands": "£1300.01 to £1400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6708,7 +6708,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31400.01 to \u00a31500.00" + "monthly_award_amount_bands": "£1400.01 to £1500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6738,7 +6738,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31500.01 to \u00a31600.00" + "monthly_award_amount_bands": "£1500.01 to £1600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6768,7 +6768,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31600.01 to \u00a31700.00" + "monthly_award_amount_bands": "£1600.01 to £1700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6798,7 +6798,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31700.01 to \u00a31800.00" + "monthly_award_amount_bands": "£1700.01 to £1800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6828,7 +6828,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31800.01 to \u00a31900.00" + "monthly_award_amount_bands": "£1800.01 to £1900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6858,7 +6858,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a31900.01 to \u00a32000.00" + "monthly_award_amount_bands": "£1900.01 to £2000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6888,7 +6888,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3200.01 to \u00a3300.00" + "monthly_award_amount_bands": "£200.01 to £300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6918,7 +6918,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a32000.01 to \u00a32100.00" + "monthly_award_amount_bands": "£2000.01 to £2100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6948,7 +6948,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a32100.01 to \u00a32200.00" + "monthly_award_amount_bands": "£2100.01 to £2200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -6978,7 +6978,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a32200.01 to \u00a32300.00" + "monthly_award_amount_bands": "£2200.01 to £2300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7008,7 +7008,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a32300.01 to \u00a32400.00" + "monthly_award_amount_bands": "£2300.01 to £2400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7038,7 +7038,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a32400.01 to \u00a32500.00" + "monthly_award_amount_bands": "£2400.01 to £2500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7068,7 +7068,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3300.01 to \u00a3400.00" + "monthly_award_amount_bands": "£300.01 to £400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7098,7 +7098,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3400.01 to \u00a3500.00" + "monthly_award_amount_bands": "£400.01 to £500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7128,7 +7128,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3500.01 to \u00a3600.00" + "monthly_award_amount_bands": "£500.01 to £600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7158,7 +7158,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3600.01 to \u00a3700.00" + "monthly_award_amount_bands": "£600.01 to £700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7188,7 +7188,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3700.01 to \u00a3800.00" + "monthly_award_amount_bands": "£700.01 to £800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7218,7 +7218,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3800.01 to \u00a3900.00" + "monthly_award_amount_bands": "£800.01 to £900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7248,7 +7248,7 @@ ], "dimension_values": { "family_type": "Single, with children", - "monthly_award_amount_bands": "\u00a3900.01 to \u00a31000.00" + "monthly_award_amount_bands": "£900.01 to £1000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7278,7 +7278,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a30.01 to \u00a3100.00" + "monthly_award_amount_bands": "£0.01 to £100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7308,7 +7308,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3100.01 to \u00a3200.00" + "monthly_award_amount_bands": "£100.01 to £200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7338,7 +7338,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31000.01 to \u00a31100.00" + "monthly_award_amount_bands": "£1000.01 to £1100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7368,7 +7368,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31100.01 to \u00a31200.00" + "monthly_award_amount_bands": "£1100.01 to £1200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7398,7 +7398,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31200.01 to \u00a31300.00" + "monthly_award_amount_bands": "£1200.01 to £1300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7428,7 +7428,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31300.01 to \u00a31400.00" + "monthly_award_amount_bands": "£1300.01 to £1400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7458,7 +7458,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31400.01 to \u00a31500.00" + "monthly_award_amount_bands": "£1400.01 to £1500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7488,7 +7488,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31500.01 to \u00a31600.00" + "monthly_award_amount_bands": "£1500.01 to £1600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7518,7 +7518,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31600.01 to \u00a31700.00" + "monthly_award_amount_bands": "£1600.01 to £1700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7548,7 +7548,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31700.01 to \u00a31800.00" + "monthly_award_amount_bands": "£1700.01 to £1800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7578,7 +7578,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31800.01 to \u00a31900.00" + "monthly_award_amount_bands": "£1800.01 to £1900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7608,7 +7608,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a31900.01 to \u00a32000.00" + "monthly_award_amount_bands": "£1900.01 to £2000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7638,7 +7638,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3200.01 to \u00a3300.00" + "monthly_award_amount_bands": "£200.01 to £300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7668,7 +7668,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a32000.01 to \u00a32100.00" + "monthly_award_amount_bands": "£2000.01 to £2100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7698,7 +7698,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a32100.01 to \u00a32200.00" + "monthly_award_amount_bands": "£2100.01 to £2200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7728,7 +7728,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a32200.01 to \u00a32300.00" + "monthly_award_amount_bands": "£2200.01 to £2300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7758,7 +7758,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a32300.01 to \u00a32400.00" + "monthly_award_amount_bands": "£2300.01 to £2400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7788,7 +7788,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a32400.01 to \u00a32500.00" + "monthly_award_amount_bands": "£2400.01 to £2500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7818,7 +7818,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3300.01 to \u00a3400.00" + "monthly_award_amount_bands": "£300.01 to £400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7848,7 +7848,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3400.01 to \u00a3500.00" + "monthly_award_amount_bands": "£400.01 to £500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7878,7 +7878,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3500.01 to \u00a3600.00" + "monthly_award_amount_bands": "£500.01 to £600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7908,7 +7908,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3600.01 to \u00a3700.00" + "monthly_award_amount_bands": "£600.01 to £700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7938,7 +7938,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3700.01 to \u00a3800.00" + "monthly_award_amount_bands": "£700.01 to £800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7968,7 +7968,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3800.01 to \u00a3900.00" + "monthly_award_amount_bands": "£800.01 to £900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -7998,7 +7998,7 @@ ], "dimension_values": { "family_type": "Couple, no children", - "monthly_award_amount_bands": "\u00a3900.01 to \u00a31000.00" + "monthly_award_amount_bands": "£900.01 to £1000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8028,7 +8028,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a30.01 to \u00a3100.00" + "monthly_award_amount_bands": "£0.01 to £100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8058,7 +8058,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3100.01 to \u00a3200.00" + "monthly_award_amount_bands": "£100.01 to £200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8088,7 +8088,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31000.01 to \u00a31100.00" + "monthly_award_amount_bands": "£1000.01 to £1100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8118,7 +8118,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31100.01 to \u00a31200.00" + "monthly_award_amount_bands": "£1100.01 to £1200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8148,7 +8148,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31200.01 to \u00a31300.00" + "monthly_award_amount_bands": "£1200.01 to £1300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8178,7 +8178,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31300.01 to \u00a31400.00" + "monthly_award_amount_bands": "£1300.01 to £1400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8208,7 +8208,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31400.01 to \u00a31500.00" + "monthly_award_amount_bands": "£1400.01 to £1500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8238,7 +8238,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31500.01 to \u00a31600.00" + "monthly_award_amount_bands": "£1500.01 to £1600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8268,7 +8268,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31600.01 to \u00a31700.00" + "monthly_award_amount_bands": "£1600.01 to £1700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8298,7 +8298,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31700.01 to \u00a31800.00" + "monthly_award_amount_bands": "£1700.01 to £1800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8328,7 +8328,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31800.01 to \u00a31900.00" + "monthly_award_amount_bands": "£1800.01 to £1900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8358,7 +8358,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a31900.01 to \u00a32000.00" + "monthly_award_amount_bands": "£1900.01 to £2000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8388,7 +8388,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3200.01 to \u00a3300.00" + "monthly_award_amount_bands": "£200.01 to £300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8418,7 +8418,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a32000.01 to \u00a32100.00" + "monthly_award_amount_bands": "£2000.01 to £2100.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8448,7 +8448,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a32100.01 to \u00a32200.00" + "monthly_award_amount_bands": "£2100.01 to £2200.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8478,7 +8478,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a32200.01 to \u00a32300.00" + "monthly_award_amount_bands": "£2200.01 to £2300.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8508,7 +8508,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a32300.01 to \u00a32400.00" + "monthly_award_amount_bands": "£2300.01 to £2400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8538,7 +8538,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a32400.01 to \u00a32500.00" + "monthly_award_amount_bands": "£2400.01 to £2500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8568,7 +8568,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3300.01 to \u00a3400.00" + "monthly_award_amount_bands": "£300.01 to £400.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8598,7 +8598,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3400.01 to \u00a3500.00" + "monthly_award_amount_bands": "£400.01 to £500.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8628,7 +8628,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3500.01 to \u00a3600.00" + "monthly_award_amount_bands": "£500.01 to £600.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8658,7 +8658,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3600.01 to \u00a3700.00" + "monthly_award_amount_bands": "£600.01 to £700.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8688,7 +8688,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3700.01 to \u00a3800.00" + "monthly_award_amount_bands": "£700.01 to £800.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8718,7 +8718,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3800.01 to \u00a3900.00" + "monthly_award_amount_bands": "£800.01 to £900.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -8748,7 +8748,7 @@ ], "dimension_values": { "family_type": "Couple, with children", - "monthly_award_amount_bands": "\u00a3900.01 to \u00a31000.00" + "monthly_award_amount_bands": "£900.01 to £1000.00" }, "geography_level": "country", "geography_id": "K03000001" @@ -43536,7 +43536,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "Under \u00a320.00" + "grouped_amount_of_benefit": "Under £20.00" }, "entity_name": "person", "period_type": "month", @@ -43575,7 +43575,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a3100.00 to under \u00a3150.00" + "grouped_amount_of_benefit": "£100.00 to under £150.00" }, "entity_name": "person", "period_type": "month", @@ -43614,7 +43614,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a3150.00 to under \u00a3200.00" + "grouped_amount_of_benefit": "£150.00 to under £200.00" }, "entity_name": "person", "period_type": "month", @@ -43653,7 +43653,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a320.00 to under \u00a340.00" + "grouped_amount_of_benefit": "£20.00 to under £40.00" }, "entity_name": "person", "period_type": "month", @@ -43692,7 +43692,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a3200.00 to under \u00a3250.00" + "grouped_amount_of_benefit": "£200.00 to under £250.00" }, "entity_name": "person", "period_type": "month", @@ -43731,7 +43731,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a3250.00 to under \u00a3300.00" + "grouped_amount_of_benefit": "£250.00 to under £300.00" }, "entity_name": "person", "period_type": "month", @@ -43770,7 +43770,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a3300.00 and over" + "grouped_amount_of_benefit": "£300.00 and over" }, "entity_name": "person", "period_type": "month", @@ -43809,7 +43809,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a360.00 to under \u00a380.00" + "grouped_amount_of_benefit": "£60.00 to under £80.00" }, "entity_name": "person", "period_type": "month", @@ -43848,7 +43848,7 @@ ], "dimension_values": { "category_of_pension": "New State Pension", - "grouped_amount_of_benefit": "\u00a380.00 to under \u00a3100.00" + "grouped_amount_of_benefit": "£80.00 to under £100.00" }, "entity_name": "person", "period_type": "month", @@ -43887,7 +43887,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "Under \u00a320.00" + "grouped_amount_of_benefit": "Under £20.00" }, "entity_name": "person", "period_type": "month", @@ -43926,7 +43926,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a3100.00 to under \u00a3150.00" + "grouped_amount_of_benefit": "£100.00 to under £150.00" }, "entity_name": "person", "period_type": "month", @@ -43965,7 +43965,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a3150.00 to under \u00a3200.00" + "grouped_amount_of_benefit": "£150.00 to under £200.00" }, "entity_name": "person", "period_type": "month", @@ -44004,7 +44004,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a320.00 to under \u00a340.00" + "grouped_amount_of_benefit": "£20.00 to under £40.00" }, "entity_name": "person", "period_type": "month", @@ -44043,7 +44043,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a3200.00 to under \u00a3250.00" + "grouped_amount_of_benefit": "£200.00 to under £250.00" }, "entity_name": "person", "period_type": "month", @@ -44082,7 +44082,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a3250.00 to under \u00a3300.00" + "grouped_amount_of_benefit": "£250.00 to under £300.00" }, "entity_name": "person", "period_type": "month", @@ -44121,7 +44121,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a3300.00 and over" + "grouped_amount_of_benefit": "£300.00 and over" }, "entity_name": "person", "period_type": "month", @@ -44160,7 +44160,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a340.00 to under \u00a360.00" + "grouped_amount_of_benefit": "£40.00 to under £60.00" }, "entity_name": "person", "period_type": "month", @@ -44199,7 +44199,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a360.00 to under \u00a380.00" + "grouped_amount_of_benefit": "£60.00 to under £80.00" }, "entity_name": "person", "period_type": "month", @@ -44238,7 +44238,7 @@ ], "dimension_values": { "category_of_pension": "Pre-2016 State Pension", - "grouped_amount_of_benefit": "\u00a380.00 to under \u00a3100.00" + "grouped_amount_of_benefit": "£80.00 to under £100.00" }, "entity_name": "person", "period_type": "month", diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/battery_bindings.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/battery_bindings.py index d301381da..653fe1bef 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/battery_bindings.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/battery_bindings.py @@ -30,7 +30,7 @@ import hashlib import json -from collections.abc import Callable, Mapping +from collections.abc import Callable, Iterable, Mapping from dataclasses import dataclass, replace from datetime import date, datetime from importlib.resources import files @@ -332,12 +332,57 @@ def _evaluate_nonnegative_columns( table = context.frame.table(entity) for column in table.columns: column_values.setdefault(str(column), table[column]) + # A declared column the release boundary drops is checked where it + # lives: on the spine frame the certifier supplies. Without that frame + # the column stays required and its absence fails, as before. + for column, values in _export_dropped_columns_from_spine( + context, [column for column in required if column not in column_values] + ).items(): + column_values[column] = values return nonnegative_columns_gate( column_values, required, ) +def _export_dropped_columns_from_spine( + context: EvidenceContext, columns: Iterable[str] +) -> dict[str, Any]: + """Columns the release export drops, read from the certifier's spine frame. + + ``UK_RELEASE_EXPORT_DROPPED_COLUMNS`` leave at the release boundary + (microcosm#1063 c9 and the salary-sacrifice pre-conversion pay carrier), + so a gate that checks every declared stage output cannot find them on + the release candidate. The certifier passes the spine frame as the + ``spine_frame`` artifact; this returns each requested column that is an + export-dropped column present on that frame, keyed by column name. Any + other requested column, or any column when no spine frame is supplied, + is left to the caller's missing-column path. + """ + + from microcosm.build.uk_runtime.national_frame import ( + UK_RELEASE_EXPORT_DROPPED_COLUMNS, + ) + + spine_frame = context.artifacts.get("spine_frame") + if spine_frame is None: + return {} + dropped = { + column: entity + for entity, names in UK_RELEASE_EXPORT_DROPPED_COLUMNS.items() + for column in names + } + found: dict[str, Any] = {} + for column in columns: + entity = dropped.get(str(column)) + if entity is None: + continue + table = spine_frame.table(entity) + if column in table.columns: + found[str(column)] = table[column] + return found + + def _evaluate_column_implication( context: EvidenceContext, parameters: Mapping[str, Any] ) -> GateResult: @@ -1439,6 +1484,26 @@ def _evaluate_tail_concentration( reviewed_exclusions=exclusions, ) values, weights, surface = uk_qrf_tail_concentration_columns(context.frame) + # Declared QRF outputs the release export drops are checked on the spine + # frame the certifier supplies, at the spine's person weights. + absent = [str(column) for column in surface.get("absent_columns", ())] + on_spine = _export_dropped_columns_from_spine(context, absent) + if on_spine: + spine_values, spine_weights, spine_surface = uk_qrf_tail_concentration_columns( + context.artifacts["spine_frame"], output_columns=tuple(on_spine) + ) + checked_on_spine = list(spine_surface["checked_columns"]) + for column in checked_on_spine: + values[column] = spine_values[column] + weights[column] = spine_weights[column] + surface = { + **surface, + "checked_columns": sorted([*surface["checked_columns"], *checked_on_spine]), + "absent_columns": [ + column for column in absent if column not in checked_on_spine + ], + "export_dropped_checked_on_spine": sorted(checked_on_spine), + } return uk_qrf_tail_concentration_gate( values, weights, @@ -1622,6 +1687,12 @@ def _ledger_compile_parity_required_artifacts( # microcosm#1069 c7 pension_credit_take_up stage-health check. "maximum_take_up_deviation", "minimum_entitled_units", + # microcosm#1063 child_benefit_take_up stage-health check. + "maximum_claim_rate_deviation", + "maximum_age_claim_rate_deviation", + "minimum_age_child_rows", + "maximum_opt_out_share_deviation", + "minimum_eligible_family_units", "absolute_tolerance", "household_weight_kind", "minimum_spi_households", @@ -1634,7 +1705,6 @@ def _ledger_compile_parity_required_artifacts( "max_grid_reciprocity_mismatches", "require_partition_closure", # #725 cgt_asset_type_summary stage-health check. - "maximum_gains_sigma", "maximum_solve_relative_error", "support_bounds_resource", "minimum_band_rows", @@ -1648,10 +1718,20 @@ def _ledger_compile_parity_required_artifacts( "headroom", "maximum_copy_weight", "maximum_relative_mass_deviation", - # PolicyEngine/chronicle#280 lane spi_income_band_donor_support check: the - # reserved bands and the donors each must carry. + # #1063 cgt_residential_split stage-health check: the + # identities' tolerance and the arm-count ceiling. + "maximum_identity_relative_error", + "maximum_liable_gainers_per_household", + # #1063 c9 lcfs support_clip check: the declared donor floor. + "donor_floor", + # spi_income_band_donor_support check (PolicyEngine/chronicle#280 + # lane; mass-conserving since #1063): the reserved bands, the + # seating rule's constants and the funding floor. "band_lower_bounds", - "donors_per_band", + "minimum_donors_per_band", + "maximum_donor_weight", + "minimum_funding_factor", + "maximum_band_taxpayer_deviation", # #890 energy_rake check: NEED shape at the DESNZ level at # prior weights, with the published gas-connected share, and # a converged (not truncated) terminal residual (#1012). @@ -1665,6 +1745,9 @@ def _ledger_compile_parity_required_artifacts( "trip_rates_period_value", "maximum_user_share_deviation", "maximum_trip_rate_deviation", + # #1063 wealth_coherence check: the reviewed excess of owner + # households without a main-residence value over the donor's. + "maximum_owner_share_without_main_residence_excess", } ), artifact_keys=frozenset({"stage_evidence"}), diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py index ef4dbd5b4..23e85b0ae 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py @@ -89,6 +89,7 @@ UK_SPINE_GATE_SCOPE = ( "uk_stage_was_wealth_support", + "uk_stage_was_wealth_coherence", "uk_stage_was_lisa_support", "uk_stage_nts_bus_travel_support", "uk_stage_nts_bus_travel_facts", @@ -103,8 +104,10 @@ "uk_stage_spi_support_channel_mass", "uk_stage_hmrc_spi_income_spine_identity", "uk_stage_pension_credit_take_up", + "uk_stage_child_benefit_take_up", "uk_stage_cgt_incidence_clone_mass", "uk_stage_cgt_support_split_mass", + "uk_stage_cgt_residential_split_mass", "uk_stage_spi_income_band_donors_support", "uk_stage_hmrc_cgt_gains_spine_summary", "uk_stage_hmrc_cgt_asset_type_spine_summary", diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_asset_type.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_asset_type.py index 021f89d0b..204875111 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_asset_type.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_asset_type.py @@ -132,27 +132,27 @@ "CGT_ASSET_TYPE_NONE", "CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES", "CGT_ASSET_TYPE_RESIDENTIAL", - "CGT_ASSET_TYPE_SEED", "CGT_ASSET_TYPE_SUB_AEA", "CGT_BADR_ELIGIBLE_TYPES", "CGT_BADR_FLAG_SEED", "CGT_BADR_GAINS_COLUMN", - "CGT_RESIDENTIAL_FLAG_SEED", + "CGT_CLAIMANT_STATUSES", "CGT_RESIDENTIAL_GAINS_COLUMN", + "HMRCCGTAssetTypeFacts", + "HMRCCGTBADRBand", "HMRC_CGT_ASSET_TYPE_RECORD_SETS", "HMRC_CGT_ASSET_TYPE_RESOURCE", "HMRC_CGT_TABLE4_BAND_LOWER_BOUNDS", "HMRC_CGT_TABLE4_SOURCE_SHA256", "HMRC_CGT_TABLE7_SOURCE_SHA256", "HMRC_CGT_TABLE8_SOURCE_SHA256", - "HMRCCGTAssetTypeFacts", - "HMRCCGTBADRBand", "UKCGTAssetTypeStageTransform", "UKCGTAssetTypeSummary", "UKCGTBADRParameters", "UK_CGT_ASSET_TYPE_MASS_CONSERVATION_REASON", "UK_CGT_ASSET_TYPE_STAGE_NAME", "assign_uk_cgt_asset_types", + "claimant_status_share_targets", "load_hmrc_cgt_asset_type_facts", "uk_cgt_asset_type_stage_transform", "uk_cgt_badr_parameters", @@ -254,11 +254,10 @@ #: type weights, not the width, carry the composition. CGT_ASSET_TYPE_LOG_SIGMA = 1.5 -#: Seeds are combined with the build period, as the amounts stage does; -#: distinct from the amounts stage's 552 so the two stages never share a -#: stream. -CGT_RESIDENTIAL_FLAG_SEED = 553 -CGT_ASSET_TYPE_SEED = 554 +#: The BADR seed is combined with the build period, as the amounts stage +#: does; distinct from the amounts stage's 552 so the two stages never share +#: a stream. The residential flag draws nothing since microcosm#1063: the +#: ``cgt_residential_split`` stage carries its probability as weight. CGT_BADR_FLAG_SEED = 555 _BISECTION_ITERATIONS = 200 @@ -915,6 +914,9 @@ def _weighted_systematic_flags( owed = 0.0 for index in order: weight = float(weights[index]) + if weight <= 0.0: + # A row without weight carries no mass to flag or to owe. + continue owed += weight * float(probabilities[index]) if owed >= (1.0 - offset) * weight: flags[index] = True @@ -922,11 +924,6 @@ def _weighted_systematic_flags( return flags -# --------------------------------------------------------------------------- -# BADR and Investors' Relief claims -# --------------------------------------------------------------------------- - - def _solve_band_logistic( gains: np.ndarray, weights: np.ndarray, @@ -1219,6 +1216,92 @@ def fit_type_weights( return type_weights, probabilities, iterations +CGT_CLAIMANT_STATUSES: tuple[str, ...] = ("claimants", "non_claimants") + + +def claimant_status_share_targets( + share_targets: Mapping[str, float], claimant_gains_share: float +) -> dict[str, dict[str, float]]: + """Table 7's composition split by claimant status (microcosm#1063). + + Claimants take Table 7's shares among the types the reliefs apply to, + ``c_t = s_t / S_E`` with ``S_E`` those types' summed share; non-claimants + take the shares that restore the published composition over everyone, + ``n_t = (s_t - k c_t) / (1 - k)`` with ``k`` the claimants' share of the + non-residential gains. A non-claimant share is negative exactly when ``k`` + exceeds ``S_E``, which the stage refuses before it calls this. + """ + + eligible_share = float(sum(share_targets[name] for name in CGT_BADR_ELIGIBLE_TYPES)) + if not eligible_share > 0.0: + raise ValueError("Table 7 gives the BADR-eligible types no gains share.") + if not 0.0 <= claimant_gains_share < 1.0: + raise ValueError( + f"The claimants' share of the non-residential gains must lie in " + f"[0, 1), got {claimant_gains_share}." + ) + claimants = { + name: ( + float(share_targets[name]) / eligible_share + if name in CGT_BADR_ELIGIBLE_TYPES + else 0.0 + ) + for name in CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES + } + non_claimants = { + name: (float(share_targets[name]) - claimant_gains_share * claimants[name]) + / (1.0 - claimant_gains_share) + for name in CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES + } + if min(non_claimants.values()) < -_SHARE_FIT_TOLERANCE: + raise ValueError( + "The non-claimant composition that restores Table 7 has a negative " + f"share: {non_claimants}." + ) + return { + "claimants": claimants, + "non_claimants": { + name: max(share, 0.0) for name, share in non_claimants.items() + }, + } + + +def _diffuse_types( + probabilities: np.ndarray, stakes: np.ndarray, order: np.ndarray +) -> tuple[np.ndarray, np.ndarray]: + """Assign one type per person so the realised gains by type track the fit. + + Walking the persons in ``order``, each adds its stake (weight times gain) + times its type probabilities to a running balance per type and takes the + type it may hold whose balance is largest, which is then drawn down by the + whole stake. An independent draw lets a handful of heavy rows decide the + realised composition; here every row's assignment repays what the rows + before it left owing, so the realised gains by type stay within a few of + the largest stakes of the fitted composition throughout. Returns the type + index per person and the final balances (expected less realised gains). + """ + + choice = np.zeros(stakes.shape, dtype=int) + owed = np.zeros(probabilities.shape[1]) + for index in order: + row = probabilities[index] + stake = float(stakes[index]) + owed += stake * row + chosen = int(np.argmax(np.where(row > 0.0, owed, -np.inf))) + choice[index] = chosen + owed[chosen] -= stake + return choice, owed + + +def _weighted_median(values: np.ndarray, weights: np.ndarray) -> float | None: + if values.size == 0 or not float(weights.sum()) > 0.0: + return None + order = np.argsort(values, kind="stable") + cumulative = np.cumsum(weights[order]) + position = int(np.searchsorted(cumulative, 0.5 * cumulative[-1], side="left")) + return float(values[order][min(position, values.size - 1)]) + + # --------------------------------------------------------------------------- # Stage # --------------------------------------------------------------------------- @@ -1255,23 +1338,27 @@ def assign_uk_cgt_asset_types( parameters: UKCGTPolicyParameters, badr_parameters: UKCGTBADRParameters, *, - residential_seed: int = CGT_RESIDENTIAL_FLAG_SEED, badr_seed: int = CGT_BADR_FLAG_SEED, - asset_type_seed: int = CGT_ASSET_TYPE_SEED, mass_change_reason: str = UK_CGT_ASSET_TYPE_MASS_CONSERVATION_REASON, ) -> tuple[Frame, UKCGTAssetTypeSummary]: - """Write the asset-type, residential gains and BADR qualifying gains columns.""" + """Write the asset-type and BADR qualifying gains columns. + + The residential arms come from ``cgt_residential_split`` (microcosm#1063): + a person whose ``capital_gains_residential_property`` is positive is a + residential gainer on this arm; the stage types the rest. + """ validate_uk_national_frame(frame) time_period = uk_time_period(frame) person = frame.table("person").reset_index(drop=True) if "capital_gains" not in person.columns: raise ValueError("Person table has no capital_gains column to classify.") - for column in ( - CGT_ASSET_TYPE_COLUMN, - CGT_RESIDENTIAL_GAINS_COLUMN, - CGT_BADR_GAINS_COLUMN, - ): + if CGT_RESIDENTIAL_GAINS_COLUMN not in person.columns: + raise ValueError( + f"Person table has no {CGT_RESIDENTIAL_GAINS_COLUMN} column; run " + "cgt_residential_split before the asset-type stage." + ) + for column in (CGT_ASSET_TYPE_COLUMN, CGT_BADR_GAINS_COLUMN): if column in person.columns: raise ValueError(f"Person table already carries {column}.") household = frame.table("household") @@ -1293,32 +1380,29 @@ def assign_uk_cgt_asset_types( stocks = _stock_signals(person, household) - # 1. Residential flag on the liable population. + # 1. Residential arms, as the residential split wrote them. liable_index = np.flatnonzero(liable) liable_gains = gains[liable_index] liable_weights = person_weight[liable_index] - count_target = facts.residential_taxpayers_individuals_basis - gains_target = facts.residential_gains_individuals_basis - residential_offset = CGT_STOCK_LOG_ODDS * stocks["residential"][ - liable_index - ].astype(float) - a, b, centre = solve_residential_logistic( - liable_gains, - liable_weights, - count_target=count_target, - gains_target=gains_target, - offset=residential_offset, - ) - probabilities = _logistic(np.log(liable_gains) - centre, a, b, residential_offset) - rng_flag = np.random.default_rng((residential_seed, int(time_period))) - order = np.lexsort((person_id[liable_index], liable_gains)) - flags = _weighted_systematic_flags( - probabilities, liable_weights, order, float(rng_flag.random()) - ) + residential_gains = pd.to_numeric( + person[CGT_RESIDENTIAL_GAINS_COLUMN], errors="raise" + ).to_numpy(dtype=float) + if not np.isfinite(residential_gains).all() or (residential_gains < 0.0).any(): + raise ValueError( + f"{CGT_RESIDENTIAL_GAINS_COLUMN} must be finite and non-negative." + ) + flags = residential_gains[liable_index] > 0.0 + if (residential_gains[~liable] > 0.0).any(): + raise ValueError("A residential gain sits on a person who is not liable.") + if not np.array_equal(residential_gains[liable_index][flags], liable_gains[flags]): + raise ValueError( + "A residential arm's residential gain differs from the person's net " + "gain; the split attributes the whole gain." + ) residential_rows = liable_index[flags] asset_type[residential_rows] = CGT_ASSET_TYPE_RESIDENTIAL - residential_gains = np.zeros(len(person)) - residential_gains[residential_rows] = gains[residential_rows] + count_target = facts.residential_taxpayers_individuals_basis + gains_target = facts.residential_gains_individuals_basis # 2. BADR and Investors' Relief claims on the non-residential remainder. remainder_index = liable_index[~flags] @@ -1400,33 +1484,85 @@ def assign_uk_cgt_asset_types( f"reliefs apply to ({eligible_share_target}); the restricted type " "fit cannot reach the Table 7 shares." ) - type_weights = np.full(len(CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES), np.nan) - fit_iterations = 0 - if remainder_index.size: - allowed = np.ones( - (remainder_index.size, len(CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES)), - dtype=bool, + type_names = CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES + status_targets = claimant_status_share_targets(share_targets, claimant_gains_share) + status_reports: dict[str, dict[str, object]] = {} + remainder_types = np.zeros(remainder_index.size, dtype=int) + for status in CGT_CLAIMANT_STATUSES: + in_status = ( + claimant_in_remainder if status == "claimants" else ~claimant_in_remainder ) - allowed[claimant_in_remainder] = eligible_type - allowed = allowed * stocks["type_factor"][remainder_index] - type_weights, type_probabilities, fit_iterations = fit_type_weights( - gains[remainder_index], - person_weight[remainder_index], + rows = remainder_index[in_status] + targets = status_targets[status] + report: dict[str, object] = { + "rows": int(rows.size), + "target_gains_share": dict(targets), + } + status_reports[status] = report + if rows.size == 0: + continue + allowed = stocks["type_factor"][rows].copy() + if status == "claimants": + allowed = allowed * eligible_type + fitted_weights, type_probabilities, iterations = fit_type_weights( + gains[rows], + person_weight[rows], medians=medians, - share_targets=share_targets, + share_targets=targets, allowed=allowed, ) - rng_type = np.random.default_rng((asset_type_seed, int(time_period))) - # Draws are consumed in person order so the stream is reproducible. - remainder_order = np.argsort(person_id[remainder_index], kind="stable") - uniforms = np.empty(remainder_index.size) - uniforms[remainder_order] = rng_type.random(remainder_index.size) - cumulative = np.cumsum(type_probabilities, axis=1) - choice = (uniforms[:, None] > cumulative).sum(axis=1) - choice = np.minimum(choice, len(CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES) - 1) - asset_type[remainder_index] = np.asarray( - CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES, dtype=object - )[choice] + stakes = person_weight[rows] * gains[rows] + # Largest gain first, so the rows that could move the composition most + # are placed while every later row can still repay the balance. + order = np.lexsort((person_id[rows], -gains[rows])) + choice, balances = _diffuse_types(type_probabilities, stakes, order) + remainder_types[in_status] = choice + mass = float(stakes.sum()) + expected = (stakes[:, None] * type_probabilities).sum(axis=0) + realised = np.asarray( + [float(stakes[choice == index].sum()) for index in range(len(type_names))] + ) + assigned_probability = type_probabilities[np.arange(rows.size), choice] + report.update( + { + "gains_mass": mass, + "type_weights": { + name: float(fitted_weights[index]) + for index, name in enumerate(type_names) + }, + "share_fit_iterations": int(iterations), + "expected_gains_share": { + name: float(expected[index] / mass) if mass > 0.0 else 0.0 + for index, name in enumerate(type_names) + }, + "achieved_gains_share": { + name: float(realised[index] / mass) if mass > 0.0 else 0.0 + for index, name in enumerate(type_names) + }, + "final_balance": { + name: float(balances[index]) + for index, name in enumerate(type_names) + }, + "largest_stake": float(stakes.max()), + # How far the walk departs from the fit row by row: the + # stake-weighted mean fitted probability of the type taken, + # beside what an independent draw gives in expectation. + "mean_assigned_probability": ( + float((stakes * assigned_probability).sum() / mass) + if mass > 0.0 + else 0.0 + ), + "independent_draw_mean_probability": ( + float((stakes * (type_probabilities**2).sum(axis=1)).sum() / mass) + if mass > 0.0 + else 0.0 + ), + } + ) + if remainder_index.size: + asset_type[remainder_index] = np.asarray(type_names, dtype=object)[ + remainder_types + ] if (asset_type[liable] == CGT_ASSET_TYPE_NONE).any() or ( asset_type[liable] == CGT_ASSET_TYPE_SUB_AEA @@ -1435,39 +1571,58 @@ def assign_uk_cgt_asset_types( if set(asset_type) - set(CGT_ASSET_TYPE_DOMAIN): raise ValueError("Asset-type draw produced a value outside the domain.") - # Reporting. - expected_count = float((liable_weights * probabilities).sum()) - expected_gains = float((liable_weights * liable_gains * probabilities).sum()) - bernoulli = probabilities * (1.0 - probabilities) - count_sigma = float(np.sqrt((liable_weights**2 * bernoulli).sum())) - gains_sigma = float( - np.sqrt(((liable_weights * liable_gains) ** 2 * bernoulli).sum()) - ) + # Reporting: the residential identities the split left, by gain band. achieved_count = float(person_weight[residential_rows].sum()) achieved_gains = float( residential_gains[residential_rows].dot(person_weight[residential_rows]) ) + bounds = np.asarray(HMRC_CGT_GAIN_BAND_LOWER_BOUNDS, dtype=float) + liable_band = np.searchsorted(bounds, liable_gains, side="right") - 1 + residential_bands = [] + for position, lower in enumerate(bounds): + in_band = liable_band == position + on_arm = in_band & flags + residential_bands.append( + { + "gain_lower_bound": float(lower), + "gain_upper_bound": ( + float(bounds[position + 1]) if position + 1 < bounds.size else None + ), + "liable_rows": int(in_band.sum()), + "achieved_rows": int(on_arm.sum()), + "achieved_count": float(liable_weights[on_arm].sum()), + "achieved_gains": float( + (liable_weights[on_arm] * liable_gains[on_arm]).sum() + ), + } + ) residential = { + "source_stage": "cgt_residential_split", "count_target_individuals_basis": count_target, "gains_target_individuals_basis": gains_target, - "expected_count": expected_count, - "expected_gains": expected_gains, "achieved_count": achieved_count, "achieved_gains": achieved_gains, "achieved_rows": int(residential_rows.size), - "max_liable_weight": float(liable_weights.max()), - "count_bernoulli_sigma": count_sigma, - "gains_bernoulli_sigma": gains_sigma, - "logistic_intercept": float(a), - "logistic_slope": float(b), - "log_gain_centre": float(centre), + "count_relative_error": ( + abs(achieved_count - count_target) / count_target + if count_target > 0.0 + else None + ), + "gains_relative_error": ( + abs(achieved_gains - gains_target) / gains_target + if gains_target > 0.0 + else None + ), + "max_liable_weight": float(liable_weights.max()) + if liable_weights.size + else 0.0, + "bands": residential_bands, "liable_taxpayer_mass": float(liable_weights.sum()), "liable_gains_mass": float((liable_weights * liable_gains).sum()), "table8a_taxpayers_total": facts.table8a_taxpayers_total, "table8a_gains_total": facts.table8a_gains_total, "table8b_individuals_taxpayer_share": facts.individuals_share("taxpayers"), "table8b_individuals_gains_share": facts.individuals_share("gains"), - "stock_log_odds": CGT_STOCK_LOG_ODDS, "stock_signal": CGT_RESIDENTIAL_STOCK_SIGNAL, "stock_share_flagged": _weighted_share( person_weight, @@ -1490,13 +1645,15 @@ def assign_uk_cgt_asset_types( asset_type_report = { "log_sigma": CGT_ASSET_TYPE_LOG_SIGMA, "median_anchor_gbp": medians, - "type_weights": { - asset_type_name: float(type_weights[index]) - for index, asset_type_name in enumerate( - CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES + "composition_by_claimant_status": status_reports, + "realization": "error diffusion in descending (gain, person_id) order", + "achieved_median_gain_gbp": { + asset_type_name: _weighted_median( + gains[asset_type == asset_type_name], + person_weight[asset_type == asset_type_name], ) + for asset_type_name in CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES }, - "share_fit_iterations": int(fit_iterations), "target_gains_share": dict(share_targets), "achieved_gains_share": { asset_type_name: (value / remainder_mass if remainder_mass > 0 else 0.0) @@ -1618,16 +1775,11 @@ def assign_uk_cgt_asset_types( "source_commit": facts.source_commit, "annual_exempt_amount": parameters.annual_exempt_amount, }, - seeds={ - "residential_flag": residential_seed, - "badr_flag": badr_seed, - "asset_type": asset_type_seed, - }, + seeds={"badr_flag": badr_seed}, ) new_person = person.copy() new_person[CGT_ASSET_TYPE_COLUMN] = asset_type - new_person[CGT_RESIDENTIAL_GAINS_COLUMN] = residential_gains new_person[CGT_BADR_GAINS_COLUMN] = qualifying weights = frame.weights_for("household") household_mass = float(weights.total) @@ -1698,11 +1850,7 @@ def __call__(self, frame: Frame) -> Frame: @staticmethod def output_columns() -> tuple[str, ...]: - return ( - CGT_ASSET_TYPE_COLUMN, - CGT_RESIDENTIAL_GAINS_COLUMN, - CGT_BADR_GAINS_COLUMN, - ) + return (CGT_ASSET_TYPE_COLUMN, CGT_BADR_GAINS_COLUMN) def checkpoint_metadata(self) -> dict[str, object]: if self.last_result is None: @@ -1726,55 +1874,11 @@ def cgt_asset_type_operation_parameters() -> dict[str, dict[str, Any]]: "runtime_sha256_required": True, "fail_on_mismatch": True, }, - "assign_residential_property_flag": { + "assign_badr_qualifying_gains": { "population": ( - "persons with net capital gains above the annual exempt amount " - "(the national taxpayer proxy)" - ), - "model": ( - "logistic probability in centred log gains with a stock shift, p = " - "1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean " - "log gain of the population, s the stock log-odds and h one where " - "the stock signal holds, zero otherwise" - ), - "stock_log_odds": CGT_STOCK_LOG_ODDS, - "stock_signal": CGT_RESIDENTIAL_STOCK_SIGNAL, - "parameter_solver": ( - "nested bisection; the intercept matches the expected weighted " - "count at each trial slope, the slope matches the expected weighted " - "gains" - ), - "count_target": ( - "Table 8a 2024-25 total taxpayers reporting residential property " - "disposals times the Table 8b 2024-25 individuals/all taxpayer share " - "on the UK Property service" - ), - "gains_target": ( - "Table 8a 2024-25 total residential property gains times the Table " - "8b 2024-25 individuals/all gains share on the UK Property service" - ), - "basis_assumption": ( - "trusts hold the same share of the Self Assessment component as of " - "the UK Property service, and a flagged person's whole net gain is " - "attributed to residential property" - ), - "realization": ( - "weighted systematic sampling in ascending gain order with one " - "seeded offset: a person is flagged when the expected weight owed " - "so far reaches their own weight, so the flagged weight tracks the " - "expected weight within one person's weight along the gains axis " - "and the realised count and gains sit close to their expectations; " - "the Bernoulli sigma of each is reported as the envelope" - ), - "seed": CGT_RESIDENTIAL_FLAG_SEED, - "seed_mixing": "seed combined with the build period", - "output_column": CGT_RESIDENTIAL_GAINS_COLUMN, - "output_semantics": ( - "the person's capital_gains where flagged residential, 0 otherwise" + "liable gainers whose arm is not residential " + "(capital_gains_residential_property == 0, cgt_residential_split)" ), - }, - "assign_badr_qualifying_gains": { - "population": "liable gainers not flagged residential", "bands": ( "HMRC Table 4.1 2024-25 individuals claiming Business Asset " "Disposal Relief or Investors' Relief by band of qualifying gain " @@ -1860,22 +1964,38 @@ def cgt_asset_type_operation_parameters() -> dict[str, dict[str, Any]]: "Table 7 2023-24 gains by asset type excluding residential land " "and buildings, as shares" ), + "claimant_share_targets": ( + "the share targets among the claimant categories, each over their sum" + ), + "non_claimant_share_targets": ( + "(share target - k x claimant share target) / (1 - k) with k the " + "claimants' share of the non-residential gains, the composition " + "that restores the share targets over claimants and non-claimants " + "together" + ), + "basis_assumption": ( + "Table 7 does not split gains by relief, so claimants are given " + "its composition among the claimant categories" + ), "weight_fitting": ( - "multiplicative updates of the type weights, with claimants " - "restricted to the claimant categories, until the " - "household-weighted expected gains shares by type match the share " - "targets; the fit refuses if it has not converged within the " - "iteration limit, and refuses up front when claimants hold more " - "of the gains than the claimant categories' share" + "one fit per claimant status: multiplicative updates of the type " + "weights until the household-weighted expected gains shares by " + "type match that status's share targets; a fit refuses if it has " + "not converged within the iteration limit, and the stage refuses " + "up front when claimants hold more of the gains than the claimant " + "categories' share" ), "weight_fitting_iterations": _SHARE_FIT_ITERATIONS, "weight_fitting_tolerance": _SHARE_FIT_TOLERANCE, "realization": ( - "one seeded uniform per person against the cumulative type " - "probabilities, consumed in person_id order" + "error diffusion per claimant status in descending (gain, " + "person_id) order: each person adds weight x gain x type " + "probability to a running balance per type and takes the type it " + "may hold with the largest balance, which is drawn down by the " + "whole weight x gain, so the realised gains by type track the " + "fitted composition instead of following the heaviest rows; " + "deterministic, no seed" ), - "seed": CGT_ASSET_TYPE_SEED, - "seed_mixing": "seed combined with the build period", "output_column": CGT_ASSET_TYPE_COLUMN, "value_domain": list(CGT_ASSET_TYPE_DOMAIN), "none_semantics": "no positive net gains", diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_residential_split.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_residential_split.py new file mode 100644 index 000000000..2a4db3f39 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_residential_split.py @@ -0,0 +1,796 @@ +"""UK capital-gains residential split: the residential-property flag carried as +weight, not drawn (microcosm#1063). + +HMRC Table 8a counts the individuals who reported a residential property +disposal and their gains; the asset-type stage solved a logistic in log gains +to both totals and then realised the flag by a weighted systematic walk, one +whole flag per row. The walk's count error is bounded by one row's weight, +but its gains error by one row's stake (weight times gain), and at the top of +the gains distribution one stake is several times the band's whole expected +residential gains, so the realised gains sat either short of their expectation +by the lottery expectation of the giant stakes or over it by one of them. + +This stage keeps the logistic and drops the lottery. Every household carrying +liable gainers is split into arms, identical in every cell except +``capital_gains_residential_property``: with one liable gainer of probability +``p`` the household's mass is divided into a non-residential arm at +``(1 - p) w`` (the household's own ids) and a residential arm at ``p w`` +whose gainer's whole net gain is residential; with ``k`` liable gainers the +household becomes ``2 ** k`` arms at product weights, one per subset of +residential gainers (independent flags). At design weights the residential +count and gains are then the mass identities ``sum(p w)`` and +``sum(p w g)``, which the solve put on Table 8a, in every gain band and with +no draw, no offset and no seed; the calibration, which binds the national +residential taxpayers and gains (``hmrc.cgt.residential_property_taxpayers`` +and ``_gains``; the by-band rows stay fenced), decides between the arms +afterwards exactly as it decides between the incidence clone's halves. Every household's mass and composition and the +total household mass are conserved; arm ``j`` takes every entity id plus +``j`` times the frame's id multiplier, the lineage rule the executor and the +geography identity kernel read for the support split and the clone. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from typing import Any + +import numpy as np +import pandas as pd + +from microcosm.build.source_manifest import SourceStageSpec +from microcosm.build.uk_runtime.cgt_asset_type import ( + CGT_RESIDENTIAL_GAINS_COLUMN, + CGT_RESIDENTIAL_STOCK_SIGNAL, + CGT_STOCK_LOG_ODDS, + HMRC_CGT_ASSET_TYPE_RESOURCE, + HMRCCGTAssetTypeFacts, + _logistic, + _stock_signals, + _weighted_share, + load_hmrc_cgt_asset_type_facts, + solve_residential_logistic, +) +from microcosm.build.uk_runtime.cgt_imputation import ( + UKCGTPolicyParameters, + uk_cgt_policy_parameters, +) +from microcosm.build.uk_runtime.cgt_structure import _assert_closed_world_operations +from microcosm.build.uk_runtime.hmrc_capital_gains import ( + HMRC_CGT_GAIN_BAND_LOWER_BOUNDS, +) +from microcosm.build.uk_runtime.national_frame import ( + uk_national_frame, + uk_time_period, + validate_uk_national_frame, +) +from microcosm.build.uk_runtime.rowwise_geography import id_multiplier_for_values +from microcosm.build.uk_runtime.spi_support import ( + _exact_total_correction, + _importance_weights_with_exact_total, +) +from microcosm.frame import Frame, MassChangeRecord, WeightKind, Weights + +CGT_RESIDENTIAL_SPLIT_STAGE_NAME = "cgt_residential_split" +CGT_RESIDENTIAL_SPLIT_OPERATION_KIND = "split_liable_gainers_by_residential_probability" +#: True on every arm the split created; the household's own ids stay on the +#: arm with no residential gainer. +HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE = "household_is_cgt_residential_clone" +#: Arm index ``j`` (0 on the household's own arm and on unsplit households): +#: bit ``m`` of ``j`` says whether the household's ``m``-th liable gainer, in +#: person_id order, is residential on that arm. +CGT_RESIDENTIAL_CLONE_INDEX_COLUMN = "cgt_residential_clone_index" +#: The solved residential probability of each liable gainer (0 elsewhere), +#: carried on every arm so the arm weights can be reconstructed from ids. +CGT_RESIDENTIAL_PROBABILITY_COLUMN = "cgt_residential_probability" +#: A household with more liable gainers than this would need more than eight +#: arms; none exists on the FRS spine, so the stage refuses rather than grows. +CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS = 3 +#: Concentration diagnostic: the share of the residential gains mass on the +#: largest residential arm stakes (arm weight times gain). +CGT_RESIDENTIAL_TOP_ARM_ROWS = 10 +CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON = ( + "Capital-gains residential split divides every household carrying liable " + "gainers into arms weighted by the solved residential probabilities, one " + "arm per subset of residential gainers; every household's mass and the " + "total household mass are conserved." +) +_PERSON_ID_COLUMNS = ("person_id", "person_household_id", "person_benunit_id") +_POLICY_PARAMETER_NAMES = ("gov.hmrc.cgt.annual_exempt_amount",) +_TABLE8_RECORD_SETS = ( + "hmrc.cgt_table8_2026.table8a.ty2024", + "hmrc.cgt_table8_2026.table8b.ty2024", +) + + +@dataclass(frozen=True) +class UKCGTResidentialSplitResult: + """Split frame and the executed-effect receipt of the residential split.""" + + frame: Frame + identities: Mapping[str, float] + bands: tuple[Mapping[str, object], ...] + logistic: Mapping[str, object] + households_by_liable_gainers: Mapping[int, int] + households_before: int + households_after: int + arms_created: int + arm_weights: Mapping[str, object] + concentration: Mapping[str, object] + stock: Mapping[str, object] + facts: Mapping[str, object] + old_total: float + new_total: float + arm_weights_exact: bool + + def evidence(self) -> dict[str, object]: + return { + "stage": CGT_RESIDENTIAL_SPLIT_STAGE_NAME, + "mass": { + "old_total": float(self.old_total), + "new_total": float(self.new_total), + }, + "households_before": int(self.households_before), + "households_after": int(self.households_after), + "arms_created": int(self.arms_created), + "households_by_liable_gainers": { + str(k): int(v) + for k, v in sorted(self.households_by_liable_gainers.items()) + }, + "maximum_liable_gainers_per_household": CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS, + "arm_weights_exact": bool(self.arm_weights_exact), + "identities": dict(self.identities), + "bands": [dict(row) for row in self.bands], + "logistic": dict(self.logistic), + "arm_weights": dict(self.arm_weights), + "concentration": dict(self.concentration), + "stock": dict(self.stock), + "facts": dict(self.facts), + } + + +@dataclass(frozen=True) +class UKCGTResidentialSplitStageTransform: + """Whole-stage transform for the residential split. + + ``facts`` default to the vendored Table 8 rows and ``parameters`` to the + policyengine-uk annual exempt amount at the frame's build period, read at + apply time so the base package never imports the engine at import time. + """ + + stage: SourceStageSpec + facts: HMRCCGTAssetTypeFacts | None = None + parameters: UKCGTPolicyParameters | None = None + last_result: UKCGTResidentialSplitResult | None = field(default=None, init=False) + + def __post_init__(self) -> None: + _assert_cgt_residential_split_stage_parameters(self.stage) + + def __call__(self, frame: Frame) -> Frame: + _assert_cgt_residential_split_stage_parameters(self.stage) + facts = self.facts + if facts is None: + facts = load_hmrc_cgt_asset_type_facts() + parameters = self.parameters + if parameters is None: + parameters = uk_cgt_policy_parameters(uk_time_period(frame)) + result = split_cgt_residential_households( + frame, facts=facts, parameters=parameters + ) + object.__setattr__(self, "last_result", result) + return result.frame + + @staticmethod + def output_columns() -> tuple[str, ...]: + return ( + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + CGT_RESIDENTIAL_CLONE_INDEX_COLUMN, + CGT_RESIDENTIAL_PROBABILITY_COLUMN, + CGT_RESIDENTIAL_GAINS_COLUMN, + ) + + def checkpoint_metadata(self) -> dict[str, object]: + if self.last_result is None: + raise RuntimeError("checkpoint metadata requires a completed stage run.") + return {"evidence": self.last_result.evidence()} + + +def residential_arm_factors(probabilities: np.ndarray) -> np.ndarray: + """Weight factor of every arm of a household with these gainers. + + Row ``j`` of the result is the factor of arm ``j``: the product over the + gainers of ``p`` where bit ``m`` of ``j`` is set and ``1 - p`` where it is + not. Row 0 is the household's own arm; the rows sum to one. + """ + + p = np.asarray(probabilities, dtype=float) + if p.ndim != 1 or p.size == 0: + raise ValueError("A household needs at least one liable gainer to split.") + if not np.isfinite(p).all() or (p < 0.0).any() or (p > 1.0).any(): + raise ValueError("Residential probabilities must lie in [0, 1].") + arms = np.arange(2**p.size) + bits = ((arms[:, None] >> np.arange(p.size)[None, :]) & 1).astype(bool) + return np.where(bits, p[None, :], 1.0 - p[None, :]).prod(axis=1) + + +def split_cgt_residential_households( + frame: Frame, + *, + facts: HMRCCGTAssetTypeFacts, + parameters: UKCGTPolicyParameters, +) -> UKCGTResidentialSplitResult: + """Split every household carrying liable gainers into residential arms. + + Deterministic and mass-conserving: no draw, no seed, no salt. The logistic + is solved exactly as the asset-type stage solved it (intercept to Table + 8a's count on the individuals basis, slope to its gains, with the stock + shift); a household with ``k`` liable gainers becomes ``2 ** k`` arms at + product weights, the arm with no residential gainer keeping the + household's ids and every other arm ``j`` taking every entity id plus + ``j`` times the frame's id multiplier. The exact-total correction lands on + the heaviest unsplit incumbent so every arm weight stays bitwise its + product. Refuses a frame the stage has already split, one the asset-type + stage has already classified, or a household with more liable gainers + than the declared maximum. + """ + + validate_uk_national_frame(frame) + person = frame.table("person").copy().reset_index(drop=True) + benunit = frame.table("benunit").copy().reset_index(drop=True) + household = frame.table("household").copy().reset_index(drop=True) + if "capital_gains" not in person.columns: + raise ValueError("Person table has no capital_gains column to split on.") + for column, table in ( + (HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, household), + (CGT_RESIDENTIAL_CLONE_INDEX_COLUMN, household), + (CGT_RESIDENTIAL_PROBABILITY_COLUMN, person), + (CGT_RESIDENTIAL_GAINS_COLUMN, person), + ): + if column in table.columns: + raise ValueError( + f"The CGT residential split already ran: the frame carries {column!r}." + ) + weights = np.asarray(frame.weights_for("household").values, dtype=float) + old_total = frame.weights_for("household").total + household_ids = household["household_id"].to_numpy(dtype="int64") + weight_by_household = pd.Series(weights, index=household_ids) + person_weight = ( + person["person_household_id"].map(weight_by_household).to_numpy(dtype=float) + ) + if not np.isfinite(person_weight).all(): + raise ValueError("Every person must map to a weighted household.") + gains = pd.to_numeric(person["capital_gains"], errors="raise").to_numpy(dtype=float) + if not np.isfinite(gains).all(): + raise ValueError("capital_gains must be finite for every person.") + liable = gains > parameters.annual_exempt_amount + # The solve walks the liable gainers in person_id order whatever the row + # order of the frame: its bisection sums are pairwise float sums, which + # move in their last bits under a permutation, and every arm weight is a + # product of the solved probabilities (the E8 identity receipt found + # 1e-13 between two orders of the 2 October spine's persons). On a frame + # sorted by person_id, as the spine's tables are, this is the identity. + liable_index = np.flatnonzero(liable) + liable_index = liable_index[ + np.argsort( + person["person_id"].to_numpy(dtype="int64")[liable_index], kind="stable" + ) + ] + liable_gains = gains[liable_index] + liable_weights = person_weight[liable_index] + stocks = _stock_signals(person, household) + offset = CGT_STOCK_LOG_ODDS * stocks["residential"][liable_index].astype(float) + count_target = float(facts.residential_taxpayers_individuals_basis) + gains_target = float(facts.residential_gains_individuals_basis) + a, b, centre = solve_residential_logistic( + liable_gains, + liable_weights, + count_target=count_target, + gains_target=gains_target, + offset=offset, + ) + probabilities = _logistic(np.log(liable_gains) - centre, a, b, offset) + probability = np.zeros(len(person)) + probability[liable_index] = probabilities + person[CGT_RESIDENTIAL_PROBABILITY_COLUMN] = probability + person[CGT_RESIDENTIAL_GAINS_COLUMN] = 0.0 + household[HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE] = False + household[CGT_RESIDENTIAL_CLONE_INDEX_COLUMN] = np.zeros( + len(household), dtype="int64" + ) + + # Liable gainers by household, in person_id order within the household, + # so bit m of an arm index always names the same gainer. + liable_table = pd.DataFrame( + { + "position": liable_index, + "household_id": person["person_household_id"].to_numpy(dtype="int64")[ + liable_index + ], + "person_id": person["person_id"].to_numpy(dtype="int64")[liable_index], + "probability": probabilities, + } + ).sort_values(["household_id", "person_id"], kind="stable") + liable_table["rank"] = liable_table.groupby("household_id").cumcount() + gainers_per_household = liable_table.groupby("household_id").size() + too_many = gainers_per_household[ + gainers_per_household > CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS + ] + if len(too_many): + raise ValueError( + f"{len(too_many)} household(s) carry more than " + f"{CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS} liable gainers; the " + "residential split refuses to enumerate their arms." + ) + multiplier = id_multiplier_for_values( + person["person_id"], + person["person_household_id"], + person["person_benunit_id"], + benunit["benunit_id"], + household["household_id"], + ) + household_position = pd.Series(np.arange(len(household)), index=household_ids) + incumbent_weights = weights.copy() + arm_person: list[pd.DataFrame] = [] + arm_benunit: list[pd.DataFrame] = [] + arm_household: list[pd.DataFrame] = [] + arm_weights: list[np.ndarray] = [] + arm_exact = True + households_by_k: dict[int, int] = {} + for k, family_ids in gainers_per_household.groupby(gainers_per_household): + k = int(k) + family_ids = family_ids.index.to_numpy(dtype="int64") + households_by_k[k] = int(family_ids.size) + rows = liable_table.loc[liable_table["household_id"].isin(family_ids)] + # Probability matrix: one row per family, one column per gainer rank. + matrix = ( + rows.pivot(index="household_id", columns="rank", values="probability") + .reindex(family_ids) + .to_numpy(dtype=float) + ) + factors = np.stack([residential_arm_factors(row) for row in matrix]) + if not np.allclose(factors.sum(axis=1), 1.0, rtol=0.0, atol=1e-12): + arm_exact = False + positions = household_position.loc[family_ids].to_numpy() + incumbent_weights[positions] = weights[positions] * factors[:, 0] + family_set = set(family_ids.tolist()) + members = person["person_household_id"].isin(family_set).to_numpy() + family_person = person.loc[members] + family_benunit = benunit.loc[ + benunit["benunit_id"].isin(set(family_person["person_benunit_id"])) + ] + family_household = household.loc[positions] + rank_by_position = pd.Series( + rows["rank"].to_numpy(), index=rows["position"].to_numpy() + ) + member_positions = np.flatnonzero(members) + member_rank = rank_by_position.reindex(member_positions).to_numpy() + for j in range(1, 2**k): + id_offset = j * multiplier + copy_person = family_person.copy() + for column in _PERSON_ID_COLUMNS: + copy_person[column] = copy_person[column].astype("int64") + id_offset + residential = np.zeros(len(copy_person), dtype=bool) + for m in range(k): + if (j >> m) & 1: + residential |= member_rank == m + copy_person[CGT_RESIDENTIAL_GAINS_COLUMN] = np.where( + residential, gains[member_positions], 0.0 + ) + copy_benunit = family_benunit.copy() + copy_benunit["benunit_id"] = ( + copy_benunit["benunit_id"].astype("int64") + id_offset + ) + copy_household = family_household.copy() + copy_household["household_id"] = ( + copy_household["household_id"].astype("int64") + id_offset + ) + copy_household[HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE] = True + copy_household[CGT_RESIDENTIAL_CLONE_INDEX_COLUMN] = np.full( + len(copy_household), j, dtype="int64" + ) + arm_person.append(copy_person) + arm_benunit.append(copy_benunit) + arm_household.append(copy_household) + arm_weights.append(weights[positions] * factors[:, j]) + # Every arm id exceeds every incoming id (the multiplier is larger than + # any id), so the arms follow the incumbents; within the arms the rows are + # ordered by id so the group tables stay sorted whatever the household + # order the arms were built in. + arms_person = pd.concat(arm_person, ignore_index=True) if arm_person else None + arms_benunit = pd.concat(arm_benunit, ignore_index=True) if arm_benunit else None + arms_household = ( + pd.concat(arm_household, ignore_index=True) if arm_household else None + ) + arms_weights = ( + np.concatenate(arm_weights) if arm_weights else np.zeros(0, dtype=float) + ) + if arms_household is not None: + household_order = np.argsort( + arms_household["household_id"].to_numpy(dtype="int64"), kind="stable" + ) + arms_household = arms_household.iloc[household_order].reset_index(drop=True) + arms_weights = arms_weights[household_order] + arms_benunit = arms_benunit.iloc[ + np.argsort( + arms_benunit["benunit_id"].to_numpy(dtype="int64"), kind="stable" + ) + ].reset_index(drop=True) + arms_person = arms_person.iloc[ + np.argsort(arms_person["person_id"].to_numpy(dtype="int64"), kind="stable") + ].reset_index(drop=True) + final_person = pd.concat( + [person, *([arms_person] if arms_person is not None else [])], + ignore_index=True, + ) + final_benunit = pd.concat( + [benunit, *([arms_benunit] if arms_benunit is not None else [])], + ignore_index=True, + ) + final_household = pd.concat( + [household, *([arms_household] if arms_household is not None else [])], + ignore_index=True, + ) + final_weights = np.concatenate([incumbent_weights, arms_weights]) + split_households = np.zeros(len(household), dtype=bool) + split_households[household_position.loc[gainers_per_household.index].to_numpy()] = ( + True + ) + exact = _exact_residential_weights( + final_weights, + target=old_total, + unsplit=np.flatnonzero(~split_households & (incumbent_weights > 0.0)), + household_ids=household_ids, + ) + receipt = MassChangeRecord( + entity="household", + old_total=old_total, + new_total=exact.total, + declared_factor=1.0, + reason=CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON, + ) + result = uk_national_frame( + person=final_person, + benunit=final_benunit, + household=final_household, + time_period=uk_time_period(frame), + weight_kind=WeightKind.IMPORTANCE, + household_weights=exact.values, + mass_log=(*frame.mass_log, receipt), + ) + validate_uk_national_frame(result) + + # Receipt: the identities at design weights, by gain band and in all. + final_household_weight = pd.Series( + exact.values, index=final_household["household_id"].to_numpy(dtype="int64") + ) + final_person_weight = ( + final_person["person_household_id"].map(final_household_weight).to_numpy(float) + ) + final_gains = final_person["capital_gains"].to_numpy(dtype=float) + final_residential = final_person[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy(float) + is_residential = final_residential > 0.0 + expected_count = float((liable_weights * probabilities).sum()) + expected_gains = float((liable_weights * liable_gains * probabilities).sum()) + achieved_count = float(final_person_weight[is_residential].sum()) + achieved_gains = float( + (final_person_weight * final_residential)[is_residential].sum() + ) + bounds = np.asarray(HMRC_CGT_GAIN_BAND_LOWER_BOUNDS, dtype=float) + liable_band = np.searchsorted(bounds, liable_gains, side="right") - 1 + final_band = np.searchsorted(bounds, final_gains, side="right") - 1 + liable_stakes = liable_weights * liable_gains + residential_stakes = (final_person_weight * final_residential)[is_residential] + band_rows: list[dict[str, object]] = [] + for position, lower in enumerate(bounds): + in_band = liable_band == position + out_band = is_residential & (final_band == position) + arms_in_band = final_person_weight[out_band] + band_rows.append( + { + "gain_lower_bound": float(lower), + "gain_upper_bound": ( + float(bounds[position + 1]) if position + 1 < bounds.size else None + ), + "liable_rows": int(in_band.sum()), + "liable_mass": float(liable_weights[in_band].sum()), + "liable_gains": float(liable_stakes[in_band].sum()), + "heaviest_liable_weight": ( + float(liable_weights[in_band].max()) if in_band.any() else 0.0 + ), + "largest_liable_stake": ( + float(liable_stakes[in_band].max()) if in_band.any() else 0.0 + ), + "expected_count": float( + (liable_weights[in_band] * probabilities[in_band]).sum() + ), + "expected_gains": float( + (liable_stakes[in_band] * probabilities[in_band]).sum() + ), + "achieved_count": float(arms_in_band.sum()), + "achieved_gains": float( + (final_person_weight * final_residential)[out_band].sum() + ), + "residential_rows": int(out_band.sum()), + "largest_residential_arm_weight": ( + float(arms_in_band.max()) if out_band.any() else 0.0 + ), + } + ) + sorted_stakes = np.sort(residential_stakes)[::-1] + top_stakes = float(sorted_stakes[:CGT_RESIDENTIAL_TOP_ARM_ROWS].sum()) + residential_arm_weights = exact.values[len(household) :] + identities = { + "count_target_individuals_basis": count_target, + "gains_target_individuals_basis": gains_target, + "expected_count": expected_count, + "expected_gains": expected_gains, + "achieved_count": achieved_count, + "achieved_gains": achieved_gains, + "count_solve_relative_error": abs(expected_count - count_target) / count_target, + "gains_solve_relative_error": abs(expected_gains - gains_target) / gains_target, + "count_identity_relative_error": ( + abs(achieved_count - expected_count) / expected_count + ), + "gains_identity_relative_error": ( + abs(achieved_gains - expected_gains) / expected_gains + ), + "liable_taxpayer_mass": float(liable_weights.sum()), + "liable_gains_mass": float(liable_stakes.sum()), + } + return UKCGTResidentialSplitResult( + frame=result, + identities=identities, + bands=tuple(band_rows), + logistic={ + "intercept": float(a), + "slope": float(b), + "log_gain_centre": float(centre), + "stock_log_odds": CGT_STOCK_LOG_ODDS, + "stock_signal": CGT_RESIDENTIAL_STOCK_SIGNAL, + }, + households_by_liable_gainers=households_by_k, + households_before=int(len(household)), + households_after=int(len(final_household)), + arms_created=int(len(final_household) - len(household)), + arm_weights={ + "residential_arms": int(residential_arm_weights.size), + "minimum": ( + float(residential_arm_weights.min()) + if residential_arm_weights.size + else 0.0 + ), + "maximum": ( + float(residential_arm_weights.max()) + if residential_arm_weights.size + else 0.0 + ), + "below_one_household": int((residential_arm_weights < 1.0).sum()), + "below_one_tenth": int((residential_arm_weights < 0.1).sum()), + }, + concentration={ + "top_arm_rows": CGT_RESIDENTIAL_TOP_ARM_ROWS, + "top_arms_share_of_achieved_gains": ( + top_stakes / achieved_gains if achieved_gains > 0.0 else 0.0 + ), + "largest_residential_arm_stake": ( + float(sorted_stakes[0]) if sorted_stakes.size else 0.0 + ), + "largest_liable_stake": ( + float(liable_stakes.max()) if liable_stakes.size else 0.0 + ), + "largest_liable_stake_share_of_gains_target": ( + float(liable_stakes.max()) / gains_target + if liable_stakes.size and gains_target > 0.0 + else 0.0 + ), + "max_liable_weight": ( + float(liable_weights.max()) if liable_weights.size else 0.0 + ), + }, + stock={ + "share_liable": _weighted_share( + person_weight, liable, stocks["residential"] + ), + "expected_share_residential": ( + float( + (liable_weights * probabilities)[ + stocks["residential"][liable_index] + ].sum() + ) + / expected_count + if expected_count > 0.0 + else None + ), + }, + facts={ + "table8a_taxpayers_total": float(facts.table8a_taxpayers_total), + "table8a_gains_total": float(facts.table8a_gains_total), + "table8b_individuals_taxpayer_share": float( + facts.individuals_share("taxpayers") + ), + "table8b_individuals_gains_share": float(facts.individuals_share("gains")), + }, + old_total=float(old_total), + new_total=float(exact.total), + arm_weights_exact=arm_exact, + ) + + +def _exact_residential_weights( + values: np.ndarray, + *, + target: float, + unsplit: np.ndarray, + household_ids: np.ndarray, +) -> Weights: + """Importance weights summing bitwise to ``target``. + + The correction lands on the heaviest unsplit incumbent (the lowest + household id among equal weights, so the choice does not depend on the + row order) so every arm weight stays exactly its product; a frame with no + unsplit positive incumbent falls back to the shared multi-candidate + correction. + """ + + if unsplit.size: + order = np.lexsort( + (np.asarray(household_ids)[unsplit], -np.asarray(values)[unsplit]) + ) + heaviest = int(unsplit[order[0]]) + corrected = _exact_total_correction( + values, target=target, correction_index=heaviest + ) + if corrected is not None: + return Weights(corrected, WeightKind.IMPORTANCE) + return _importance_weights_with_exact_total(values, target) + + +def cgt_residential_split_operation_parameters() -> tuple[ + tuple[str, dict[str, Any]], ... +]: + """The reviewed operations the manifests must carry verbatim.""" + + return ( + ( + "verify_vendored_fact_resource", + { + "artifact_role": "cgt_asset_type_facts", + "resource": HMRC_CGT_ASSET_TYPE_RESOURCE, + "feed_pin": "chronicle_feed.json", + "record_sets": list(_TABLE8_RECORD_SETS), + "source_vintage": "2024-25", + "mapped_build_period": 2024, + "period_mapping": "published_tax_year_equals_build_period", + "require_before_source_read": True, + "runtime_sha256_required": True, + "fail_on_mismatch": True, + }, + ), + ( + CGT_RESIDENTIAL_SPLIT_OPERATION_KIND, + { + "population": ( + "persons with net capital gains above the annual exempt amount " + "(the national taxpayer proxy)" + ), + "model": ( + "logistic probability in centred log gains with a stock shift, " + "p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted " + "mean log gain of the population, s the stock log-odds and h one " + "where the stock signal holds, zero otherwise" + ), + "stock_log_odds": CGT_STOCK_LOG_ODDS, + "stock_signal": CGT_RESIDENTIAL_STOCK_SIGNAL, + "parameter_solver": ( + "nested bisection; the intercept matches the expected weighted " + "count at each trial slope, the slope matches the expected " + "weighted gains" + ), + "count_target": ( + "Table 8a 2024-25 total taxpayers reporting residential property " + "disposals times the Table 8b 2024-25 individuals/all taxpayer " + "share on the UK Property service" + ), + "gains_target": ( + "Table 8a 2024-25 total residential property gains times the " + "Table 8b 2024-25 individuals/all gains share on the UK Property " + "service" + ), + "basis_assumption": ( + "trusts hold the same share of the Self Assessment component as " + "of the UK Property service, a residential gainer's whole net " + "gain is attributed to residential property, and the gainers of " + "one household are residential independently of each other" + ), + "realization": ( + "probability carried as weight, no draw: a household carrying k " + "liable gainers becomes 2^k arms, one per subset of residential " + "gainers, at the household's weight times the product of p over " + "the residential gainers and 1 - p over the others; the arm with " + "no residential gainer keeps the household's ids; at design " + "weights the residential count and gains equal the solved " + "expectations sum(p w) and sum(p w g) in every gain band" + ), + "maximum_liable_gainers_per_household": ( + CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS + ), + "gainer_order": "person_id ascending within the household", + "id_remapping": ( + "id_multiplier_for_values; arm j takes every entity id + j x " + "multiplier, bit m of j marking the household's m-th liable " + "gainer residential" + ), + "flag_column": HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + "index_column": CGT_RESIDENTIAL_CLONE_INDEX_COLUMN, + "probability_column": CGT_RESIDENTIAL_PROBABILITY_COLUMN, + "output_column": CGT_RESIDENTIAL_GAINS_COLUMN, + "output_semantics": ( + "the person's capital_gains on an arm where the person is " + "residential, 0 otherwise" + ), + "values": "every other column copied unchanged; no draw, no seed, no salt", + "weight_kind_out": WeightKind.IMPORTANCE.value, + "conservation": "exact_total", + "declared_factor": 1.0, + "reason": CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON, + }, + ), + ) + + +def _assert_cgt_residential_split_stage_parameters(stage: SourceStageSpec) -> None: + """Bind every stage manifest parameter to reviewed code constants.""" + + _assert_closed_world_operations(stage, cgt_residential_split_operation_parameters()) + if stage.stage != CGT_RESIDENTIAL_SPLIT_STAGE_NAME: + raise ValueError( + f"Expected stage {CGT_RESIDENTIAL_SPLIT_STAGE_NAME!r}, got {stage.stage!r}." + ) + expected_outputs = UKCGTResidentialSplitStageTransform.output_columns() + if tuple(stage.outputs) != expected_outputs or tuple(stage.rewrites): + raise ValueError( + "The CGT residential split writes exactly its flag, arm-index, " + "probability and residential-gains columns; the manifest must declare " + f"outputs {expected_outputs} and no rewrites, got " + f"{tuple(stage.outputs)} / {tuple(stage.rewrites)}." + ) + if stage.grain != "household": + raise ValueError( + "The CGT residential split moves household mass; grain must be " + f"'household', got {stage.grain!r}." + ) + artifacts = {str(artifact.get("role")): artifact for artifact in stage.artifacts} + facts = artifacts.get("cgt_asset_type_facts") + if ( + facts is None + or facts.get("resource") != HMRC_CGT_ASSET_TYPE_RESOURCE + or facts.get("runtime_sha256_required") is not True + ): + raise ValueError( + "The CGT residential split must declare the cgt_asset_type_facts " + f"artifact {HMRC_CGT_ASSET_TYPE_RESOURCE!r} with " + "runtime_sha256_required: true." + ) + policy = artifacts.get("policy_parameters") + declared = tuple(policy.get("parameters", ())) if policy is not None else () + if policy is None or any(name not in declared for name in _POLICY_PARAMETER_NAMES): + raise ValueError( + "The CGT residential split must declare the annual exempt amount " + f"policy-parameter artifact with {_POLICY_PARAMETER_NAMES}." + ) + + +__all__ = [ + "CGT_RESIDENTIAL_CLONE_INDEX_COLUMN", + "CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS", + "CGT_RESIDENTIAL_PROBABILITY_COLUMN", + "CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON", + "CGT_RESIDENTIAL_SPLIT_OPERATION_KIND", + "CGT_RESIDENTIAL_SPLIT_STAGE_NAME", + "HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE", + "UKCGTResidentialSplitResult", + "UKCGTResidentialSplitStageTransform", + "cgt_residential_split_operation_parameters", + "residential_arm_factors", + "split_cgt_residential_households", +] diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_structure.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_structure.py index dc78de2fe..292309927 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_structure.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/cgt_structure.py @@ -9,6 +9,7 @@ from __future__ import annotations +import math from collections.abc import Mapping from dataclasses import dataclass, field from typing import Any @@ -421,6 +422,19 @@ def reporter_composition_quantiles( return lower[indexes], upper[indexes] +def _order_free_sum(values: np.ndarray) -> float: + """The correctly rounded sum, which does not depend on the row order. + + Every sum the anchor's factors depend on goes through here: a pairwise + float sum moves in its last bits when the rows are permuted, the bisection + then settles on a neighbouring scale, and the anchored weights differ by a + few ulps between two orderings of the same households (the E8 identity + receipt found 1e-13 on the licensed spine; microcosm#1063). + """ + + return math.fsum(np.asarray(values, dtype=float).tolist()) + + def _solve_group_factors( weights: np.ndarray, propensity: np.ndarray, @@ -440,8 +454,8 @@ def _solve_group_factors( middle = 0.5 * (lower + upper) if middle <= lower or middle >= upper: break - achieved = float( - (weights * np.minimum(CGT_ANCHOR_MAXIMUM_FACTOR, middle * propensity)).sum() + achieved = _order_free_sum( + weights * np.minimum(CGT_ANCHOR_MAXIMUM_FACTOR, middle * propensity) ) if achieved < target: lower = middle @@ -504,7 +518,7 @@ def anchor_cgt_incidence( raise ValueError("CGT incidence anchor found a person without a household.") person_positions = person_positions.astype(np.int64) liable = gains > annual_exempt_amount - liable_mass = float(weights[person_positions][liable].sum()) + liable_mass = _order_free_sum(weights[person_positions][liable]) if not liable_mass > 0.0: raise ValueError( "CGT incidence anchor requires positive liable mass above the annual " @@ -541,10 +555,11 @@ def anchor_cgt_incidence( ) clone_weights = weights[clone_positions] mix = clone_weights * propensity - if not mix.sum() > 0.0: + mix_total = _order_free_sum(mix) + if not mix_total > 0.0: raise ValueError("CGT incidence anchor found no clone mass to compose.") - zero_quantile = float((mix * zero_quantiles).sum() / mix.sum()) - exempt_quantile = float((mix * exempt_quantiles).sum() / mix.sum()) + zero_quantile = _order_free_sum(mix * zero_quantiles) / mix_total + exempt_quantile = _order_free_sum(mix * exempt_quantiles) / mix_total if not 0.0 <= zero_quantile < exempt_quantile < 1.0: raise ValueError( "CGT incidence anchor derived an unusable reporter composition: " @@ -565,7 +580,9 @@ def anchor_cgt_incidence( "loss": carrier_gains < 0.0, } liable_clone = carrier_gains > annual_exempt_amount - before = {name: float(clone_weights[mask].sum()) for name, mask in groups.items()} + before = { + name: _order_free_sum(clone_weights[mask]) for name, mask in groups.items() + } before["liable"] = float(clone_weights[liable_clone].sum()) factors = np.ones(len(clone_positions), dtype=float) scale: dict[str, float | None] = {} diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/child_benefit_take_up.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/child_benefit_take_up.py new file mode 100644 index 000000000..1531cb6f9 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/child_benefit_take_up.py @@ -0,0 +1,811 @@ +"""Child Benefit claims and opt-outs on post-SPI incomes (microcosm#1063). + +``frs_take_up`` draws ``would_claim_child_benefit`` early, at one flat rate +over every benefit unit with the FRS reporters added on top, and +``child_benefit_opts_out`` at a flat rate over every benefit unit whatever its +income; the engine pays on the first and never reads the second. This stage +redraws both once the SPI income chain has set the incomes the charge is +assessed on, against HMRC's Child Benefit statistics. + +Claims. One temporary engine materialization gives each person's eligibility +(a child, or a qualifying young person). HMRC publishes the share of eligible +children for whom Child Benefit is claimed, opted-out families included, by +single year of age. A family claims for all its children or for none, so the +draw is by family, keyed on the age of its eldest eligible child: a family +that reports the benefit claims, and the other families of key age ``g`` +claim at a rate ``r_g``. The rates are solved from the eldest age down so that +the claimed share of the children of each single year of age is the published +one: + + r_g = (t_g * C_g - R_g - sum_{k > g} N_{g,k} * r_k) / N_{g,g} + +with ``t_g`` the published rate at age ``g``, ``C_g`` the eligible children +aged ``g``, ``R_g`` those in reporting families and ``N_{g,k}`` those in +non-reporting families whose eldest eligible child is aged ``k``, all weighted +at the stage's household weights. A rate outside [0, 1] is clipped and the +receipt says so (reporters alone exceed the published rate, or every family +claiming still falls short of it). + +Opt-outs. HMRC publishes the families registered and, of them, the families +that opted out of payment because of the High Income Child Benefit Charge. +That share of the claiming families opts out, drawn among claiming families +that do not report receipt and whose highest adjusted net income is at or +above the income at which the charge takes the whole benefit; where that pool +is lighter than the target the rest is drawn from the families inside the +taper. A family in the taper still gains from the payment, so the fully +charged families come first. + +The engine pays on ``would_claim_child_benefit`` alone, so the stage stores +``claims and not opted out`` there and the opt-out in +``child_benefit_opts_out``. A benefit unit with no eligible child keeps its +early draw and is never opted out. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np +import pandas as pd + +from microcosm.build.source_manifest import SourceStageSpec +from microcosm.build.stochastic_assignment import stable_identity_uniforms +from microcosm.build.uk_runtime.ledger_fact_vendoring import vendored_rows +from microcosm.build.uk_runtime.national_frame import ( + uk_household_weight_kind, + uk_national_frame, + uk_time_period, + validate_uk_national_frame, +) +from microcosm.build.uk_runtime.uc_capital_coherence import ( + _household_to_benunit_weights, +) +from microcosm.frame import Frame +from microcosm.frame.rules import assert_rules_engine_country + +CHILD_BENEFIT_TAKE_UP_STAGE_NAME = "child_benefit_take_up" +CHILD_BENEFIT_CLAIM_OUTPUT = "would_claim_child_benefit" +CHILD_BENEFIT_OPT_OUT_OUTPUT = "child_benefit_opts_out" +CHILD_BENEFIT_TAKE_UP_SEED = 0 +CHILD_BENEFIT_ELIGIBILITY_VARIABLE = ( + "is_child_or_qualifying_young_person_for_child_benefit" +) +CHILD_BENEFIT_CHARGE_INCOME_VARIABLE = "adjusted_net_income" +CHILD_BENEFIT_ENGINE_VARIABLES = ( + CHILD_BENEFIT_ELIGIBILITY_VARIABLE, + CHILD_BENEFIT_CHARGE_INCOME_VARIABLE, +) +CHILD_BENEFIT_REPORTED_ANCHOR = "child_benefit_reported_anchor" +CHILD_BENEFIT_AGGREGATES = {CHILD_BENEFIT_REPORTED_ANCHOR: "child_benefit_reported"} +CHILD_BENEFIT_STATISTICS_RESOURCE = "hmrc_child_benefit_statistics.json" +#: HMRC Child Benefit statistics, annual release at August 2025: the claim +#: rates are estimated at the May reference month, the caseload at August. +CHILD_BENEFIT_CLAIM_RATE_PERIOD = "2025-05" +CHILD_BENEFIT_CASELOAD_PERIOD = "2025-08" +CHILD_BENEFIT_CLAIM_RATE_MEASURE = "claim_rate" +CHILD_BENEFIT_FAMILIES_REGISTERED_MEASURE = "families_registered" +CHILD_BENEFIT_FAMILIES_IN_PAYMENT_MEASURE = "families_in_payment" +CHILD_BENEFIT_FAMILIES_OPTED_OUT_MEASURE = "families_opted_out_claimant_all" +CHILD_BENEFIT_CHILDREN_IN_PAYMENT_MEASURE = "children_in_payment_united_kingdom" +CHILD_BENEFIT_CHILDREN_OPTED_OUT_MEASURE = "children_opted_out_united_kingdom" +CHILD_BENEFIT_ALL_AGES = "All ages" +CHILD_BENEFIT_CHARGE_PARAMETERS = "gov.hmrc.income_tax.charges.CB_HITC" +CHILD_BENEFIT_FAMILY_KEY = "age of the benefit unit's eldest eligible child" +CHILD_BENEFIT_CLAIM_RULE = ( + "a reporter claims; a non-reporting family whose eldest eligible child is " + "aged g claims at r_g = (t_g * C_g - R_g - sum over k > g of N_gk * r_k) / " + "N_gg, weighted, solved from the eldest age down and clipped to [0, 1]" +) +CHILD_BENEFIT_OPT_OUT_RULE = ( + "published opted-out share of registered families, of the claiming " + "families; drawn among claiming non-reporting families with the highest " + "adjusted net income at or above the full-charge income, then, for any " + "remainder, among those inside the taper" +) +CHILD_BENEFIT_PAYMENT_RULE = ( + "would_claim_child_benefit = claims and not opted out on families with an " + "eligible child; a benefit unit without one keeps its early draw" +) + + +@dataclass(frozen=True) +class UKChildBenefitStatistics: + """HMRC's published rows the stage draws against.""" + + claim_rates: Mapping[int, float] + all_ages_claim_rate: float + families_registered: float + families_in_payment: float + families_opted_out: float + children_in_payment: float + children_opted_out: float + claim_rate_period: str = CHILD_BENEFIT_CLAIM_RATE_PERIOD + caseload_period: str = CHILD_BENEFIT_CASELOAD_PERIOD + + def __post_init__(self) -> None: + if not self.claim_rates: + raise ValueError("Child Benefit claim rates by age are empty.") + for age, rate in self.claim_rates.items(): + if not isinstance(age, int) or not 0.0 <= float(rate) <= 1.0: + raise ValueError( + f"Child Benefit claim rate at age {age!r} must lie in [0, 1], " + f"got {rate!r}." + ) + if not 0.0 < self.families_opted_out < self.families_registered: + raise ValueError( + "Child Benefit opted-out families must be a positive part of the " + "registered families." + ) + + @property + def opt_out_share(self) -> float: + """Opted-out families over registered families.""" + + return float(self.families_opted_out) / float(self.families_registered) + + +@dataclass(frozen=True) +class UKChildBenefitChargeThresholds: + """Adjusted net income at which the charge starts and takes it all.""" + + phase_out_start: float + phase_out_end: float + source: str = "caller" + + def __post_init__(self) -> None: + if not 0.0 < self.phase_out_start < self.phase_out_end: + raise ValueError( + "Child Benefit charge thresholds must satisfy 0 < start < end." + ) + + +def load_child_benefit_statistics() -> UKChildBenefitStatistics: + """Read the vendored HMRC rows; a copy that lags the feed pin refuses.""" + + def one(measure: str, period: str, **dimensions: object) -> float: + rows = vendored_rows( + CHILD_BENEFIT_STATISTICS_RESOURCE, + measure_id=measure, + period_type="month", + period_value=period, + **({"dimensions": dimensions} if dimensions else {}), + ) + if len(rows) != 1: + raise ValueError( + f"{CHILD_BENEFIT_STATISTICS_RESOURCE} must carry exactly one " + f"{measure} row for {period} {dimensions or ''}; found {len(rows)}." + ) + return float(rows[0]["value"]) + + rate_rows = vendored_rows( + CHILD_BENEFIT_STATISTICS_RESOURCE, + measure_id=CHILD_BENEFIT_CLAIM_RATE_MEASURE, + period_type="month", + period_value=CHILD_BENEFIT_CLAIM_RATE_PERIOD, + ) + rates: dict[int, float] = {} + all_ages: float | None = None + for row in rate_rows: + if row.get("unit") != "percent": + raise ValueError( + f"{CHILD_BENEFIT_STATISTICS_RESOURCE} claim rates must be in " + f"percent, got {row.get('unit')!r}." + ) + age = (row.get("dimensions") or {}).get("child_age") + # The publisher states a percentage; the stage's arithmetic is on the + # fraction. + value = float(row["value"]) / 100.0 + if str(age) == CHILD_BENEFIT_ALL_AGES: + all_ages = value + continue + key = int(age) + if key in rates: + raise ValueError( + f"{CHILD_BENEFIT_STATISTICS_RESOURCE} repeats the claim rate at " + f"age {key}." + ) + rates[key] = value + if all_ages is None or sorted(rates) != list(range(min(rates), max(rates) + 1)): + raise ValueError( + f"{CHILD_BENEFIT_STATISTICS_RESOURCE} must carry the all-ages claim " + "rate and one rate for every single year of age without gaps." + ) + caseload = CHILD_BENEFIT_CASELOAD_PERIOD + return UKChildBenefitStatistics( + claim_rates=rates, + all_ages_claim_rate=all_ages, + families_registered=one(CHILD_BENEFIT_FAMILIES_REGISTERED_MEASURE, caseload), + families_in_payment=one(CHILD_BENEFIT_FAMILIES_IN_PAYMENT_MEASURE, caseload), + families_opted_out=one(CHILD_BENEFIT_FAMILIES_OPTED_OUT_MEASURE, caseload), + children_in_payment=one( + CHILD_BENEFIT_CHILDREN_IN_PAYMENT_MEASURE, + caseload, + child_age=CHILD_BENEFIT_ALL_AGES, + ), + children_opted_out=one( + CHILD_BENEFIT_CHILDREN_OPTED_OUT_MEASURE, + caseload, + child_age=CHILD_BENEFIT_ALL_AGES, + ), + ) + + +def uk_child_benefit_charge_thresholds( + build_period: int | str, +) -> UKChildBenefitChargeThresholds: + """Read the charge's taper from the engine at ``{year}-01-01``.""" + + try: + import policyengine_uk + from policyengine_core.parameters import ParameterNode + except ImportError as exc: + raise ImportError( + "The Child Benefit charge thresholds require " + "`uv sync --all-packages --extra uk`." + ) from exc + from importlib import metadata + + parameters = ParameterNode( + directory_path=str(Path(policyengine_uk.__file__).parent / "parameters") + ) + instant = f"{int(build_period)}-01-01" + charge = parameters.gov.hmrc.income_tax.charges.CB_HITC + return UKChildBenefitChargeThresholds( + phase_out_start=float(charge.phase_out_start(instant)), + phase_out_end=float(charge.phase_out_end(instant)), + source=( + f"policyengine-uk {metadata.version('policyengine-uk')} " + f"{CHILD_BENEFIT_CHARGE_PARAMETERS} at {instant}" + ), + ) + + +@dataclass(frozen=True) +class UKChildBenefitTakeUpResult: + """Output frame and executed-effect receipt for the Child Benefit redraw.""" + + frame: Frame + claims: Mapping[str, object] + opt_outs: Mapping[str, object] + in_payment: Mapping[str, object] + reporters_without_eligible_child: Mapping[str, float] + changed_units: Mapping[str, int] + + def evidence(self) -> dict[str, object]: + return { + "stage": CHILD_BENEFIT_TAKE_UP_STAGE_NAME, + "seed": CHILD_BENEFIT_TAKE_UP_SEED, + "family_key": CHILD_BENEFIT_FAMILY_KEY, + "claim_rule": CHILD_BENEFIT_CLAIM_RULE, + "opt_out_rule": CHILD_BENEFIT_OPT_OUT_RULE, + "payment_rule": CHILD_BENEFIT_PAYMENT_RULE, + "claims": dict(self.claims), + "opt_outs": dict(self.opt_outs), + "in_payment": dict(self.in_payment), + "reporters_without_eligible_child": dict( + self.reporters_without_eligible_child + ), + "changed_units": dict(self.changed_units), + } + + +@dataclass(frozen=True) +class UKChildBenefitTakeUpStageTransform: + """Whole-stage callable for the post-SPI Child Benefit redraw.""" + + stage: SourceStageSpec + engine: object + statistics: UKChildBenefitStatistics | None = None + thresholds: UKChildBenefitChargeThresholds | None = None + last_result: UKChildBenefitTakeUpResult | None = field(default=None, init=False) + + def __call__(self, frame: Frame) -> Frame: + _assert_stage_parameters(self.stage) + result = redraw_child_benefit_take_up( + frame, + engine=self.engine, + statistics=self.statistics or load_child_benefit_statistics(), + thresholds=self.thresholds + or uk_child_benefit_charge_thresholds(uk_time_period(frame)), + ) + object.__setattr__(self, "last_result", result) + return result.frame + + @staticmethod + def output_columns() -> tuple[str, ...]: + return () + + def checkpoint_metadata(self) -> dict[str, object]: + if self.last_result is None: + raise RuntimeError("checkpoint metadata requires a completed stage run.") + return {"evidence": self.last_result.evidence()} + + +def redraw_child_benefit_take_up( + frame: Frame, + *, + engine: object, + statistics: UKChildBenefitStatistics, + thresholds: UKChildBenefitChargeThresholds, +) -> UKChildBenefitTakeUpResult: + """Redraw the claim and opt-out flags from engine eligibility and income.""" + + validate_uk_national_frame(frame) + assert_rules_engine_country(engine, "uk") + person = frame.table("person").copy() + benunit = frame.table("benunit").copy() + household = frame.table("household").copy() + for table, columns, label in ( + ( + person, + ( + "person_benunit_id", + "person_household_id", + "age", + "child_benefit_reported", + ), + "person", + ), + ( + benunit, + ("benunit_id", CHILD_BENEFIT_CLAIM_OUTPUT, CHILD_BENEFIT_OPT_OUT_OUTPUT), + "benunit", + ), + ): + missing = sorted(set(columns) - set(table.columns)) + if missing: + raise ValueError( + f"Child Benefit take-up {label} columns missing: {missing}." + ) + period = uk_time_period(frame) + materialized = engine.materialize( + frame, list(CHILD_BENEFIT_ENGINE_VARIABLES), period + ) + missing = sorted(set(CHILD_BENEFIT_ENGINE_VARIABLES) - set(materialized)) + if missing: + raise ValueError(f"Child Benefit take-up engine outputs missing: {missing}.") + eligible_child = _aligned( + materialized[CHILD_BENEFIT_ELIGIBILITY_VARIABLE], + len(person), + CHILD_BENEFIT_ELIGIBILITY_VARIABLE, + ).astype(bool) + income = _aligned( + materialized[CHILD_BENEFIT_CHARGE_INCOME_VARIABLE], + len(person), + CHILD_BENEFIT_CHARGE_INCOME_VARIABLE, + ).astype(np.float64) + weights = _household_to_benunit_weights( + benunit, + person=person, + household=household, + household_weights=frame.weights_for("household").values, + ) + benunit_ids = benunit["benunit_id"].to_numpy() + position = pd.Series(np.arange(len(benunit)), index=benunit_ids) + person_position = person["person_benunit_id"].map(position).to_numpy() + if np.isnan(person_position.astype(np.float64)).any(): + raise ValueError("Child Benefit take-up found a person outside every unit.") + person_position = person_position.astype(np.int64) + ages = np.floor(pd.to_numeric(person["age"], errors="raise").to_numpy(float)) + child_units = person_position[eligible_child] + child_ages = ages[eligible_child].astype(np.int64) + eldest = np.full(len(benunit), -1, dtype=np.int64) + np.maximum.at(eldest, child_units, child_ages) + children = np.bincount(child_units, minlength=len(benunit)).astype(np.int64) + # The charge falls on the unit's highest adjusted net income among the + # people who are not themselves the children the benefit is paid for. + highest_income = np.full(len(benunit), -np.inf) + np.maximum.at( + highest_income, person_position[~eligible_child], income[~eligible_child] + ) + highest_income[~np.isfinite(highest_income)] = 0.0 + reporter_ids = person.loc[ + pd.to_numeric(person["child_benefit_reported"], errors="coerce").fillna(0.0) + > 0, + "person_benunit_id", + ] + reporter = benunit["benunit_id"].isin(reporter_ids).to_numpy(dtype=bool) + claim_draws = stable_identity_uniforms( + benunit_ids, seed=CHILD_BENEFIT_TAKE_UP_SEED, salt=CHILD_BENEFIT_CLAIM_OUTPUT + ) + opt_out_draws = stable_identity_uniforms( + benunit_ids, seed=CHILD_BENEFIT_TAKE_UP_SEED, salt=CHILD_BENEFIT_OPT_OUT_OUTPUT + ) + + claims, claim_receipt = assign_child_benefit_claims( + child_units=child_units, + child_ages=child_ages, + eldest=eldest, + reporter=reporter, + weights=weights, + draws=claim_draws, + statistics=statistics, + ) + opt_out, opt_out_receipt = assign_child_benefit_opt_outs( + claims=claims, + reporter=reporter, + highest_income=highest_income, + children=children, + weights=weights, + draws=opt_out_draws, + statistics=statistics, + thresholds=thresholds, + ) + family = children > 0 + paid = claims & ~opt_out + previous_claim = benunit[CHILD_BENEFIT_CLAIM_OUTPUT].fillna(False).to_numpy(bool) + previous_opt_out = ( + benunit[CHILD_BENEFIT_OPT_OUT_OUTPUT].fillna(False).to_numpy(bool) + ) + would_claim = np.where(family, paid, previous_claim) + benunit[CHILD_BENEFIT_CLAIM_OUTPUT] = would_claim + benunit[CHILD_BENEFIT_OPT_OUT_OUTPUT] = opt_out + result = uk_national_frame( + person=person, + benunit=benunit, + household=household, + time_period=period, + weight_kind=uk_household_weight_kind(frame), + household_weights=frame.weights_for("household").values, + mass_log=frame.mass_log, + ) + validate_uk_national_frame(result) + paid_families = float(weights[family & paid].sum()) + paid_children = float((weights * children)[family & paid].sum()) + charged = family & paid & (highest_income > thresholds.phase_out_start) + outside = reporter & ~family + return UKChildBenefitTakeUpResult( + frame=result, + claims=claim_receipt, + opt_outs=opt_out_receipt, + in_payment={ + "weights": "household weights at the stage, before calibration", + "families": paid_families, + "children": paid_children, + "published_families": statistics.families_in_payment, + "published_children": statistics.children_in_payment, + "published_period": statistics.caseload_period, + "families_with_income_above_charge_start": float(weights[charged].sum()), + }, + reporters_without_eligible_child={ + "units": int(outside.sum()), + "weighted_units": float(weights[outside].sum()), + }, + changed_units={ + CHILD_BENEFIT_CLAIM_OUTPUT: int((previous_claim != would_claim).sum()), + CHILD_BENEFIT_OPT_OUT_OUTPUT: int((previous_opt_out != opt_out).sum()), + }, + ) + + +def solve_child_benefit_claim_rates( + *, + child_ages: np.ndarray, + child_key_ages: np.ndarray, + child_reporter: np.ndarray, + child_weights: np.ndarray, + claim_rates: Mapping[int, float], +) -> tuple[dict[int, float], list[dict[str, object]]]: + """Non-reporter claim rate by key age, and one receipt row per age. + + ``child_*`` describe the eligible children: own age, the family's key age + (its eldest eligible child's), whether the family reports, and the weight. + """ + + child_ages = np.asarray(child_ages, dtype=np.int64) + child_key_ages = np.asarray(child_key_ages, dtype=np.int64) + child_reporter = np.asarray(child_reporter, dtype=bool) + child_weights = np.asarray(child_weights, dtype=np.float64) + unknown = sorted(set(child_ages.tolist()) - set(claim_rates)) + if unknown: + raise ValueError( + "HMRC publishes no Child Benefit claim rate for eligible children " + f"aged {unknown}." + ) + if (child_key_ages < child_ages).any(): + raise ValueError("A family's key age is below one of its children's ages.") + residual: dict[int, float] = {} + rows: list[dict[str, object]] = [] + for age in sorted(claim_rates, reverse=True): + of_age = child_ages == age + eligible = float(child_weights[of_age].sum()) + reporting = float(child_weights[of_age & child_reporter].sum()) + own = of_age & ~child_reporter & (child_key_ages == age) + own_mass = float(child_weights[own].sum()) + # Children of this age in non-reporting families keyed on an older + # sibling claim at that sibling's rate, already solved. + older = of_age & ~child_reporter & (child_key_ages > age) + older_claimed = float( + sum( + child_weights[older & (child_key_ages == key)].sum() * rate + for key, rate in residual.items() + ) + ) + raw: float | None = None + rate = 0.0 + if own_mass > 0.0: + raw = (claim_rates[age] * eligible - reporting - older_claimed) / own_mass + rate = float(np.clip(raw, 0.0, 1.0)) + residual[age] = rate + rows.append( + { + "age": int(age), + "published_rate": float(claim_rates[age]), + "eligible_child_rows": int(of_age.sum()), + "eligible_children": eligible, + "reporter_children": reporting, + "residual_rate_unclipped": raw, + "residual_rate": rate, + "clipped": bool(raw is not None and not 0.0 <= raw <= 1.0), + } + ) + rows.sort(key=lambda row: row["age"]) + return residual, rows + + +def assign_child_benefit_claims( + *, + child_units: np.ndarray, + child_ages: np.ndarray, + eldest: np.ndarray, + reporter: np.ndarray, + weights: np.ndarray, + draws: np.ndarray, + statistics: UKChildBenefitStatistics, +) -> tuple[np.ndarray, dict[str, object]]: + """Family claims: reporters, and the rest at their key age's solved rate. + + ``child_units`` holds each eligible child's benefit-unit position and + ``child_ages`` its whole age; ``eldest`` is each unit's eldest eligible + child's age, or -1 for a unit without one. + """ + + child_units = np.asarray(child_units, dtype=np.int64) + child_ages = np.asarray(child_ages, dtype=np.int64) + eldest = np.asarray(eldest, dtype=np.int64) + reporter = np.asarray(reporter, dtype=bool) + weights = np.asarray(weights, dtype=np.float64) + draws = np.asarray(draws, dtype=np.float64) + family = eldest >= 0 + child_weights = weights[child_units] + residual, rows = solve_child_benefit_claim_rates( + child_ages=child_ages, + child_key_ages=eldest[child_units], + child_reporter=reporter[child_units], + child_weights=child_weights, + claim_rates=statistics.claim_rates, + ) + unit_rate = np.zeros(len(eldest), dtype=np.float64) + unit_rate[family] = np.asarray([residual[int(age)] for age in eldest[family]]) + claims = family & (reporter | (draws < unit_rate)) + child_claims = claims[child_units] + for row in rows: + of_age = child_ages == row["age"] + eligible = float(row["eligible_children"]) + row["realized_rate"] = ( + float(child_weights[of_age & child_claims].sum()) / eligible + if eligible > 0.0 + else None + ) + total = float(child_weights.sum()) + target = ( + float( + sum( + statistics.claim_rates[int(row["age"])] * row["eligible_children"] + for row in rows + ) + ) + / total + if total > 0.0 + else None + ) + receipt = { + "published_period": statistics.claim_rate_period, + "published_all_ages_rate": statistics.all_ages_claim_rate, + # The published rates by age at the frame's own age mix: what the + # solved rates aim at overall. + "target_rate": target, + "realized_rate": ( + float(child_weights[child_claims].sum()) / total if total > 0.0 else None + ), + "eligible_children": total, + "eligible_child_rows": int(len(child_units)), + "eligible_families": float(weights[family].sum()), + "eligible_family_units": int(family.sum()), + "reporter_families": float(weights[family & reporter].sum()), + "claiming_families": float(weights[claims].sum()), + "clipped_ages": [int(row["age"]) for row in rows if row["clipped"]], + "ages": rows, + } + return claims, receipt + + +def assign_child_benefit_opt_outs( + *, + claims: np.ndarray, + reporter: np.ndarray, + highest_income: np.ndarray, + children: np.ndarray, + weights: np.ndarray, + draws: np.ndarray, + statistics: UKChildBenefitStatistics, + thresholds: UKChildBenefitChargeThresholds, +) -> tuple[np.ndarray, dict[str, object]]: + """Opt out the published share of claiming families, fully charged first.""" + + claims = np.asarray(claims, dtype=bool) + reporter = np.asarray(reporter, dtype=bool) + highest_income = np.asarray(highest_income, dtype=np.float64) + children = np.asarray(children, dtype=np.float64) + weights = np.asarray(weights, dtype=np.float64) + draws = np.asarray(draws, dtype=np.float64) + candidates = claims & ~reporter + pools = ( + ( + "fully_charged", + candidates & (highest_income >= thresholds.phase_out_end), + ), + ( + "taper", + candidates + & (highest_income > thresholds.phase_out_start) + & (highest_income < thresholds.phase_out_end), + ), + ) + claiming = float(weights[claims].sum()) + target = statistics.opt_out_share * claiming + remaining = target + opt_out = np.zeros(len(claims), dtype=bool) + pool_rows = [] + for name, pool in pools: + mass = float(weights[pool].sum()) + rate = float(np.clip(remaining / mass, 0.0, 1.0)) if mass > 0.0 else 0.0 + drawn = pool & (draws < rate) + opt_out |= drawn + pool_rows.append( + { + "pool": name, + "units": int(pool.sum()), + "weighted_families": mass, + "rate": rate, + "opted_out_units": int(drawn.sum()), + "opted_out_families": float(weights[drawn].sum()), + } + ) + remaining = max(0.0, remaining - mass) + realized = float(weights[opt_out].sum()) + opted_children = float((weights * children)[opt_out].sum()) + receipt = { + "published_period": statistics.caseload_period, + "published_families_registered": statistics.families_registered, + "published_families_opted_out": statistics.families_opted_out, + "target_share": statistics.opt_out_share, + "claiming_families": claiming, + "target_families": target, + "opted_out_families": realized, + "realized_share": realized / claiming if claiming > 0.0 else None, + # Target mass no charged family was left to carry. + "pool_shortfall_families": remaining, + "pool_exhausted": bool(remaining > 0.0), + "opted_out_children": opted_children, + "children_per_opted_out_family": ( + opted_children / realized if realized > 0.0 else None + ), + "published_children_per_opted_out_family": ( + statistics.children_opted_out / statistics.families_opted_out + ), + "charge_income": CHILD_BENEFIT_CHARGE_INCOME_VARIABLE, + "phase_out_start": thresholds.phase_out_start, + "phase_out_end": thresholds.phase_out_end, + "thresholds_source": thresholds.source, + "reporters_above_charge_start": float( + weights[ + claims & reporter & (highest_income > thresholds.phase_out_start) + ].sum() + ), + "pools": pool_rows, + } + return opt_out, receipt + + +def _aligned(values: object, expected: int, label: str) -> np.ndarray: + array = np.asarray(values) + if array.shape != (expected,): + raise ValueError(f"{label} must align to the person table.") + if array.dtype != bool and not np.isfinite(array.astype(np.float64)).all(): + raise ValueError(f"{label} must be finite.") + return array + + +def child_benefit_take_up_operation_parameters() -> dict[str, dict[str, object]]: + """The reviewed operation payload the manifests must carry verbatim.""" + + return { + "materialize_rules_engine_predictors": { + "predictors": list(CHILD_BENEFIT_ENGINE_VARIABLES), + "consumed_only": True, + }, + "aggregate_person_to_benunit": { + "method": "any_positive", + "consumed_only": True, + "aggregates": dict(CHILD_BENEFIT_AGGREGATES), + }, + "assign_family_claims_by_child_age": { + "output": CHILD_BENEFIT_CLAIM_OUTPUT, + "draw": CHILD_BENEFIT_CLAIM_OUTPUT, + "seed": CHILD_BENEFIT_TAKE_UP_SEED, + "anchor": CHILD_BENEFIT_REPORTED_ANCHOR, + "eligibility": CHILD_BENEFIT_ELIGIBILITY_VARIABLE, + "family_key": CHILD_BENEFIT_FAMILY_KEY, + "resource": CHILD_BENEFIT_STATISTICS_RESOURCE, + "rate_measure": CHILD_BENEFIT_CLAIM_RATE_MEASURE, + "rate_period": CHILD_BENEFIT_CLAIM_RATE_PERIOD, + "weight_mapping": "household_to_benunit", + "rule": CHILD_BENEFIT_CLAIM_RULE, + }, + "assign_opt_out_by_charge_income": { + "output": CHILD_BENEFIT_OPT_OUT_OUTPUT, + "draw": CHILD_BENEFIT_OPT_OUT_OUTPUT, + "seed": CHILD_BENEFIT_TAKE_UP_SEED, + "resource": CHILD_BENEFIT_STATISTICS_RESOURCE, + "numerator_measure": CHILD_BENEFIT_FAMILIES_OPTED_OUT_MEASURE, + "denominator_measure": CHILD_BENEFIT_FAMILIES_REGISTERED_MEASURE, + "caseload_period": CHILD_BENEFIT_CASELOAD_PERIOD, + "charge_income": CHILD_BENEFIT_CHARGE_INCOME_VARIABLE, + "charge_parameters": CHILD_BENEFIT_CHARGE_PARAMETERS, + "pools": ["fully_charged", "taper"], + "weight_mapping": "household_to_benunit", + "rule": CHILD_BENEFIT_OPT_OUT_RULE, + "payment_rule": CHILD_BENEFIT_PAYMENT_RULE, + }, + } + + +def _assert_stage_parameters(stage: SourceStageSpec) -> None: + if stage.stage != CHILD_BENEFIT_TAKE_UP_STAGE_NAME: + raise ValueError( + f"Child Benefit take-up received stage {stage.stage!r}, expected " + f"{CHILD_BENEFIT_TAKE_UP_STAGE_NAME!r}." + ) + expected = child_benefit_take_up_operation_parameters() + kinds = [operation.kind for operation in stage.operations] + if kinds != list(expected): + raise ValueError( + f"{CHILD_BENEFIT_TAKE_UP_STAGE_NAME} operation order drifted: " + f"expected {list(expected)}, got {kinds}." + ) + actual = { + operation.kind: dict(operation.parameters) for operation in stage.operations + } + if actual != expected: + drifted = sorted(kind for kind in expected if actual[kind] != expected[kind]) + raise ValueError( + f"{CHILD_BENEFIT_TAKE_UP_STAGE_NAME} parameters drifted from the " + f"reviewed mapping on operation(s) {drifted}." + ) + rewrites = (CHILD_BENEFIT_CLAIM_OUTPUT, CHILD_BENEFIT_OPT_OUT_OUTPUT) + if stage.outputs != () or stage.rewrites != rewrites: + raise ValueError( + f"{CHILD_BENEFIT_TAKE_UP_STAGE_NAME} must declare no new outputs and " + f"exactly the rewrites {list(rewrites)}." + ) + + +__all__ = [ + "CHILD_BENEFIT_ENGINE_VARIABLES", + "CHILD_BENEFIT_STATISTICS_RESOURCE", + "CHILD_BENEFIT_TAKE_UP_STAGE_NAME", + "UKChildBenefitChargeThresholds", + "UKChildBenefitStatistics", + "UKChildBenefitTakeUpResult", + "UKChildBenefitTakeUpStageTransform", + "assign_child_benefit_claims", + "assign_child_benefit_opt_outs", + "child_benefit_take_up_operation_parameters", + "load_child_benefit_statistics", + "redraw_child_benefit_take_up", + "solve_child_benefit_claim_rates", + "uk_child_benefit_charge_thresholds", +] diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py index 8ce2bbc2f..4a961a604 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/diagnostics.py @@ -487,6 +487,9 @@ def uk_weight_summary( "household_is_capital_gains_clone", ) UK_SUPPORT_COPY_FLAG_COLUMN = "household_is_cgt_support_copy" +#: The residential split's arms (microcosm#1063) carry their household's +#: lineage key, so they fold onto it like the support copies. +UK_RESIDENTIAL_ARM_FLAG_COLUMN = "household_is_cgt_residential_clone" def uk_support_family_geographic_clone_column() -> str: @@ -517,11 +520,10 @@ def uk_support_family_weights( weights = pd.to_numeric(household["household_weight"], errors="coerce").to_numpy( dtype=np.float64 ) - copies = ( - household[UK_SUPPORT_COPY_FLAG_COLUMN].fillna(False).to_numpy(dtype=bool) - if UK_SUPPORT_COPY_FLAG_COLUMN in household.columns - else np.zeros(len(household), dtype=bool) - ) + copies = np.zeros(len(household), dtype=bool) + for flag in (UK_SUPPORT_COPY_FLAG_COLUMN, UK_RESIDENTIAL_ARM_FLAG_COLUMN): + if flag in household.columns: + copies |= household[flag].fillna(False).to_numpy(dtype=bool) if not copies.any(): return weights.copy(), { "basis": "rows", diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_education.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_education.py index 668a43553..79ebf76b2 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_education.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_education.py @@ -31,6 +31,12 @@ "POST_SECONDARY", ) FRS_APPROVED_TRAINING_CODES = tuple(range(1, 10)) +#: FRS 2024-25 (SN 9563) renames ``TRAIN`` to ``TRAIN2`` and recodes it: the +#: scheme codes are 1-8 (Work Club or Enterprise Club, New Enterprise +#: Allowance, Work Experience, Work Trial, Work Programme, Fair Start +#: Scotland, other local employment support, any other government scheme) +#: and 10 (Kickstart Scheme); 9 is "None of these" (microcosm#1063 c9). +FRS_APPROVED_TRAINING_CODES_TRAIN2 = (1, 2, 3, 4, 5, 6, 7, 8, 10) UNKNOWN_QUALIFYING_EDUCATION_OR_TRAINING_ENTRY_AGE = 1000 BENEFITS_IN_OWN_RIGHT_REPORTED_COLUMNS = ( "universal_credit_reported", @@ -129,10 +135,14 @@ def add_frs_education( ) -> Frame: artifacts = _artifact_by_table(stage) adult = normalize_ids( - read_pinned_tab(Path(raw_dir) / str(artifacts["adult"]["locator"]), artifacts["adult"]) + read_pinned_tab( + Path(raw_dir) / str(artifacts["adult"]["locator"]), artifacts["adult"] + ) ) child = normalize_ids( - read_pinned_tab(Path(raw_dir) / str(artifacts["child"]["locator"]), artifacts["child"]) + read_pinned_tab( + Path(raw_dir) / str(artifacts["child"]["locator"]), artifacts["child"] + ) ) raw_person = pd.concat([adult, child], ignore_index=True, sort=False) person = frame.table("person").copy() @@ -152,7 +162,9 @@ def add_frs_education( return result -def derive_frs_education(person: pd.DataFrame, raw_person: pd.DataFrame) -> pd.DataFrame: +def derive_frs_education( + person: pd.DataFrame, raw_person: pd.DataFrame +) -> pd.DataFrame: raw = raw_person.set_index("person_id").reindex(person["person_id"]) values = pd.DataFrame(index=person.index) age = pd.to_numeric(person["age"], errors="coerce").fillna(0).to_numpy() @@ -167,12 +179,17 @@ def derive_frs_education(person: pd.DataFrame, raw_person: pd.DataFrame) -> pd.D values["is_in_non_advanced_education"] = np.isin( current, NON_ADVANCED_EDUCATION_LEVELS ) - train = _num(raw, "train") if "train" in raw.columns else pd.Series(0, index=raw.index) - values["is_in_approved_training"] = train.isin(FRS_APPROVED_TRAINING_CODES).to_numpy() - in_qualifying = ( - values["is_in_non_advanced_education"].to_numpy(dtype=bool) - | values["is_in_approved_training"].to_numpy(dtype=bool) - ) + if "train" in raw.columns: + approved = _num(raw, "train").isin(FRS_APPROVED_TRAINING_CODES) + elif "train2" in raw.columns: + # The 2024-25 vintage's recoded variable; code 9 is "None of these". + approved = _num(raw, "train2").isin(FRS_APPROVED_TRAINING_CODES_TRAIN2) + else: + approved = pd.Series(False, index=raw.index) + values["is_in_approved_training"] = approved.to_numpy(dtype=bool) + in_qualifying = values["is_in_non_advanced_education"].to_numpy( + dtype=bool + ) | values["is_in_approved_training"].to_numpy(dtype=bool) # Integer by the stage's own contract, whatever dtype ``age`` arrives in # (int64 from the root since #845; float in synthetic frames). values["age_started_or_accepted_current_education_or_training"] = np.where( @@ -181,8 +198,8 @@ def derive_frs_education(person: pd.DataFrame, raw_person: pd.DataFrame) -> pd.D UNKNOWN_QUALIFYING_EDUCATION_OR_TRAINING_ENTRY_AGE, ).astype("int64") values["is_before_universal_credit_qualifying_young_person_terminal_date"] = ( - (age == 19) & in_qualifying - ) + age == 19 + ) & in_qualifying if "adema" in raw.columns: values["adult_ema"] = _ema(raw, code_column="adema", amount_column="ademaamt") else: @@ -264,5 +281,7 @@ def _artifact_by_table(stage: SourceStageSpec) -> dict[str, Mapping[str, Any]]: by_table = {str(artifact.get("table")): artifact for artifact in stage.artifacts} missing = sorted({"adult", "child"} - set(by_table)) if missing: - raise ValueError(f"frs_education manifest is missing tab artifact(s): {missing}.") + raise ValueError( + f"frs_education manifest is missing tab artifact(s): {missing}." + ) return by_table diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_take_up.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_take_up.py index 338422e6a..3adef30c1 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_take_up.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/frs_take_up.py @@ -74,9 +74,11 @@ # would_claim_pc is redrawn after the SPI chain by ``pension_credit_take_up``, # whose rates are shares of the entitled units the engine finds; its stage # gate measures them, so the terminal signal gate does not (microcosm#1069 R9). +# would_claim_child_benefit and child_benefit_opts_out are redrawn by +# ``child_benefit_take_up`` against HMRC's claim rates of eligible children and +# opted-out share of registered families; its stage gate measures them +# (microcosm#1063). UK_TAKE_UP_SIGNAL_OUTPUTS = ( - ("benunit", "would_claim_child_benefit", "child_benefit"), - ("benunit", "child_benefit_opts_out", "child_benefit_opts_out_rate"), ("benunit", "would_claim_uc", "universal_credit"), ("benunit", "would_claim_tfc", "tax_free_childcare"), ("benunit", "would_claim_extended_childcare", "extended_childcare"), diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/full_build_cli.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/full_build_cli.py index ff684bde8..ffbd03c4b 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/full_build_cli.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/full_build_cli.py @@ -146,7 +146,11 @@ rowwise_candidate_manifest_from_graph, ) from .national_doctrine import uk_doctrine_with_overrides -from .national_frame import load_uk_national_frame, write_uk_national_frame +from .national_frame import ( + load_uk_national_frame, + uk_release_export_frame, + write_uk_national_frame, +) from .national_sampling import UK_SAMPLE_RUNG_TOKENS from .rowwise_cli import ( MANIFEST_FILENAME, @@ -2091,7 +2095,9 @@ def _execute_national_build( ) # 7. The H5, then the graph's readback of it. stage(telemetry, "candidate_h5_creation", "started") - write_uk_national_frame(frame, paths["dataset"]) + # The release boundary (microcosm#1063 c9): the reviewed export exclusions + # leave the candidate here, and the graph readback expects the same. + write_uk_national_frame(uk_release_export_frame(frame), paths["dataset"]) if state is not None: append_phase(state, "staging_h5_written") stage( diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/full_gates.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/full_gates.py index c9e799c73..692f43089 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/full_gates.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/full_gates.py @@ -20,7 +20,9 @@ from microcosm.build.gate_battery import EvidenceContext from microcosm.build.uk_runtime.calibration_run import uk_aggregate_admin_totals from microcosm.build.uk_runtime.graph_evidence import uk_spine_gate_artifacts +from microcosm.build.uk_runtime.national_frame import uk_release_export_frame from microcosm.build.uk_runtime.parity_reference import load_efrs_parity_reference +from microcosm.build.uk_runtime.terminal_gates import uk_export_candidate_columns from microcosm.calibrate import TargetRegistry from microcosm.calibrate.artifacts import OrderedProblem, OrderedSolution from microcosm.calibrate.solve import CalibrationResult, _build_diagnostics @@ -255,11 +257,9 @@ def build_full_gate_context( "matrix_target_count": len(problem.names), }, "parity_evidence": SimpleNamespace( - candidate_columns={ - f"{entity}.{column}" - for entity in frame.entities - for column in frame.table(entity).columns - }, + candidate_columns=uk_export_candidate_columns( + uk_release_export_frame(frame) + ), reference_columns={ f"{entity}.{name}" for name, entity in reference.input_entities.items() diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph.py index 1ddbc1f59..df857db58 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph.py @@ -52,6 +52,7 @@ "cgt_support_split", "cgt_incidence_clone", "cgt_incidence_anchor", + "cgt_residential_split", } ) @@ -72,11 +73,16 @@ # invariant itself (``UKExpandStageKernel``). _STRUCTURAL_MASS = { "spi_support_channel": "declared", - # Reserved income rows add their published band mass (microcosm#1063 owes - # them the channel treatment). - "spi_income_band_donors": "free", + # The reserved income rows are funded from the incumbent households of + # each donor's region (microcosm#1063): household mass is conserved + # exactly, and person mass moves with the composition of the copied + # households, as it does for the support channel. + "spi_income_band_donors": "declared", "cgt_support_split": "conserve", "cgt_incidence_clone": "conserve", + # The residential split divides each liable household into arms that are + # identical in composition (microcosm#1063), so person mass is conserved. + "cgt_residential_split": "conserve", "cgt_incidence_anchor": "conserve", } @@ -85,6 +91,7 @@ "spi_income_band_donors": "importance", "cgt_support_split": "importance", "cgt_incidence_clone": "importance", + "cgt_residential_split": "importance", "cgt_incidence_anchor": "importance", } @@ -105,6 +112,9 @@ # The Pension Credit redraw rewrites ``frs_take_up``'s would_claim_pc, # which the open-surface readers before it bind (microcosm#1069). "pension_credit_take_up", + # The Child Benefit redraw rewrites ``frs_take_up``'s two Child Benefit + # flags, which the open-surface readers before it bind (microcosm#1063). + "child_benefit_take_up", # ``frs_education_grant_split`` rewrites the root cell # ``education_grants`` that the open-surface ``frs_legacy_proxies`` # reader already bound to. In the root version that rewrite opened a @@ -229,6 +239,9 @@ ("household", "region"), ("household", "main_residence_value"), ("household", "property_wealth"), + ("household", "owned_land"), + ("household", "other_residential_property_value"), + ("household", "non_residential_property_value"), ("household", "household_support_channel"), } ), @@ -269,6 +282,9 @@ # One temporary engine materialization of Pension Credit entitlement over # the whole frame: an open surface, like the UC award screen. "pension_credit_take_up": None, + # One temporary engine materialization of Child Benefit eligibility and + # adjusted net income over the whole frame: an open surface. + "child_benefit_take_up": None, "uc_capital_coherence": frozenset( { ("person", "is_benunit_head"), @@ -286,6 +302,9 @@ # The support split copies whole households, so its surface is open. "cgt_support_split": None, "cgt_incidence_clone": None, + # The residential split copies whole households into arms, so its + # surface is open (microcosm#1063). + "cgt_residential_split": None, # The amounts redraw conditions on age and household region as well as # the income proxy (microcosm#725), ranks gainers on household investable # wealth (microcosm#1014) and keys its placement receipts on the support @@ -640,7 +659,14 @@ def _cells( _Cell("household", "household_local_bus_trips", "float64"), ), "regional_property_uprating": _cells( - "household", ("main_residence_value", "property_wealth") + "household", + ( + "main_residence_value", + "property_wealth", + "owned_land", + "other_residential_property_value", + "non_residential_property_value", + ), ), "lcfs_consumption": ( *_cells( @@ -767,6 +793,10 @@ def _cells( _Cell("benunit", "would_claim_uc", "bool"), ), "pension_credit_take_up": (_Cell("benunit", "would_claim_pc", "bool"),), + "child_benefit_take_up": ( + _Cell("benunit", "would_claim_child_benefit", "bool"), + _Cell("benunit", "child_benefit_opts_out", "bool"), + ), "uc_deduction_attributes": ( _Cell("benunit", "uc_deduction_random_draw", "float64"), _Cell("benunit", "uc_deduction_type_random_draw", "float64"), @@ -785,13 +815,20 @@ def _cells( _Cell("person", "capital_gains", "float64"), ), "hmrc_cgt_gains_spine": (_Cell("person", "capital_gains", "float64"),), + # Weights only: the anchor owns no cell (microcosm#970). + "cgt_incidence_anchor": (), + # The residential split writes its lineage cells, the solved probability + # and the residential gains of each arm (microcosm#1063). + "cgt_residential_split": ( + _Cell("household", "household_is_cgt_residential_clone", "bool"), + _Cell("household", "cgt_residential_clone_index", "int64"), + _Cell("person", "cgt_residential_probability", "float64"), + _Cell("person", "capital_gains_residential_property", "float64"), + ), "hmrc_cgt_asset_type_spine": ( _Cell("person", "capital_gains_asset_type", "string"), - _Cell("person", "capital_gains_residential_property", "float64"), _Cell("person", "capital_gains_badr", "float64"), ), - # Weights only: the anchor owns no cell (microcosm#970). - "cgt_incidence_anchor": (), "salary_sacrifice": _cells( "person", ( @@ -799,6 +836,9 @@ def _cells( "employee_pension_contributions", # Converted records' pay drops by the sacrificed amount (microcosm#1069). "employment_income", + # A converted record's pay before the conversion, zero elsewhere: + # the carrier that makes the rewrite reversible (microcosm#1063). + "salary_sacrifice_pre_conversion_pay", ), ), "student_loans": (_Cell("person", "student_loan_plan", "string"),), @@ -1343,7 +1383,7 @@ def uk_spine_operation_inventory( "artifact_outputs": [output.name for output in node.artifact_outputs], "randomness": "Existing literal/child seeds and draw order are preserved inside the registered transform.", "coupling": ( - "Donor and recipient region encoding, four segmented fit/draw chains and their child seeds remain coupled." + "Donor and recipient region and tenure encoding, seven segmented fit/draw chains with their child seeds, the tenure strata and the totals derived from drawn components remain coupled." if stage.stage == "was_wealth" else "Source assembly, declared household sample selection and same-kind mass normalization execute once in CREATE." if stage.stage == "frs_spine" diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_kernels.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_kernels.py index b8e610d55..ff99c0d30 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_kernels.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_kernels.py @@ -97,11 +97,13 @@ "uc_reporter_redraw": "uc_reporter_redraw", "uc_capital_coherence": "uc_capital_coherence", "pension_credit_take_up": "pension_credit_take_up", + "child_benefit_take_up": "child_benefit_take_up", "uc_deduction_attributes": "uc_deduction_attributes", "cgt_support_split": "cgt_support", "cgt_incidence_clone": "cgt_structure", "cgt_incidence_anchor": "cgt_structure", "hmrc_cgt_gains_spine": "cgt_imputation", + "cgt_residential_split": "cgt_residential_split", "hmrc_cgt_asset_type_spine": "cgt_asset_type", "salary_sacrifice": "salary_sacrifice", "student_loans": "student_loans", @@ -129,6 +131,8 @@ # The Pension Credit redraw reuses the UC stage's household-to-benefit-unit # weight mapping and the adapter's engine materialization. "pension_credit_take_up": (uc_capital_coherence, uk_engine_adapter), + # The Child Benefit redraw shares the weight mapping and the adapter. + "child_benefit_take_up": (uc_capital_coherence, uk_engine_adapter), } _COMPUTE = Capabilities( @@ -400,7 +404,7 @@ def _fixture_descriptor( missing = sorted(set(_STAGE_MODULES) - set(stages)) extra = sorted(set(stages) - set(_STAGE_MODULES)) raise ValueError( - "UK parity fixture must describe the current 36-stage spine " + "UK parity fixture must describe the current 38-stage spine " f"(missing={missing}, extra={extra})." ) return descriptor, stages @@ -414,11 +418,13 @@ def _fixture_implementations(source: Path) -> Mapping[str, object]: from .age_tail import UKAgeTailStageTransform from .cgt_asset_type import UKCGTAssetTypeStageTransform, UKCGTBADRParameters from .cgt_imputation import UKCGTPolicyParameters, uk_cgt_spine_stage_transform + from .cgt_residential_split import UKCGTResidentialSplitStageTransform from .cgt_structure import ( UKCGTIncidenceAnchorStageTransform, UKCGTIncidenceCloneStageTransform, ) from .cgt_support import UKCGTSupportSplitStageTransform + from .child_benefit_take_up import UKChildBenefitTakeUpStageTransform from .etb_services import UKETBServicesStageTransform from .etb_vat import UKETBVATStageTransform from .frs_brma import UKFRSBRMAStageTransform @@ -619,6 +625,9 @@ def _fixture_implementations(source: Path) -> Mapping[str, object]: "pension_credit_take_up": UKPensionCreditTakeUpStageTransform( stage=stages["pension_credit_take_up"], engine=engine ), + "child_benefit_take_up": UKChildBenefitTakeUpStageTransform( + stage=stages["child_benefit_take_up"], engine=engine + ), "uc_deduction_attributes": UKUCDeductionAttributesStageTransform( stage=stages["uc_deduction_attributes"] ), @@ -645,6 +654,11 @@ def _fixture_implementations(source: Path) -> Mapping[str, object]: stage=stages["cgt_incidence_anchor"], parameters=cgt_parameters, ), + "cgt_residential_split": UKCGTResidentialSplitStageTransform( + stage=stages["cgt_residential_split"], + facts=cgt_asset_type_facts, + parameters=cgt_parameters, + ), "salary_sacrifice": UKSalarySacrificeStageTransform( stage=stages["salary_sacrifice"] ), @@ -1045,7 +1059,11 @@ def run(self, context: KernelContext) -> KernelResult: ) cells = _expand_cells(context) id_offset = None - if self.stage in {"cgt_support_split", "cgt_incidence_clone"}: + if self.stage in { + "cgt_support_split", + "cgt_incidence_clone", + "cgt_residential_split", + }: id_offset = id_multiplier_for_values( *( before.table(entity)[before.schema.entity_id_column(entity)] @@ -1172,6 +1190,7 @@ def build_uk_registry( "cgt_support_split", "cgt_incidence_clone", "cgt_incidence_anchor", + "cgt_residential_split", }: registry.register(UKExpandStageKernel(stage, transform, fixture_resolver)) else: diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_national.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_national.py index 3e6041cfc..6365f88b8 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_national.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_national.py @@ -120,7 +120,7 @@ national_calibration_mass_reason, ) from .national_doctrine import UKNationalSolveDoctrine, uk_national_target_loss_weights -from .national_frame import load_uk_national_frame +from .national_frame import load_uk_national_frame, uk_release_export_frame from .rowwise_posture import UK_ROWWISE_NATIONAL_POSTURE, UKRowwisePosture NATIONAL_TARGET_TYPE = ArtifactType("microcosm.uk.national-target-registry", 1) @@ -992,7 +992,9 @@ def run(self, context: KernelContext) -> KernelResult: dataset = file_artifact(path) written, _provenance = load_uk_national_frame(path) failures = [] - expected = uk_frame_content_identity(frame) + # The writer drops the reviewed export exclusions (microcosm#1063 c9), + # so the expectation is the export frame, not the raw population. + expected = uk_frame_content_identity(uk_release_export_frame(frame)) actual = uk_frame_content_identity(written) if actual != expected: failures.append( diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_population.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_population.py index 04191878a..a0a56481d 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_population.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_population.py @@ -81,6 +81,8 @@ "household_is_cgt_support_copy", "cgt_support_copy_index", "household_is_capital_gains_clone", + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", ladder_clone_index_column("household"), ) _BASE_SUPPORT_CHANNEL = "frs" @@ -397,9 +399,14 @@ def run(self, context: KernelContext) -> KernelResult: raise ValueError("Geography identity requires int64 source household ids.") if lineage["cgt_support_copy_index"].dtype != np.dtype("int64"): raise ValueError("Geography identity requires an int64 support copy index.") + if lineage["cgt_residential_clone_index"].dtype != np.dtype("int64"): + raise ValueError( + "Geography identity requires an int64 residential arm index." + ) for column in ( "household_is_cgt_support_copy", "household_is_capital_gains_clone", + "household_is_cgt_residential_clone", ): dtype = lineage[column].dtype if dtype != np.dtype("bool") and not isinstance(dtype, pd.BooleanDtype): @@ -412,6 +419,8 @@ def run(self, context: KernelContext) -> KernelResult: support_copy, copy_index, cgt, + residential_arm, + residential_index, clone_index, ) in zip( lineage["source_household_id"].to_numpy(), @@ -420,12 +429,21 @@ def run(self, context: KernelContext) -> KernelResult: lineage["household_is_cgt_support_copy"].to_numpy(), lineage["cgt_support_copy_index"].to_numpy(), lineage["household_is_capital_gains_clone"].to_numpy(), + lineage["household_is_cgt_residential_clone"].to_numpy(), + lineage["cgt_residential_clone_index"].to_numpy(), lineage[ladder_clone_index_column("household")].to_numpy(), strict=True, ): channel = str(channel) support_index, clone_index = int(support_index), int(clone_index) copy_index = int(copy_index) + residential_index = int(residential_index) + if residential_index < 0: + raise ValueError("Geography identity refuses negative clone indices.") + if bool(residential_arm) != (residential_index != 0): + raise ValueError( + "Household residential-arm flag and arm index disagree." + ) if channel not in {_BASE_SUPPORT_CHANNEL, _SPI_SUPPORT_CHANNEL}: raise ValueError(f"Unknown household support channel {channel!r}.") if support_index < 0 or clone_index < 0 or copy_index < 0: @@ -445,6 +463,8 @@ def run(self, context: KernelContext) -> KernelResult: path.append(("cgt_support_split", copy_index)) if bool(cgt): path.append(("cgt_incidence_clone", 1)) + if residential_index: + path.append(("cgt_residential_split", residential_index)) if clone_index: path.append(("geographic_support", clone_index)) keys.append( diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py index 5df6d210a..75a4bf45f 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/graph_terminal.py @@ -49,6 +49,7 @@ from .geography_ladder import uk_geography_ladder_gate from .graph_population import context_frame, population_columns, population_slices from .national_frame import ( + UK_RELEASE_EXPORT_DROPPED_COLUMNS, _read_uk_national_tables, _write_uk_single_year_tables, uk_household_weight_kind, @@ -85,6 +86,11 @@ def _tables(frame: Frame) -> dict[str, pd.DataFrame]: aliases = [c for c in UK_NATIVE_ALIAS_COLUMNS if c in renamed["household"]] if aliases: renamed["household"] = renamed["household"].drop(columns=aliases) + # The reviewed export exclusions leave at the same boundary (microcosm#1063 c9). + for entity, columns in UK_RELEASE_EXPORT_DROPPED_COLUMNS.items(): + dropped = [c for c in columns if c in renamed[entity]] + if dropped: + renamed[entity] = renamed[entity].drop(columns=dropped) return renamed diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/hmrc_uprating.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/hmrc_uprating.py index 08cd57454..09d275957 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/hmrc_uprating.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/hmrc_uprating.py @@ -586,7 +586,14 @@ def align_hmrc_row_by_itl_growth( ``ledger_targets.CALENDAR_YEAR_WINDOW_WEIGHTS`` and the measure is summed over the Table 2.5 bands the SPI band spans. Counts follow taxpayer numbers; total income and tax rows (the regional Table 3.11 anchors) - follow HMRC's projected total income and liabilities in the band. + follow HMRC's projected total income and liabilities in the band, and so + do the Table 3.6 employment, self-employment and private-pension amounts + by band (microcosm#1063, 2026-10-02): an amount held in a fixed nominal + band moves with the band's membership, which a flat national earnings or + pension index ignores. Under the flat index the GBP 15,000-20,000 band's + employment amount rose 10.7% from 2023-24 to 2025 while HMRC projects the + band's total income to fall 2.3%; the SPI tape re-banded under the + engine's own indices agrees with HMRC's direction. """ if reference.uprating_index != index_concept: diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/lcfs_consumption.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/lcfs_consumption.py index 475c0c1a7..1ee67b6c1 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/lcfs_consumption.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/lcfs_consumption.py @@ -2,6 +2,7 @@ from __future__ import annotations +import math from collections.abc import Mapping, Sequence from dataclasses import dataclass, field from pathlib import Path @@ -256,6 +257,7 @@ class UKLCFSConsumptionResult: energy_pricing: Mapping[str, Any] | None = None energy_rake: Mapping[str, Any] | None = None fuel_litres_audit: Mapping[str, Any] | None = None + donor_floor: Mapping[str, Any] | None = None def evidence(self) -> dict[str, object]: evidence: dict[str, object] = { @@ -274,6 +276,8 @@ def evidence(self) -> dict[str, object]: evidence["energy_rake"] = dict(self.energy_rake) if self.fuel_litres_audit is not None: evidence["fuel_litres_audit"] = dict(self.fuel_litres_audit) + if self.donor_floor is not None: + evidence["donor_floor"] = dict(self.donor_floor) return evidence @@ -333,6 +337,9 @@ def __call__(self, frame: Frame) -> Frame: energy=energy, donor_rake_iterations=_donor_energy_rake_iterations(self.stage), ) + donor, donor_floor_receipt = floor_negative_donor_consumption( + donor, floor=_donor_consumption_floor(self.stage) + ) ice_share, ice_share_receipt = lcfs_ice_share(self.stage) recipient = recipient_predictors(frame, self.engine) recipient["has_fuel_consumption"] = assign_recipient_has_fuel( @@ -415,6 +422,7 @@ def __call__(self, frame: Frame) -> Frame: energy_pricing=None if energy is None else energy.receipt, energy_rake=energy_rake_receipt, fuel_litres_audit=litres_audit, + donor_floor=donor_floor_receipt, ) return result @@ -1048,6 +1056,69 @@ def clean_lcfs_consumption_table( ].dropna() +def _donor_consumption_floor(stage: SourceStageSpec) -> float: + """The declared floor of the donor's diary consumption columns.""" + + parameters = _operation_parameters(stage, "derive") + floor = parameters.get("floor") + if floor is None: + raise ValueError( + f"{stage.stage}: the derive operation must declare 'floor', the " + "value negative diary consumption is raised to on the donor." + ) + value = float(floor) + if not math.isfinite(value) or value != 0.0: + raise ValueError( + f"{stage.stage}: the declared donor consumption floor must be 0, " + f"got {floor!r}." + ) + return value + + +def floor_negative_donor_consumption( + donor: pd.DataFrame, *, floor: float = 0.0 +) -> tuple[pd.DataFrame, dict[str, Any]]: + """Raise negative diary consumption on the donor to ``floor`` (microcosm#1063 c9). + + An LCFS diary spend nets refunds against purchases, so a handful of donor + rows carry a negative annual total (250 release rows inherited a minimum + of -14,165 in ``housing_water_and_electricity_consumption`` on the + 2026-09-30 build, because the support clip takes its floor from the + donor's realised range). The release surface declares these columns + non-negative, so the donor is floored before the imputation, the clip + ranges and the rake see it. The receipt records, per consumption column, + the rows raised and the (negative) mass they carried; columns without a + negative row are listed with zeros so the receipt names the whole surface. + """ + + floored = donor.copy() + columns: dict[str, dict[str, float | int]] = {} + for column in CONSUMPTION_VARIABLE_RENAMES.values(): + if column not in floored.columns: + raise ValueError(f"LCFS donor is missing consumption column {column!r}.") + values = _numeric(floored[column]).to_numpy(dtype=float) + negative = np.isfinite(values) & (values < floor) + columns[column] = { + "rows_raised": int(negative.sum()), + "negative_mass": float(values[negative].sum()), + "minimum_before": float(np.nanmin(values)) if values.size else 0.0, + } + if negative.any(): + floored[column] = np.where(negative, floor, values) + receipt = { + "floor": float(floor), + "columns": columns, + "rows_raised": int(sum(entry["rows_raised"] for entry in columns.values())), + "remaining_negative_rows": int( + sum( + int((_numeric(floored[column]).to_numpy(dtype=float) < floor).sum()) + for column in columns + ) + ), + } + return floored, receipt + + def derive_energy_from_lcfs(household: pd.DataFrame) -> pd.DataFrame: """Split LCFS domestic energy into electricity and gas weekly amounts.""" diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/local_targets.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/local_targets.py index 5f3227b5c..cb93ecf0d 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/local_targets.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/local_targets.py @@ -17,6 +17,14 @@ import numpy as np import pandas as pd +from microcosm.build.uk_runtime.tenure_constants import ( + UK_TENURE_OWNED_OUTRIGHT, + UK_TENURE_OWNED_WITH_MORTGAGE, + UK_TENURE_PRIVATE_RENT, + UK_TENURE_SOCIAL_RENT, + UK_TENURE_TYPE_TO_CATEGORY, +) + AREA_TYPES = ("constituency", "la") AREA_TYPE_TO_LEDGER_GEOGRAPHY_LEVEL = { "constituency": "constituency", @@ -45,6 +53,13 @@ #: the engine enum, so it binds the Welsh cells that publish it. Appended #: last, so the positional metric indices of A-H are unchanged. COUNCIL_TAX_BANDS = tuple("ABCDEFGHI") +#: Local tenure metric -> the shared tenure category it counts, in matrix order. +_TENURE_METRIC_CATEGORIES = { + "tenure/owned_outright": UK_TENURE_OWNED_OUTRIGHT, + "tenure/owned_mortgage": UK_TENURE_OWNED_WITH_MORTGAGE, + "tenure/private_rent": UK_TENURE_PRIVATE_RENT, + "tenure/social_rent": UK_TENURE_SOCIAL_RENT, +} COUNTRY_TO_REGION = { "England": "SOUTH_EAST", "Scotland": "SCOTLAND", @@ -310,16 +325,11 @@ def compute_household_metrics( matrix["ons/equiv_housing_costs"] = housing_costs tenure_type = _values(_calculate(sim, "tenure_type", period)) - matrix["tenure/owned_outright"] = (tenure_type == "OWNED_OUTRIGHT").astype( - float - ) - matrix["tenure/owned_mortgage"] = (tenure_type == "OWNED_WITH_MORTGAGE").astype( - float + tenure_category = ( + pd.Series(tenure_type).map(UK_TENURE_TYPE_TO_CATEGORY).to_numpy() ) - matrix["tenure/private_rent"] = (tenure_type == "RENT_PRIVATELY").astype(float) - matrix["tenure/social_rent"] = ( - (tenure_type == "RENT_FROM_COUNCIL") | (tenure_type == "RENT_FROM_HA") - ).astype(float) + for metric, category in _TENURE_METRIC_CATEGORIES.items(): + matrix[metric] = (tenure_category == category).astype(float) is_private_renter = (tenure_type == "RENT_PRIVATELY").astype(float) benunit_rent = _values(_calculate(sim, "benunit_rent", period)) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_frame.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_frame.py index 2dd1348dc..d068dc835 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_frame.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_frame.py @@ -26,7 +26,7 @@ import json import uuid -from collections.abc import Callable, Sequence +from collections.abc import Callable, Mapping, Sequence from dataclasses import dataclass from pathlib import Path from typing import Any @@ -60,6 +60,7 @@ "uk_household_mass_conservation_receipt", "uk_time_period", "validate_uk_national_frame", + "uk_release_export_frame", "write_uk_national_frame", ] @@ -453,6 +454,61 @@ def _write_uk_single_year_tables( return path +#: Columns the release boundary drops from the written candidate (microcosm#1063 +#: c9): the enhanced FRS stores ``incapacity_benefit_reported`` as an all-zero +#: legacy layer and the export-surface gate's reviewed exclusion requires the +#: candidate to drop it. The spine keeps the column (the SPI income stage +#: reads it); only the release H5 and its readback expectation lose it. +#: ``terminal_gates.UK_REVIEWED_EXPORT_EXCLUSIONS`` names the first; the +#: five internal disability carriers (``frs_disability``'s +#: ``UK_INTERNAL_DISABILITY_REPORTED_COLUMNS``) are stage-internal inputs +#: that the export registers keep off the release (review of #1089). +UK_RELEASE_EXPORT_DROPPED_COLUMNS: Mapping[str, tuple[str, ...]] = { + "person": ( + "incapacity_benefit_reported", + "attendance_allowance_reported", + "dla_sc_reported", + "dla_m_reported", + "pip_m_reported", + "pip_dl_reported", + # The salary-sacrifice conversion's pre-conversion pay carrier + # (salary_sacrifice.SALSAC_PRE_CONVERSION_PAY_COLUMN): it exists so + # the identity receipts can reverse the stage's in-place rewrite of a + # converted record's pay, and is no engine input. + "salary_sacrifice_pre_conversion_pay", + ), +} + + +def uk_release_export_frame(frame: Frame) -> Frame: + """The frame the release boundary writes: reviewed export exclusions dropped.""" + + tables: dict[str, pd.DataFrame] = {} + changed = False + for entity in ("person", "benunit", "household"): + table = frame.table(entity) + dropped = [ + column + for column in UK_RELEASE_EXPORT_DROPPED_COLUMNS.get(entity, ()) + if column in table.columns + ] + if dropped: + table = table.drop(columns=dropped) + changed = True + tables[entity] = table + if not changed: + return frame + return uk_national_frame( + person=tables["person"], + benunit=tables["benunit"], + household=tables["household"], + time_period=uk_time_period(frame), + weight_kind=uk_household_weight_kind(frame), + household_weights=frame.weights_for("household").values, + mass_log=frame.mass_log, + ) + + def write_uk_national_frame(frame: Frame, path: str | Path) -> Path: """Atomically write a validated UK national Frame as a staging H5.""" diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_sampling.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_sampling.py index 9232a2a33..1b130f25d 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_sampling.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_sampling.py @@ -129,6 +129,7 @@ def _str_column(values: pd.Series, *, label: str) -> np.ndarray: _SPINE_SOURCE_HOUSEHOLD_ID_COLUMN = "source_household_id" _SPINE_SUPPORT_CLONE_INDEX_COLUMN = "household_support_clone_index" _SPINE_CGT_SUPPORT_COPY_COLUMN = "household_is_cgt_support_copy" +_SPINE_CGT_RESIDENTIAL_ARM_COLUMN = "household_is_cgt_residential_clone" def uk_spine_source_family_units(frame: Frame) -> tuple[np.ndarray, np.ndarray]: @@ -148,6 +149,7 @@ def uk_spine_source_family_units(frame: Frame) -> tuple[np.ndarray, np.ndarray]: HOUSEHOLD_IS_SPI_SYNTHETIC_COLUMN, _CAPITAL_GAINS_CLONE_COLUMN, _SPINE_CGT_SUPPORT_COPY_COLUMN, + _SPINE_CGT_RESIDENTIAL_ARM_COLUMN, } required = {household_id_column, _REGION_COLUMN, *lineage_columns} missing = sorted(required - set(household.columns)) @@ -200,9 +202,17 @@ def uk_spine_source_family_units(frame: Frame) -> tuple[np.ndarray, np.ndarray]: household[_SPINE_CGT_SUPPORT_COPY_COLUMN], label=_SPINE_CGT_SUPPORT_COPY_COLUMN, ) + # A residential split's incumbent arm keeps its raw-row role and ids; + # the residential arms join the family the same way (microcosm#1063). + residential_arm = _strict_bool( + household[_SPINE_CGT_RESIDENTIAL_ARM_COLUMN], + label=_SPINE_CGT_RESIDENTIAL_ARM_COLUMN, + ) regions = _str_column(household[_REGION_COLUMN], label=_REGION_COLUMN) - raw = (support_clone == 0) & ~spi & ~capital_gains & ~support_copy + raw = ( + (support_clone == 0) & ~spi & ~capital_gains & ~support_copy & ~residential_arm + ) source_rows = raw & (household_ids == units) source_keys = set(int(value) for value in units[source_rows]) missing_sources = sorted(set(int(value) for value in units) - source_keys) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/regional_uprating.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/regional_uprating.py index b9164af8e..b52c752be 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/regional_uprating.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/regional_uprating.py @@ -25,7 +25,19 @@ ) from microcosm.frame import Frame -UK_REGIONAL_PROPERTY_REWRITES = ("main_residence_value", "property_wealth") +#: The property total and the components of it the frame carries. The factor +#: is read off the main residence and applied to the total, so it is applied to +#: every component too: the total stays the sum of its parts (microcosm#1063). +UK_REGIONAL_PROPERTY_COMPONENT_REWRITES = ( + "owned_land", + "other_residential_property_value", + "non_residential_property_value", +) +UK_REGIONAL_PROPERTY_REWRITES = ( + "main_residence_value", + "property_wealth", + *UK_REGIONAL_PROPERTY_COMPONENT_REWRITES, +) #: The household-mass receipt this stage records (the manifest's @@ -88,7 +100,9 @@ def uprate_household_property_by_region( The factor for a region is its public mean house price over the unweighted mean ``main_residence_value`` of FRS-base owners. When the frame carries SPI support rows, which are imputed from higher incomes, they are uprated - by the same factor but do not enter the mean. + by the same factor but do not enter the mean. The factor scales the + property total and every component of it the table carries, so an owner's + ``property_wealth`` stays the sum of its components after the uprating. """ required = {"region", "main_residence_value", "property_wealth"} @@ -130,6 +144,7 @@ def uprate_household_property_by_region( if imputed_mean <= 0: continue factor = hpi_price / imputed_mean - result.loc[owners_mask, "main_residence_value"] *= factor - result.loc[owners_mask, "property_wealth"] *= factor + for column in UK_REGIONAL_PROPERTY_REWRITES: + if column in result.columns: + result.loc[owners_mask, column] *= factor return result diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/release_certification.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/release_certification.py index 537e698d9..c1160980c 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/release_certification.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/release_certification.py @@ -40,10 +40,10 @@ BlockingMode, EvidenceContext, GateBatteryRun, + canonical_report_bytes, gate_signing_key_env, ) from microcosm.build.gates import FitWeightRecord -from microcosm.build.logbook import canonical_json_bytes from microcosm.build.uk_runtime.battery_bindings import UK_GATE_REGISTRY from microcosm.build.uk_runtime.calibration_run import ( UK_CALIBRATION_GATE_SCOPE, @@ -59,6 +59,7 @@ UK_CGT_PROJECTION_ENGINE_LABEL_PREFIX, uk_cgt_projection_read_from_engine, ) +from microcosm.build.uk_runtime.terminal_gates import uk_export_candidate_columns __all__ = [ "UK_RELEASE_CERTIFICATION_KIND", @@ -244,11 +245,7 @@ def uk_release_parity_evidence( ) target_relative_errors[name] = float(row["relative_error"]) return SimpleNamespace( - candidate_columns={ - f"{entity}.{column}" - for entity in frame.entities - for column in frame.table(entity).columns - }, + candidate_columns=uk_export_candidate_columns(frame), reference_columns={ f"{entity}.{name}" for name, entity in parity_reference.input_entities.items() @@ -638,8 +635,13 @@ def _verify_part_signature( raise UKReleaseCertificationError(f"{part_name}: signature absent.") unsigned = json.loads(json.dumps(payload)) unsigned["attestation"]["signature"] = None + # The parts are gate-battery reports, signed over the battery's + # canonical bytes; a verifier that canonicalised them any other way + # would refuse every real report (the logbook form renders 1.0 as 1, + # so the two agree only on reports without such floats, which is why + # synthetic fixtures never showed the difference; microcosm#1063). recomputed = hmac.new( - signing_key, canonical_json_bytes(unsigned), hashlib.sha256 + signing_key, canonical_report_bytes(unsigned), hashlib.sha256 ).hexdigest() if not hmac.compare_digest(recomputed, signature): raise UKReleaseCertificationError( @@ -996,7 +998,7 @@ def _sign_certification(payload: dict[str, Any], signing_key: bytes) -> None: } payload["attestation"] = attestation attestation["signature"] = hmac.new( - signing_key, canonical_json_bytes(payload), hashlib.sha256 + signing_key, canonical_report_bytes(payload), hashlib.sha256 ).hexdigest() diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/release_input_coverage.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/release_input_coverage.py index 90c70456c..d3fd996d5 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/release_input_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/release_input_coverage.py @@ -41,6 +41,7 @@ __all__ = [ "RESTORED_REFERENCE_EFRS_REQUIRED_INPUTS", "UKEffectiveMassCoveragePolicy", + "UK_FAMILY_MASS_CHANGE_SEMANTICS", "UK_LOADER_INPUT_ALIASES", "UK_RELEASE_INPUT_COVERAGE_RESOURCE", "PolicyEngineUKCoverageEngine", @@ -70,6 +71,9 @@ {_REQUIRED_AT_BUILD_STATUS, _DEFERRED_UNTIL_RESTORED_STATUS} ) _DISTRIBUTIONAL_REQUIRED_STATUS = "distributional_required" +#: The one mass-change semantics a stage family may declare: its household +#: mass record conserves the total and declares factor one. +UK_FAMILY_MASS_CHANGE_SEMANTICS = "mass_conserving" _UK_PACKAGE = "microcosm.build.uk" _EFRS_PARITY_REFERENCE_RESOURCE = "efrs_parity_reference.json" @@ -342,15 +346,17 @@ def _parse_family_coverage( "required_mass_change_reason." ) mass_change_semantics = str( - raw_family.get("mass_change_semantics", "mass_conserving") + raw_family.get("mass_change_semantics", UK_FAMILY_MASS_CHANGE_SEMANTICS) ).strip() - if mass_change_semantics not in { - "mass_conserving", - "mass_increasing_support", - }: + # Every spine stage conserves household mass: the last stage that + # added support mass (the SPI income band donors) became a funded + # channel in microcosm#1063, and ``mass_increasing_support`` retired + # with it. + if mass_change_semantics != UK_FAMILY_MASS_CHANGE_SEMANTICS: raise ValueError( f"{resource}: family {name!r} has invalid " - f"mass_change_semantics {mass_change_semantics!r}." + f"mass_change_semantics {mass_change_semantics!r}; the only " + f"semantics is {UK_FAMILY_MASS_CHANGE_SEMANTICS!r}." ) raw_requirements = raw_family.get("effective_mass_requirements", {}) @@ -993,18 +999,16 @@ def _family_build_state_diagnostics( required_reason = str(family.get("required_mass_change_reason", "")).strip() if required_reason: - semantics = str(family.get("mass_change_semantics", "mass_conserving")) + semantics = str( + family.get("mass_change_semantics", UK_FAMILY_MASS_CHANGE_SEMANTICS) + ) records = tuple(getattr(frame, "mass_log", ())) matches = [ record for record in records if _mass_record_field(record, "reason") == required_reason ] - valid_matches = [ - record - for record in matches - if _valid_mass_record(record, semantics=semantics) - ] + valid_matches = [record for record in matches if _valid_mass_record(record)] details["required_mass_change_reason"] = required_reason details["mass_change_semantics"] = semantics details["matching_mass_change_records"] = len(matches) @@ -1027,7 +1031,9 @@ def _mass_record_field(record: object, name: str) -> object: return getattr(record, name, None) -def _valid_mass_record(record: object, *, semantics: str) -> bool: +def _valid_mass_record(record: object) -> bool: + """A household record that conserves mass and declares factor one.""" + old_total = _mass_record_field(record, "old_total") new_total = _mass_record_field(record, "new_total") declared_factor = _mass_record_field(record, "declared_factor") @@ -1044,8 +1050,6 @@ def _valid_mass_record(record: object, *, semantics: str) -> bool: ) if not common: return False - if semantics == "mass_increasing_support": - return bool(new > old and declared_factor is None) try: factor = float(declared_factor) except (TypeError, ValueError): diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_dataset.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_dataset.py index d5811307d..7713bdc76 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_dataset.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_dataset.py @@ -66,6 +66,8 @@ "household_is_capital_gains_clone", "household_is_cgt_support_copy", "cgt_support_copy_index", + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", ) PERSON_ID_COLUMNS = ( diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/salary_sacrifice.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/salary_sacrifice.py index 555e4d1bc..e31364c5d 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/salary_sacrifice.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/salary_sacrifice.py @@ -32,6 +32,15 @@ SALSAC_PREDICTORS = ("age", "employment_income") SALSAC_OUTPUT = "pension_contributions_via_salary_sacrifice" +#: A converted record's employment income before the conversion, zero for +#: everyone the stage does not convert (microcosm#1063). The conversion +#: rewrites a converted record's pay and employee contribution in place after +#: the capital-gains stages have banded on them, which left the income those +#: stages saw unreconstructible from the artifact (the E8 identity receipt on +#: the 2 October spine). The carrier makes the rewrite reversible; it is an +#: internal column and leaves at the release boundary +#: (``UK_RELEASE_EXPORT_DROPPED_COLUMNS``). +SALSAC_PRE_CONVERSION_PAY_COLUMN = "salary_sacrifice_pre_conversion_pay" SALSAC_STAGE_TARGET = 5_400_000.0 SALSAC_HMRC_ANCHOR = 7_700_000.0 SALSAC_ABOVE_2000_ANCHOR = 3_300_000.0 @@ -42,6 +51,14 @@ SALSAC_QRF_ESTIMATORS = 100 SALSAC_CONVERSION_SEED = 2024 SALSAC_CONVERSION_SALT = "salary_sacrifice_conversion" +#: Whom the QRF predicts for. Salary sacrifice is pay given up for a pension +#: contribution, so a person not asked the question who has no employment +#: income has none: the prediction covers the people not asked who are paid, +#: and everyone else not asked is zero (microcosm#1063). +SALSAC_TARGET_POPULATION = ( + "salary_sacrifice_asked != 1 and employment_income > 0, frame-wide; people " + "not asked who have no employment income are set to zero" +) SALSAC_MASS_CHANGE_REASON = ( "Salary-sacrifice support stage rewrites pension columns and converted " "records' pay; household rows and typed household weights pass through and " @@ -66,6 +83,8 @@ class UKSalarySacrificeResult: frame: Frame training_rows: int prediction_rows: int + not_asked_without_pay_rows: int + not_asked_without_pay_mass: float pre_headcount: float post_headcount: float shortfall: float @@ -84,6 +103,11 @@ def evidence(self) -> dict[str, object]: "qrf": { "training_rows": self.training_rows, "prediction_rows": self.prediction_rows, + "target_population": SALSAC_TARGET_POPULATION, + # People not asked who have no employment income: set to + # zero, never predicted for. + "not_asked_without_pay_rows": self.not_asked_without_pay_rows, + "not_asked_without_pay_mass": self.not_asked_without_pay_mass, "seed": SALSAC_QRF_SEED, }, "headcount_receipt": { @@ -100,6 +124,7 @@ def evidence(self) -> dict[str, object]: "moved_amount": self.moved_amount, "expected_converted_mass": self.expected_converted_mass, "realization_deviation": self.realization_deviation, + "pre_conversion_pay_column": SALSAC_PRE_CONVERSION_PAY_COLUMN, }, } @@ -121,7 +146,12 @@ def __call__(self, frame: Frame) -> Frame: @staticmethod def output_columns() -> tuple[str, ...]: - return (SALSAC_OUTPUT, "employee_pension_contributions", "employment_income") + return ( + SALSAC_OUTPUT, + "employee_pension_contributions", + "employment_income", + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) def checkpoint_metadata(self) -> dict[str, object]: if self.last_result is None: @@ -177,12 +207,18 @@ def impute_salary_sacrifice(frame: Frame) -> UKSalarySacrificeResult: ) if not training_mask.any(): raise ValueError("Salary-sacrifice QRF has no eligible asked FRS rows.") - predict_mask = ~asked.eq(1) numeric = person.loc[:, [*SALSAC_PREDICTORS, SALSAC_OUTPUT]].apply( pd.to_numeric, errors="coerce" ) if not np.isfinite(numeric.to_numpy(dtype=float)).all(): raise ValueError("Salary-sacrifice QRF columns must be finite numeric values.") + # The prediction population is the people not asked who are paid; a + # person not asked with no employment income sacrifices none of it. + not_asked = ~asked.eq(1) + paid = numeric["employment_income"].gt(0.0) + predict_mask = not_asked & paid + without_pay = not_asked & ~paid + person.loc[without_pay, SALSAC_OUTPUT] = 0.0 household_weights = pd.Series( frame.weights_for("household").values, index=household["household_id"], @@ -238,6 +274,11 @@ def impute_salary_sacrifice(frame: Frame) -> UKSalarySacrificeResult: # becomes salary sacrifice. Without this the converted contribution left # taxable pay and NI unchanged while losing its net-pay relief # (microcosm#1069 c9). + # The converted record's pay before the conversion stays on its carrier + # (zero elsewhere), so the rewrite is reversible for the receipts of the + # stages that banded on it. + pre_conversion_pay = np.zeros(len(person), dtype=float) + pre_conversion_pay[converted] = employment_income[converted] employment_income = employment_income.copy() employment_income[converted] = np.maximum( employment_income[converted] - employee[converted], 0.0 @@ -246,6 +287,7 @@ def impute_salary_sacrifice(frame: Frame) -> UKSalarySacrificeResult: person[SALSAC_OUTPUT] = final_ss person["employee_pension_contributions"] = employee person["employment_income"] = employment_income + person[SALSAC_PRE_CONVERSION_PAY_COLUMN] = pre_conversion_pay post_headcount = float(person_weights[final_ss > 0.0].sum()) converted_mass = float(person_weights[converted].sum()) total = frame.weights_for("household").total @@ -270,6 +312,8 @@ def impute_salary_sacrifice(frame: Frame) -> UKSalarySacrificeResult: frame=result_frame, training_rows=int(training_mask.sum()), prediction_rows=int(predict_mask.sum()), + not_asked_without_pay_rows=int(without_pay.sum()), + not_asked_without_pay_mass=float(person_weights[without_pay.to_numpy()].sum()), pre_headcount=pre_headcount, post_headcount=post_headcount, shortfall=shortfall, @@ -311,7 +355,7 @@ def _assert_salary_sacrifice_stage_parameters( "support_channel == frs and not capital-gains clone and " "not CGT support copy and salary_sacrifice_asked == 1" ), - "target_population": "salary_sacrifice_asked != 1 frame-wide", + "target_population": SALSAC_TARGET_POPULATION, "predictors": list(SALSAC_PREDICTORS), "targets": [SALSAC_OUTPUT], "weights": "household_weight", @@ -337,7 +381,9 @@ def _assert_salary_sacrifice_stage_parameters( "move": ( "full employee_pension_contributions to " "pension_contributions_via_salary_sacrifice; source zeroed; " - "employment_income lowered by the amount sacrificed" + "employment_income lowered by the amount sacrificed; the " + "converted record's pay before the conversion kept on " + "salary_sacrifice_pre_conversion_pay (zero elsewhere)" ), "seed": SALSAC_CONVERSION_SEED, "salt": SALSAC_CONVERSION_SALT, diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/source_runtime.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/source_runtime.py index a03dc21e7..b9c1d8df1 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/source_runtime.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/source_runtime.py @@ -76,6 +76,7 @@ def uk_stage_implementations( hmrc_cgt_gains_spine_transform: Callable[[Frame], Frame] | None = None, hmrc_cgt_asset_type_spine_transform: Callable[[Frame], Frame] | None = None, cgt_incidence_anchor_transform: Callable[[Frame], Frame] | None = None, + cgt_residential_split_transform: Callable[[Frame], Frame] | None = None, salary_sacrifice_transform: Callable[[Frame], Frame] | None = None, student_loans_transform: Callable[[Frame], Frame] | None = None, ) -> dict[str, Callable[[Frame], Frame]]: @@ -107,8 +108,9 @@ def uk_stage_implementations( "cgt_support_split": cgt_support_split_transform, "cgt_incidence_clone": cgt_incidence_clone_transform, "hmrc_cgt_gains_spine": hmrc_cgt_gains_spine_transform, - "hmrc_cgt_asset_type_spine": hmrc_cgt_asset_type_spine_transform, "cgt_incidence_anchor": cgt_incidence_anchor_transform, + "cgt_residential_split": cgt_residential_split_transform, + "hmrc_cgt_asset_type_spine": hmrc_cgt_asset_type_spine_transform, "salary_sacrifice": salary_sacrifice_transform, "student_loans": student_loans_transform, } diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_band_donors.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_band_donors.py index 5e8826119..10c4ffae4 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_band_donors.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_band_donors.py @@ -10,13 +10,23 @@ way the retired ``cgt_band_donors`` stage reserved rows for the HMRC gain bands (``cgt_support_split`` replaced it in microcosm#1045): whole FRS households copied into the SPI channel, one carrier adult each, drawn by the -tape's own propensity for the band given region, sex and age, at the band's -published taxpayer count over the donors per band. The income stage then -gives each carrier a band-conditional draw from the tape (PolicyEngine/chronicle#280 lane). +tape's own propensity for the band given region, sex and age. The income stage +then gives each carrier a band-conditional draw from the tape +(PolicyEngine/chronicle#280 lane). + +The donors are a support channel, not created mass (microcosm#1063). Each band +seats enough donors that none weighs more than a declared maximum, at equal +weights of the band's published taxpayers over its count, and the mass they +carry is taken from the households of the funding stratum each donor was drawn +from, in proportion, exactly as the SPI support channel takes its copies' mass +from the households of their region. Total household mass and every funding +stratum's mass are conserved; the seats are drawn from identity-keyed uniforms, +so the draw can be reconstructed from the ids. """ from __future__ import annotations +import math from collections.abc import Mapping, Sequence from dataclasses import dataclass, field from pathlib import Path @@ -25,6 +35,7 @@ import pandas as pd from microcosm.build.source_manifest import SourceOperationSpec, SourceStageSpec +from microcosm.build.stochastic_assignment import stable_identity_uniforms from microcosm.build.uk_runtime.ledger_fact_vendoring import vendored_rows from microcosm.build.uk_runtime.national_frame import ( uk_national_frame, @@ -34,7 +45,9 @@ from microcosm.build.uk_runtime.rowwise_geography import id_multiplier_for_values from microcosm.build.uk_runtime.spi_income import ( SPI_MINIMUM_RECIPIENT_AGE, + SPIDonorAgeModel, VerifiedSPIDonorIdentity, + load_spi_donor_age_model, prepare_spi_donor_table, verify_spi_donor_identity, ) @@ -46,6 +59,7 @@ HOUSEHOLD_IS_SPI_SYNTHETIC_COLUMN, SPI_SYNTHETIC_SUPPORT_CHANNEL, _clone_support_frame, + _exact_total_correction, support_channel_column, ) from microcosm.frame import Frame, MassChangeRecord, WeightKind, engine_tables @@ -59,11 +73,33 @@ 1_000_000, 2_000_000, ) -SPI_INCOME_BAND_DONORS_PER_BAND = 120 -SPI_INCOME_BAND_DONOR_COUNT = SPI_INCOME_BAND_DONORS_PER_BAND * len( - SPI_INCOME_BAND_DONOR_LOWER_BOUNDS -) +#: Every band seats at least this many donor households, and enough of them +#: that no donor starts above the maximum weight: the count is +#: ``max(minimum, ceil(published taxpayers / maximum weight))`` and the weight +#: the published taxpayers over the count, equal within the band. +SPI_INCOME_BAND_MINIMUM_DONORS = 120 +SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT = 250.0 +#: Small frames: donors may take at most this share of the eligible base +#: households (the seating scale), and of any funding stratum's mass (the mass +#: scale). Both scales are one on the full frame, where the published band +#: masses are met exactly. +SPI_INCOME_BAND_MAXIMUM_DONOR_HOUSEHOLD_SHARE = 0.15 +SPI_INCOME_BAND_MAXIMUM_REALLOCATION_SHARE = 0.5 +#: The stratum a donor's mass is taken from: the households of its source +#: household's region, base and synthetic channel alike. +SPI_INCOME_BAND_FUNDING_STRATA: tuple[str, ...] = ("region",) SPI_INCOME_BAND_DONOR_SEED = 3 +#: The tape is prepared exactly as ``hmrc_spi_income_spine`` prepares it (its +#: stage-1 seed and its ONS age draw), so the propensity table and the income +#: stage's band pools read one set of donor ages. It matters for the records +#: the tape publishes no age band for (the composites) and for State Pension +#: recipients, whom the age draw keeps at or above State Pension age. +SPI_INCOME_BAND_DONOR_TAPE_SEED = 42 +SPI_INCOME_BAND_DONOR_TAPE_PREPARATION = ( + "as hmrc_spi_income_spine prepares the tape: its stage-1 seed and " + "draw_spi_donor_ages_by_population" +) +SPI_INCOME_BAND_DONOR_DRAW_SALT = "spi_income_band_donor_draw" #: The support channel copies at clone index 1; the reserved copies sit at 2. SPI_INCOME_BAND_DONOR_CLONE_INDEX = 2 HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR = "household_is_spi_income_band_donor" @@ -76,12 +112,36 @@ SPI_INCOME_BAND_DONOR_PROPENSITY = ( "SPI 2022-23 FACT-weighted band share by region, sex and age band" ) -SPI_INCOME_BAND_DONOR_INITIAL_WEIGHT = "published band taxpayers / donors_per_band" +SPI_INCOME_BAND_DONOR_COUNT_RULE = ( + "max(minimum_donors_per_band, ceil(published band taxpayers / " + "maximum_donor_weight))" +) +SPI_INCOME_BAND_DONOR_WEIGHT_RULE = ( + "published band taxpayers / band donor count, equal within a band" +) +SPI_INCOME_BAND_DONOR_DRAW = ( + "weighted_without_replacement_identity_keyed: Efraimidis-Spirakis keys " + "log(u) / (household weight x propensity) on stable_identity_uniforms(" + "person_id, seed, salt:band); the largest keys are seated, one carrier per " + "household, the highest band first" +) +SPI_INCOME_BAND_DONOR_FUNDING = ( + "stratum reallocation: every incumbent household of the stratum a donor's " + "source household belongs to is scaled by 1 - donor mass / stratum mass" +) +SPI_INCOME_BAND_DONOR_SEATING_SCALE = ( + "min(1, maximum_donor_household_share x eligible base households / expected " + "donor count); band counts floored, at least one per band" +) +SPI_INCOME_BAND_DONOR_MASS_SCALE = ( + "min(1, min over funding strata of maximum_reallocation_share x stratum " + "mass / seated donor mass); every band's weight scaled alike" +) SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON = ( - "SPI income band donors stack positive-weight support households per HMRC " - "Table 2.5 total-income band from GBP 200,000, one carrier adult each; the " - "published band taxpayer mass is added explicitly and the base and " - "synthetic channels keep their weights." + "Reallocate the published HMRC Table 2.5 taxpayer mass of each total-income " + "band from GBP 200,000 from the households of each donor's region to its " + "income band donor support rows; every region's household mass and the " + "total national mass are conserved." ) #: Age bands for the propensity table: the tape's own AGERANGE cut points, #: with the open 75-and-over band absorbing the oldest respondents. @@ -107,7 +167,21 @@ class UKSPIIncomeBandDonorResult: frame: Frame band_rows: tuple[Mapping[str, object], ...] - donors_per_band: int + funding_rows: tuple[Mapping[str, object], ...] + minimum_donors_per_band: int + maximum_donor_weight: float + maximum_donor_household_share: float + maximum_reallocation_share: float + seating_scale: float + mass_scale: float + eligible_households: int + expected_donor_count: int + donor_count: int + reallocated_mass: float + old_total: float + new_total: float + seed: int + sample_fraction: float taxpayer_period: int carrier_region_counts: Mapping[str, int] propensity_cells: int @@ -115,16 +189,93 @@ class UKSPIIncomeBandDonorResult: def evidence(self) -> dict[str, object]: return { "stage": UK_SPI_INCOME_BAND_DONORS_STAGE_NAME, - "donors_per_band": self.donors_per_band, + "minimum_donors_per_band": self.minimum_donors_per_band, + "maximum_donor_weight": self.maximum_donor_weight, + "maximum_donor_household_share": self.maximum_donor_household_share, + "maximum_reallocation_share": self.maximum_reallocation_share, + "funding_strata": list(SPI_INCOME_BAND_FUNDING_STRATA), + "seating_scale": self.seating_scale, + "mass_scale": self.mass_scale, + "sample_fraction": self.sample_fraction, + "eligible_households": self.eligible_households, + "expected_donor_count": self.expected_donor_count, + "donor_count": self.donor_count, + "reallocated_mass": self.reallocated_mass, + "mass": {"old_total": self.old_total, "new_total": self.new_total}, + "draw": {"seed": self.seed, "salt": SPI_INCOME_BAND_DONOR_DRAW_SALT}, "taxpayer_count_resource": SPI_INCOME_BAND_TAXPAYER_RESOURCE, "taxpayer_count_period": self.taxpayer_period, "propensity": SPI_INCOME_BAND_DONOR_PROPENSITY, "propensity_cells": self.propensity_cells, "bands": [dict(row) for row in self.band_rows], + "funding": [dict(row) for row in self.funding_rows], "carrier_region_counts": dict(self.carrier_region_counts), } +def spi_income_band_donor_plan( + band_taxpayers: Mapping[int, float], + *, + minimum_donors: int = SPI_INCOME_BAND_MINIMUM_DONORS, + maximum_donor_weight: float = SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT, +) -> dict[int, tuple[int, float]]: + """Each band's donor count and equal donor weight on the full frame. + + The count is the smallest that keeps every donor at or below the maximum + weight, and never below the minimum; the weight is the band's published + taxpayers over that count, so the band's donors carry its published mass. + """ + + if not isinstance(minimum_donors, int) or minimum_donors <= 0: + raise ValueError("minimum_donors must be a positive integer.") + if not np.isfinite(maximum_donor_weight) or maximum_donor_weight <= 0.0: + raise ValueError("maximum_donor_weight must be positive.") + plan: dict[int, tuple[int, float]] = {} + for band in sorted(int(value) for value in band_taxpayers): + taxpayers = float(band_taxpayers[band]) + if not np.isfinite(taxpayers) or taxpayers <= 0.0: + raise ValueError( + f"SPI income band donors need positive published taxpayers; the " + f"band from {band} carries {taxpayers!r}." + ) + count = max(minimum_donors, math.ceil(taxpayers / maximum_donor_weight)) + weight = taxpayers / count + if not 0.0 < weight <= maximum_donor_weight * (1.0 + 1e-12): + raise ValueError( + f"SPI income band donor weight {weight!r} for the band from {band} " + f"is outside (0, {maximum_donor_weight}]." + ) + plan[band] = (count, weight) + return plan + + +def spi_income_band_donor_seats( + plan: Mapping[int, tuple[int, float]], + *, + eligible_households: int, + maximum_donor_household_share: float = ( + SPI_INCOME_BAND_MAXIMUM_DONOR_HOUSEHOLD_SHARE + ), +) -> tuple[float, dict[int, int]]: + """The seating scale and each band's seats on a frame of this size. + + The scale is one while the planned donors fit inside the declared share of + the eligible base households; below that every band's count is scaled + alike and floored, never below one, so a small frame keeps every band + without copying most of its households. + """ + + expected = sum(count for count, _ in plan.values()) + if eligible_households <= 0: + raise ValueError("SPI income band donors found no eligible base household.") + scale = min(1.0, maximum_donor_household_share * eligible_households / expected) + if scale >= 1.0: + return 1.0, {band: count for band, (count, _) in plan.items()} + return scale, { + band: max(1, math.floor(count * scale)) for band, (count, _) in plan.items() + } + + def age_band_lower_bound(ages: np.ndarray) -> np.ndarray: """Map ages to the propensity table's age-band lower bounds.""" @@ -257,6 +408,24 @@ def spi_income_band_propensity( return result +def spi_income_band_donor_propensity( + raw: pd.DataFrame, + *, + age_model: SPIDonorAgeModel, + tape_seed: int = SPI_INCOME_BAND_DONOR_TAPE_SEED, +) -> pd.DataFrame: + """The stage's propensity table from the raw tape. + + One preparation for the stage and for the identity receipt that reruns + it: the income stage's seed and age draw, then the band shares. + """ + + donor = prepare_spi_donor_table(raw, seed=tape_seed, age_model=age_model) + return spi_income_band_propensity( + donor, lower_bounds=SPI_INCOME_BAND_DONOR_LOWER_BOUNDS + ) + + def _candidate_propensity( candidates: pd.DataFrame, propensity: pd.DataFrame, @@ -280,26 +449,40 @@ def stack_spi_income_band_donors( *, propensity: pd.DataFrame, band_taxpayers: Mapping[int, float], - donors_per_band: int = SPI_INCOME_BAND_DONORS_PER_BAND, + minimum_donors: int = SPI_INCOME_BAND_MINIMUM_DONORS, + maximum_donor_weight: float = SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT, + maximum_donor_household_share: float = ( + SPI_INCOME_BAND_MAXIMUM_DONOR_HOUSEHOLD_SHARE + ), + maximum_reallocation_share: float = SPI_INCOME_BAND_MAXIMUM_REALLOCATION_SHARE, seed: int = SPI_INCOME_BAND_DONOR_SEED, taxpayer_period: int, + sample_fraction: float = 1.0, ) -> UKSPIIncomeBandDonorResult: - """Copy ``donors_per_band`` FRS households per band into the SPI channel. - - Carriers are adults of the raw FRS base channel drawn without replacement - with probability proportional to the tape's propensity for the band given - their region, sex and age band; a household is drawn at most once across - bands, the highest band first. Every copy carries the support channel's - lineage columns at clone index 2, the synthetic flag (so the income stage - draws for it), the donor flag, its band and one carrier flag, and starts - at the band's published taxpayers over ``donors_per_band``. + """Seat each band's donors and fund them from their source strata. + + Carriers are adults of the raw FRS base channel, seated without + replacement with probability proportional to their household's weight + times the tape's propensity for the band given their region, sex and age + band, so the donors' composition follows the population's rather than the + sample's; a household is seated at most once across bands, the highest + band first. Every copy carries the support channel's lineage columns at + clone index 2, the synthetic flag (so the income stage draws for it), the + donor flag, its band and one carrier flag, and starts at its band's equal + weight. The mass the donors carry leaves the incumbent households of each + donor's funding stratum in proportion, so the total and every stratum's + mass are unchanged. """ validate_uk_national_frame(frame) - if not isinstance(donors_per_band, int) or donors_per_band <= 0: - raise ValueError("donors_per_band must be a positive integer.") if not isinstance(seed, int): raise ValueError("seed must be an integer.") + for label, share in ( + ("maximum_donor_household_share", maximum_donor_household_share), + ("maximum_reallocation_share", maximum_reallocation_share), + ): + if not np.isfinite(share) or not 0.0 < share < 1.0: + raise ValueError(f"{label} must lie strictly between zero and one.") tables = engine_tables(frame) person = tables["person"].copy() benunit = tables["benunit"].copy() @@ -320,17 +503,17 @@ def stack_spi_income_band_donors( ) if HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR in household.columns: raise ValueError("SPI income band donors were already stacked.") - bands = sorted(int(value) for value in band_taxpayers) - weights_by_band = {} - for band in bands: - taxpayers = float(band_taxpayers[band]) - weight = taxpayers / donors_per_band - if not np.isfinite(weight) or weight <= 0.0: - raise ValueError( - f"SPI income band donors must carry positive initial weight; " - f"band from {band} implies {weight!r}." - ) - weights_by_band[band] = weight + plan = spi_income_band_donor_plan( + band_taxpayers, + minimum_donors=minimum_donors, + maximum_donor_weight=maximum_donor_weight, + ) + bands = sorted(plan) + incoming = frame.weights_for("household") + incoming_weights = np.asarray(incoming.values, dtype=float) + weight_by_household = pd.Series( + incoming_weights, index=household["household_id"].to_numpy() + ) base_households = household.loc[ household[household_channel].eq(BASE_FRS_SUPPORT_CHANNEL) @@ -350,49 +533,79 @@ def stack_spi_income_band_donors( candidates = candidates.sort_values("person_id", kind="stable").reset_index( drop=True ) + candidate_ids = candidates["person_id"].to_numpy() + candidate_households = candidates["person_household_id"].to_numpy() + candidate_weights = ( + candidates["person_household_id"].map(weight_by_household).to_numpy(float) + ) + if not np.isfinite(candidate_weights).all() or (candidate_weights < 0.0).any(): + raise ValueError("SPI income band donor candidates need finite weights.") + + # Rate of seating by weight x propensity for each band; a candidate with a + # zero rate for every band can never be seated and is not eligible. + rates: dict[int, np.ndarray] = {} + for band in bands: + probabilities = _candidate_propensity(candidates, propensity, band=band) + if not np.isfinite(probabilities).all() or (probabilities < 0.0).any(): + raise ValueError("SPI band propensities must be finite and non-negative.") + rates[band] = candidate_weights * probabilities + any_rate = np.zeros(len(candidates), dtype=bool) + for band in bands: + any_rate |= rates[band] > 0.0 + eligible_households = int(pd.unique(candidate_households[any_rate]).size) + seating_scale, seats = spi_income_band_donor_seats( + plan, + eligible_households=eligible_households, + maximum_donor_household_share=maximum_donor_household_share, + ) - rng = np.random.default_rng(seed) selected_household: dict[int, int] = {} carrier_person: dict[int, int] = {} + band_eligible: dict[int, int] = {} for band in sorted(bands, reverse=True): - available = ~candidates["person_household_id"].isin(selected_household) - pool = candidates.loc[available] - probabilities = _candidate_propensity(pool, propensity, band=band) - if not np.isfinite(probabilities).all() or (probabilities < 0.0).any(): - raise ValueError("SPI band propensities must be finite and non-negative.") - positive = probabilities > 0.0 - if positive.sum() < donors_per_band: + rate = rates[band] + taken_before = np.fromiter( + (int(value) in selected_household for value in candidate_households), + dtype=bool, + count=len(candidates), + ) + eligible = (rate > 0.0) & ~taken_before + band_eligible[band] = int(pd.unique(candidate_households[eligible]).size) + if band_eligible[band] < seats[band]: raise ValueError( - f"SPI income band donors need {donors_per_band} candidate " - f"households with positive propensity for the band from {band}; " - f"found {int(positive.sum())}." + f"SPI income band donors need {seats[band]} candidate households " + f"with positive weight and propensity for the band from {band}; " + f"found {band_eligible[band]}." ) - # Draw persons, one per household: a person draw whose household was - # already taken in this band is skipped and the draw continues, so - # the selection stays a weighted draw without replacement over - # households at the person-level propensity. - eligible = np.flatnonzero(positive) - order = rng.choice( - eligible, - size=len(eligible), - replace=False, - p=probabilities[eligible] / probabilities[eligible].sum(), + uniforms = stable_identity_uniforms( + candidate_ids, + seed=seed, + salt=f"{SPI_INCOME_BAND_DONOR_DRAW_SALT}:{band}", + ) + # Efraimidis-Spirakis: the largest u ** (1 / rate) are a weighted draw + # without replacement; log(u) / rate orders them identically. + keys = np.full(len(candidates), -np.inf) + keys[eligible] = ( + np.log(np.maximum(uniforms[eligible], np.finfo(float).tiny)) + / rate[eligible] ) + order = np.lexsort((candidate_ids, -keys)) taken: set[int] = set() for position in order: - row = pool.iloc[int(position)] - household_id = int(row["person_household_id"]) + if not eligible[position]: + break + household_id = int(candidate_households[position]) if household_id in taken: continue taken.add(household_id) selected_household[household_id] = band - carrier_person[int(row["person_id"])] = band - if len(taken) == donors_per_band: + carrier_person[int(candidate_ids[position])] = band + if len(taken) == seats[band]: break - if len(taken) != donors_per_band: + if len(taken) != seats[band]: raise ValueError( - f"SPI income band donors could not seat {donors_per_band} " - f"households for the band from {band}." + f"SPI income band donors could not seat {seats[band]} households " + f"for the band from {band}." ) selected_ids = set(selected_household) @@ -455,21 +668,54 @@ def stack_spi_income_band_donors( ): raise ValueError("SPI income band donors lost a carrier in the copy.") - donor_weights = np.asarray( - [weights_by_band[int(value)] for value in donor_band], dtype=float + # Funding: the donors' mass leaves the incumbent households of each donor's + # stratum in proportion (the SPI support channel's own rule), so no mass is + # created and every stratum keeps its mass. + planned_weights = np.asarray( + [plan[int(value)][1] for value in donor_band], dtype=float + ) + incumbent_strata = _funding_strata(household) + donor_strata = _funding_strata(donor_household) + incumbent_mass = ( + pd.Series(incoming_weights).groupby(incumbent_strata.to_numpy()).sum() ) + donor_mass = pd.Series(planned_weights).groupby(donor_strata.to_numpy()).sum() + unfunded = sorted(set(donor_mass.index) - set(incumbent_mass.index)) + if unfunded or not (incumbent_mass.reindex(donor_mass.index) > 0.0).all(): + raise ValueError( + "SPI income band donors were drawn from a stratum without incumbent " + f"mass: {unfunded}." + ) + capacity = ( + maximum_reallocation_share + * incumbent_mass.reindex(donor_mass.index) + / donor_mass + ) + mass_scale = float(min(1.0, capacity.min())) + donor_weights = planned_weights * mass_scale + factor_by_stratum = ( + 1.0 - (donor_mass * mass_scale / incumbent_mass.reindex(donor_mass.index)) + ).reindex(incumbent_mass.index, fill_value=1.0) + incumbent_factor = incumbent_strata.map(factor_by_stratum).to_numpy(dtype=float) + final_incumbent = incoming_weights * incumbent_factor + if (final_incumbent[incoming_weights > 0.0] <= 0.0).any(): + raise ValueError("SPI income band donor funding emptied an incumbent row.") + final_person = pd.concat([person, donor_person], ignore_index=True) final_benunit = pd.concat([benunit, donor_benunit], ignore_index=True) final_household = pd.concat([household, donor_household], ignore_index=True) - incoming = frame.weights_for("household") - final_weights = np.r_[incoming.values, donor_weights] old_total = float(incoming.total) + final_weights = _exact_incumbent_total( + np.r_[final_incumbent, donor_weights], + target=old_total, + incumbent_rows=len(final_incumbent), + ) new_total = float(final_weights.sum()) receipt = MassChangeRecord( entity="household", old_total=old_total, new_total=new_total, - declared_factor=None, + declared_factor=1.0, reason=SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON, ) result = uk_national_frame( @@ -502,19 +748,49 @@ def stack_spi_income_band_donors( band_rows.append( { "lower_bound": band, + "expected_donor_households": int(plan[band][0]), "donor_households": int(mask.sum()), "carriers": int(carriers.sum()), - "donor_weight": float(weights_by_band[band]), + "eligible_households": band_eligible[band], + "planned_donor_weight": float(plan[band][1]), + "donor_weight": float(plan[band][1] * mass_scale), "weighted_taxpayers": float(donor_weights[mask].sum()), "published_taxpayers": float(band_taxpayers[band]), "carrier_mean_age": float(carrier_ages[carriers].mean()), "carrier_regions": int(carrier_region[carriers].nunique()), } ) + donor_rows_by_stratum = ( + pd.Series(1, index=donor_strata.to_numpy()).groupby(level=0).sum() + ) + funding_rows = [ + { + "stratum": str(stratum), + "donor_households": int(donor_rows_by_stratum.get(stratum, 0)), + "incumbent_mass": float(incumbent_mass[stratum]), + "donor_mass": float(donor_mass.get(stratum, 0.0) * mass_scale), + "factor": float(factor_by_stratum[stratum]), + } + for stratum in sorted(incumbent_mass.index, key=str) + ] return UKSPIIncomeBandDonorResult( frame=result, band_rows=tuple(band_rows), - donors_per_band=donors_per_band, + funding_rows=tuple(funding_rows), + minimum_donors_per_band=int(minimum_donors), + maximum_donor_weight=float(maximum_donor_weight), + maximum_donor_household_share=float(maximum_donor_household_share), + maximum_reallocation_share=float(maximum_reallocation_share), + seating_scale=float(seating_scale), + mass_scale=mass_scale, + eligible_households=eligible_households, + expected_donor_count=int(sum(count for count, _ in plan.values())), + donor_count=len(selected_household), + reallocated_mass=float(donor_weights.sum()), + old_total=old_total, + new_total=new_total, + seed=seed, + sample_fraction=float(sample_fraction), taxpayer_period=int(taxpayer_period), carrier_region_counts={ str(key): int(value) @@ -524,6 +800,59 @@ def stack_spi_income_band_donors( ) +def _funding_strata(household: pd.DataFrame) -> pd.Series: + """Each household's funding stratum label.""" + + missing = sorted(set(SPI_INCOME_BAND_FUNDING_STRATA) - set(household.columns)) + if missing: + raise ValueError(f"SPI income band donor funding needs columns {missing}.") + columns = [ + household[column].astype(str) for column in SPI_INCOME_BAND_FUNDING_STRATA + ] + label = columns[0] + for column in columns[1:]: + label = label + "|" + column + return label.reset_index(drop=True) + + +#: Rows tried for the exact-total correction before the stage refuses. +_EXACT_TOTAL_CORRECTION_CANDIDATES = 256 + + +def _exact_incumbent_total( + values: np.ndarray, *, target: float, incumbent_rows: int +) -> np.ndarray: + """Weights whose sum is exactly ``target``, corrected on an incumbent row. + + The reallocation conserves mass arithmetically; the float sum can still + miss the incoming total in its last bits. The correction goes onto one + incumbent row, never a donor, so the donors keep their equal band weights + bit for bit. A heavy row moves the sum in coarse steps that can jump over + the target, so the heaviest rows are tried first and the lightest + positive rows, whose steps are finer, straight after; the order depends + only on the weights, so the corrected row is reproducible. + """ + + if float(values.sum()) == target: + return values + incumbents = values[:incumbent_rows] + positive = np.flatnonzero(incumbents > 0.0) + heaviest_first = positive[np.argsort(-incumbents[positive], kind="stable")] + half = _EXACT_TOTAL_CORRECTION_CANDIDATES // 2 + candidates = dict.fromkeys( + int(index) for index in (*heaviest_first[:half], *heaviest_first[::-1][:half]) + ) + for index in candidates: + corrected = _exact_total_correction( + values, target=target, correction_index=index + ) + if corrected is not None: + return corrected + raise ValueError( + "SPI income band donor funding has no representable exact-total correction." + ) + + @dataclass(frozen=True) class UKSPIIncomeBandDonorStageTransform: """Whole-stage transform for the reserved SPI income band rows.""" @@ -531,11 +860,16 @@ class UKSPIIncomeBandDonorStageTransform: spi_tab_path: Path stage: SourceStageSpec seed: int = SPI_INCOME_BAND_DONOR_SEED - # A #627 scale-ladder receipt build scales the reserved count with the - # survey-side sample fraction, as the support stack does. + # The survey-side sample fraction of a #627 scale-ladder build. The stage + # no longer scales its counts by it: the seating and mass scales follow + # the frame itself. It is carried into the receipt so the gate can require + # both scales to be one on a full build. sample_fraction: float = 1.0 donor_table: pd.DataFrame | None = field(default=None, repr=False, compare=False) band_taxpayers: Mapping[int, float] | None = None + # The income stage's donor age model; read from the vendored ONS resource + # and the engine's State Pension age at the build period when not given. + age_model: SPIDonorAgeModel | None = field(default=None, repr=False, compare=False) last_result: UKSPIIncomeBandDonorResult | None = field(default=None, init=False) def __init__( @@ -547,9 +881,13 @@ def __init__( sample_fraction: float = 1.0, donor_table: pd.DataFrame | None = None, band_taxpayers: Mapping[int, float] | None = None, + age_model: SPIDonorAgeModel | None = None, ) -> None: if donor_table is not None and not isinstance(donor_table, pd.DataFrame): raise TypeError("donor_table must be a pandas DataFrame.") + if age_model is not None and not isinstance(age_model, SPIDonorAgeModel): + raise TypeError("age_model must be an SPIDonorAgeModel.") + object.__setattr__(self, "age_model", age_model) object.__setattr__(self, "spi_tab_path", Path(spi_tab_path)) object.__setattr__(self, "stage", stage) object.__setattr__(self, "seed", seed) @@ -568,11 +906,7 @@ def __init__( def __call__(self, frame: Frame) -> Frame: validate_uk_national_frame(frame) - donors_per_band = _assert_band_donor_stage_parameters( - self.stage, seed=self.seed - ) - if self.sample_fraction != 1.0: - donors_per_band = max(1, int(round(donors_per_band * self.sample_fraction))) + _assert_band_donor_stage_parameters(self.stage, seed=self.seed) if self.donor_table is None: identity = verify_spi_donor_identity(self.spi_tab_path) raw = pd.read_csv(identity.path, delimiter="\t") @@ -580,11 +914,15 @@ def __call__(self, frame: Frame) -> Frame: raise TypeError("SPI donor identity verification returned no proof.") else: raw = self.donor_table.copy(deep=True) - donor = prepare_spi_donor_table(raw, seed=self.seed) - propensity = spi_income_band_propensity( - donor, lower_bounds=SPI_INCOME_BAND_DONOR_LOWER_BOUNDS - ) period = int(uk_time_period(frame)) + propensity = spi_income_band_donor_propensity( + raw, + age_model=( + load_spi_donor_age_model(period) + if self.age_model is None + else self.age_model + ), + ) band_taxpayers = ( load_hmrc_itl_band_taxpayers(period) if self.band_taxpayers is None @@ -594,9 +932,9 @@ def __call__(self, frame: Frame) -> Frame: frame, propensity=propensity, band_taxpayers=band_taxpayers, - donors_per_band=donors_per_band, seed=self.seed, taxpayer_period=period, + sample_fraction=self.sample_fraction, ) object.__setattr__(self, "last_result", result) return result.frame @@ -620,8 +958,53 @@ def _operation(stage: SourceStageSpec, kind: str) -> SourceOperationSpec: return matches[0] -def _assert_band_donor_stage_parameters(stage: SourceStageSpec, *, seed: int) -> int: - """Bind the stage manifest to the reviewed constants; return donors per band.""" +def spi_income_band_donor_operation_parameters( + *, seed: int = SPI_INCOME_BAND_DONOR_SEED +) -> dict[str, dict[str, object]]: + """The reviewed operation payload the manifests must carry verbatim.""" + + return { + "stack_income_band_donor_households": { + "taxpayer_count_resource": SPI_INCOME_BAND_TAXPAYER_RESOURCE, + "taxpayer_count_measure": SPI_INCOME_BAND_TAXPAYER_MEASURE, + "taxpayer_count_period": SPI_INCOME_BAND_TAXPAYER_PERIOD, + "band_lower_bounds": list(SPI_INCOME_BAND_DONOR_LOWER_BOUNDS), + "minimum_donors_per_band": SPI_INCOME_BAND_MINIMUM_DONORS, + "maximum_donor_weight": SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT, + "count_rule": SPI_INCOME_BAND_DONOR_COUNT_RULE, + "weight_rule": SPI_INCOME_BAND_DONOR_WEIGHT_RULE, + "expected_band_count": len(SPI_INCOME_BAND_DONOR_LOWER_BOUNDS), + "candidate_population": SPI_INCOME_BAND_DONOR_CANDIDATES, + "candidate_order": "person_id ascending breaks equal draw keys", + "draw": SPI_INCOME_BAND_DONOR_DRAW, + "propensity": SPI_INCOME_BAND_DONOR_PROPENSITY, + "tape_preparation": SPI_INCOME_BAND_DONOR_TAPE_PREPARATION, + "tape_preparation_seed": SPI_INCOME_BAND_DONOR_TAPE_SEED, + "seed": seed, + "salt": SPI_INCOME_BAND_DONOR_DRAW_SALT, + "clone_index": SPI_INCOME_BAND_DONOR_CLONE_INDEX, + "flag_column": HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR, + "band_column": SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN, + "carrier_column": PERSON_IS_SPI_INCOME_BAND_CARRIER, + "funding": SPI_INCOME_BAND_DONOR_FUNDING, + "funding_strata": list(SPI_INCOME_BAND_FUNDING_STRATA), + "maximum_reallocation_share": SPI_INCOME_BAND_MAXIMUM_REALLOCATION_SHARE, + "maximum_donor_household_share": ( + SPI_INCOME_BAND_MAXIMUM_DONOR_HOUSEHOLD_SHARE + ), + "seating_scale": SPI_INCOME_BAND_DONOR_SEATING_SCALE, + "mass_scale": SPI_INCOME_BAND_DONOR_MASS_SCALE, + "never_zero_weight": True, + "weight_kind_out": WeightKind.IMPORTANCE.value, + "conservation": "exact_total", + "declared_factor": 1.0, + "reason": SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON, + } + } + + +def _assert_band_donor_stage_parameters(stage: SourceStageSpec, *, seed: int) -> None: + """Bind the stage manifest to the reviewed constants.""" if stage.stage != UK_SPI_INCOME_BAND_DONORS_STAGE_NAME: raise ValueError( @@ -634,28 +1017,9 @@ def _assert_band_donor_stage_parameters(stage: SourceStageSpec, *, seed: int) -> f"stack_income_band_donor_households operation, got {kinds}." ) operation = _operation(stage, "stack_income_band_donor_households") - expected = { - "taxpayer_count_resource": SPI_INCOME_BAND_TAXPAYER_RESOURCE, - "taxpayer_count_measure": SPI_INCOME_BAND_TAXPAYER_MEASURE, - "taxpayer_count_period": SPI_INCOME_BAND_TAXPAYER_PERIOD, - "band_lower_bounds": list(SPI_INCOME_BAND_DONOR_LOWER_BOUNDS), - "donors_per_band": SPI_INCOME_BAND_DONORS_PER_BAND, - "expected_band_count": len(SPI_INCOME_BAND_DONOR_LOWER_BOUNDS), - "expected_donor_count": SPI_INCOME_BAND_DONOR_COUNT, - "candidate_population": SPI_INCOME_BAND_DONOR_CANDIDATES, - "candidate_order": "person_id ascending", - "draw": "weighted_without_replacement", - "propensity": SPI_INCOME_BAND_DONOR_PROPENSITY, - "seed": seed, - "clone_index": SPI_INCOME_BAND_DONOR_CLONE_INDEX, - "flag_column": HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR, - "band_column": SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN, - "carrier_column": PERSON_IS_SPI_INCOME_BAND_CARRIER, - "initial_weight": SPI_INCOME_BAND_DONOR_INITIAL_WEIGHT, - "never_zero_weight": True, - "weight_kind_out": WeightKind.IMPORTANCE.value, - "reason": SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON, - } + expected = spi_income_band_donor_operation_parameters(seed=seed)[ + "stack_income_band_donor_households" + ] actual = dict(operation.parameters) if actual != expected: drifted = sorted( @@ -665,24 +1029,33 @@ def _assert_band_donor_stage_parameters(stage: SourceStageSpec, *, seed: int) -> "spi_income_band_donors stack_income_band_donor_households declaration " f"drifted from the reviewed mapping on parameter(s) {drifted}." ) - return int(expected["donors_per_band"]) __all__ = [ "HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR", "PERSON_IS_SPI_INCOME_BAND_CARRIER", "SPI_INCOME_BAND_DONOR_CLONE_INDEX", - "SPI_INCOME_BAND_DONOR_COUNT", + "SPI_INCOME_BAND_DONOR_DRAW_SALT", "SPI_INCOME_BAND_DONOR_LOWER_BOUNDS", "SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN", + "SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON", "SPI_INCOME_BAND_DONOR_SEED", - "SPI_INCOME_BAND_DONORS_PER_BAND", - "UK_SPI_INCOME_BAND_DONOR_OUTPUT_COLUMNS", - "UK_SPI_INCOME_BAND_DONORS_STAGE_NAME", + "SPI_INCOME_BAND_DONOR_TAPE_SEED", + "SPI_INCOME_BAND_FUNDING_STRATA", + "SPI_INCOME_BAND_MAXIMUM_DONOR_HOUSEHOLD_SHARE", + "SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT", + "SPI_INCOME_BAND_MAXIMUM_REALLOCATION_SHARE", + "SPI_INCOME_BAND_MINIMUM_DONORS", "UKSPIIncomeBandDonorResult", "UKSPIIncomeBandDonorStageTransform", + "UK_SPI_INCOME_BAND_DONORS_STAGE_NAME", + "UK_SPI_INCOME_BAND_DONOR_OUTPUT_COLUMNS", "income_band_lower_bound", "load_hmrc_itl_band_taxpayers", + "spi_income_band_donor_operation_parameters", + "spi_income_band_donor_plan", + "spi_income_band_donor_propensity", + "spi_income_band_donor_seats", "spi_income_band_propensity", "stack_spi_income_band_donors", ] diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_income.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_income.py index c734c632a..a07c5bc87 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_income.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_income.py @@ -15,6 +15,7 @@ import pandas as pd from microcosm.build.gates import FitWeightRecord +from microcosm.build.stochastic_assignment import stable_identity_uniforms from microcosm.build.uk_runtime.frs_disability import ( UKDWPDisabilityCategoryRates, UKDWPDisabilityFlagRates, @@ -1015,6 +1016,10 @@ def impute_uk_spi_income_support( SPI_INCOME_BAND_CARRIER_COLUMN = "person_is_spi_income_band_carrier" SPI_INCOME_BAND_LOWER_BOUND_COLUMN = "spi_income_band_donor_lower_bound" +#: The carrier's leaf draw is keyed on the FRS person it copies (the support +#: lineage column), falling back to ``person_id`` on a frame without lineage. +SPI_INCOME_BAND_CARRIER_KEY_COLUMN = "person_source_id" +SPI_INCOME_BAND_DONOR_LEAF_DRAW_SALT = "spi_income_band_donor_leaf_draw" def _spi_agerange_codes(ages: np.ndarray) -> np.ndarray: @@ -1060,7 +1065,10 @@ def _resample_band_donor_leaves( not inherit the zero State Pension leaf of a donor drawn below it. One record is drawn FACT-weighted with replacement and all stage-1 leaves are copied from it, then uprated exactly as the forest draws were; the - accounting aggregates derive after the draw as usual. Composite records + accounting aggregates derive after the draw as usual. The draw is keyed on + the carrier's identity (the id of the FRS person it copies, where the frame + carries it), so a carrier draws the same record whatever the row order and + whoever else is seated (microcosm#1063). Composite records stay in the pools as published. Every carrier's realised total income is checked against its band after the draw and any breach refuses the stage. """ @@ -1097,6 +1105,9 @@ def _resample_band_donor_leaves( ("household_id", "region", SPI_INCOME_BAND_LOWER_BOUND_COLUMN), label="band donor households", ) + _require_columns( + person, ("person_id", "person_household_id"), label="band donor carriers" + ) for column in ("total_income", "is_composite", "region", "FACT"): if column not in donor.columns: raise ValueError(f"band donor resample needs donor column {column!r}.") @@ -1172,7 +1183,16 @@ def _resample_band_donor_leaves( if state_pension_age is not None else np.zeros(len(carrier_rows), dtype=bool) ) - rng = np.random.default_rng(seed) + key_column = ( + SPI_INCOME_BAND_CARRIER_KEY_COLUMN + if SPI_INCOME_BAND_CARRIER_KEY_COLUMN in carrier_rows.columns + else "person_id" + ) + uniforms = stable_identity_uniforms( + carrier_rows[key_column].to_numpy(), + seed=seed, + salt=SPI_INCOME_BAND_DONOR_LEAF_DRAW_SALT, + ) drawn = np.empty(len(carrier_rows), dtype=np.int64) matched = np.zeros(len(carrier_rows), dtype=bool) age_matched = np.zeros(len(carrier_rows), dtype=bool) @@ -1206,8 +1226,12 @@ def _resample_band_donor_leaves( base_is_sided, ) pool, matched[index], age_matched[index], side_matched[index] = pool_cache[key] - weights = fact[pool] - drawn[index] = int(rng.choice(pool, p=weights / weights.sum())) + # Inverse CDF over the pool in tape order at the carrier's own uniform. + cumulative = np.cumsum(fact[pool]) + position = int( + np.searchsorted(cumulative, uniforms[index] * cumulative[-1], side="right") + ) + drawn[index] = int(pool[min(position, pool.size - 1)]) leaves = leaf_values[drawn] * factors person = person.copy() person.loc[carriers, columns] = leaves @@ -1259,6 +1283,8 @@ def _resample_band_donor_leaves( "age_pool_minimum": age_pool_minimum, "state_pension_age": state_pension_age, "seed": seed, + "salt": SPI_INCOME_BAND_DONOR_LEAF_DRAW_SALT, + "draw_key": key_column, "weighting": "FACT", "with_replacement": True, "total_income_basis": "published TEI + TII, 2022-23 terms", @@ -2021,6 +2047,8 @@ def _assign_spi_values( "SPI_QRF_SOURCE_COLUMNS", "SPI_SOURCE_COMPOSITE_INDICATOR", "SPI_INCOME_BAND_CARRIER_COLUMN", + "SPI_INCOME_BAND_CARRIER_KEY_COLUMN", + "SPI_INCOME_BAND_DONOR_LEAF_DRAW_SALT", "SPI_INCOME_BAND_LOWER_BOUND_COLUMN", "prepare_spi_donor_table", "SPI_SOURCE_LEAF_RECONCILIATION_ABS_TOLERANCE_GBP", diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_spine.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_spine.py index 85d5c6753..cf3100703 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_spine.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/spi_spine.py @@ -59,6 +59,8 @@ SPI_DONOR_AGE_POPULATION_PERIOD, SPI_DONOR_AGE_POPULATION_RESOURCE, SPI_DONOR_INCOME_YEAR, + SPI_INCOME_BAND_CARRIER_KEY_COLUMN, + SPI_INCOME_BAND_DONOR_LEAF_DRAW_SALT, SPI_INCOME_UPRATING_VARIABLES, SPI_MINIMUM_RECIPIENT_AGE, SPI_SOURCE_TI_FORMULA, @@ -112,6 +114,10 @@ "minimum (composites never age-match), then to the carrier's region where " "that pool holds the regional minimum" ) +SPI_SPINE_BAND_DONOR_DRAW = ( + "identity_keyed_inverse_cdf: stable_identity_uniforms(draw_key, seed, salt) " + "over the pool's cumulative FACT in tape order" +) SPI_SPINE_BAND_DONOR_REGIONAL_POOL_MINIMUM = 20 SPI_SPINE_BAND_DONOR_AGE_POOL_MINIMUM = 20 SPI_SPINE_STATE_PENSION_AGE_SOURCE = ( @@ -1144,6 +1150,9 @@ def _assert_income_stage_parameters( "age_pool_minimum": SPI_SPINE_BAND_DONOR_AGE_POOL_MINIMUM, "weighting": "FACT", "with_replacement": True, + "draw": SPI_SPINE_BAND_DONOR_DRAW, + "draw_key": SPI_INCOME_BAND_CARRIER_KEY_COLUMN, + "salt": SPI_INCOME_BAND_DONOR_LEAF_DRAW_SALT, "outputs": "stage-1 outputs, uprated as the stage-1 draws", "seed": seed + 2, } diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/spine_build.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/spine_build.py index 523a1f226..2fd730777 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/spine_build.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/spine_build.py @@ -67,11 +67,17 @@ uk_cgt_asset_type_stage_transform, ) from microcosm.build.uk_runtime.cgt_imputation import uk_cgt_spine_stage_transform +from microcosm.build.uk_runtime.cgt_residential_split import ( + UKCGTResidentialSplitStageTransform, +) from microcosm.build.uk_runtime.cgt_structure import ( UKCGTIncidenceAnchorStageTransform, UKCGTIncidenceCloneStageTransform, ) from microcosm.build.uk_runtime.cgt_support import UKCGTSupportSplitStageTransform +from microcosm.build.uk_runtime.child_benefit_take_up import ( + UKChildBenefitTakeUpStageTransform, +) from microcosm.build.uk_runtime.content_identity import uk_frame_content_identity from microcosm.build.uk_runtime.etb_services import UKETBServicesStageTransform from microcosm.build.uk_runtime.etb_vat import UKETBVATStageTransform @@ -701,21 +707,20 @@ def _declared_seeds(stages) -> dict[str, dict[str, int]]: stage_seeds[stage.stage] = seed elif operation.kind == "draw_capital_gains_prior_from_banded_quantiles": stage_seeds[str(operation.parameters["salt"])] = seed - elif operation.kind == "stack_income_band_donor_households": - stage_seeds["stack_income_band_donor_households"] = seed - elif operation.kind == "resample_band_donor_leaves": - stage_seeds["band_donor_resample"] = seed + elif operation.kind in ( + "stack_income_band_donor_households", + "resample_band_donor_leaves", + ): + # Identity-keyed draws: the seed is declared under the + # salt it is hashed with (microcosm#1063). + stage_seeds[str(operation.parameters["salt"])] = seed elif operation.kind == "impute_spi_housing_shell": stage_seeds[stage.stage] = seed elif operation.kind == "price_domestic_energy": stage_seeds["gas_disconnection"] = seed elif operation.kind == "within_band_draws": stage_seeds["within_band_draws"] = seed - elif operation.kind in ( - "assign_residential_property_flag", - "assign_badr_qualifying_gains", - "assign_main_asset_type", - ): + elif operation.kind == "assign_badr_qualifying_gains": stage_seeds[operation.kind] = seed elif operation.kind == "convert_donors_to_target_stock": stage_seeds[str(operation.parameters["salt"])] = seed @@ -1528,6 +1533,11 @@ def prepare_uk_spine_execution( stage=stages_by_name["pension_credit_take_up"], engine=engine, ) + if "child_benefit_take_up" in stage_names: + implementations["child_benefit_take_up"] = UKChildBenefitTakeUpStageTransform( + stage=stages_by_name["child_benefit_take_up"], + engine=engine, + ) if "uc_deduction_attributes" in stage_names: implementations["uc_deduction_attributes"] = ( UKUCDeductionAttributesStageTransform( @@ -1552,6 +1562,10 @@ def prepare_uk_spine_execution( stages_by_name["hmrc_cgt_asset_type_spine"] ) ) + if "cgt_residential_split" in stage_names: + implementations["cgt_residential_split"] = UKCGTResidentialSplitStageTransform( + stage=stages_by_name["cgt_residential_split"] + ) if "cgt_incidence_anchor" in stage_names: implementations["cgt_incidence_anchor"] = UKCGTIncidenceAnchorStageTransform( stage=stages_by_name["cgt_incidence_anchor"] diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/stage_health.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/stage_health.py index c1d65504d..4b4d0da3a 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/stage_health.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/stage_health.py @@ -45,10 +45,14 @@ def uk_stage_health_gate( return _spi_income_spine_gate(stage, evidence, parameters) if check == "pension_credit_take_up": return _pension_credit_take_up_gate(stage, evidence, parameters) + if check == "child_benefit_take_up": + return _child_benefit_take_up_gate(stage, evidence, parameters) if check == "source_signal": return _source_signal_gate(stage, evidence, parameters) if check == "age_tail_targets": return _age_tail_targets_gate(stage, evidence, parameters) + if check == "cgt_residential_split": + return _cgt_residential_split_gate(stage, evidence, parameters) if check == "cgt_support_split": return _cgt_support_split_gate(stage, evidence, parameters) if check == "spi_income_band_donor_support": @@ -63,6 +67,8 @@ def uk_stage_health_gate( return _latent_attribute_realization_gate(stage, evidence) if check == "household_composition": return _household_composition_gate(stage, evidence, parameters) + if check == "wealth_coherence": + return _wealth_coherence_gate(stage, evidence, parameters) if check == "energy_rake": return _energy_rake_gate(stage, evidence, parameters) if check == "bus_travel_facts": @@ -185,6 +191,40 @@ def _support_clip_gate( "columns_checked": len(expected_columns) - len(exempt_columns), "exempt_columns": sorted(exempt_columns), } + # The donor floor (microcosm#1063 c9): negative diary consumption is raised + # to the declared floor before the clip ranges are read, so no clip range + # may start below it and no negative donor row may remain. + declared_floor = parameters.get("donor_floor") + floor = evidence.get("donor_floor") + if declared_floor is None: + pass + elif floor is None: + failures.append(f"{stage}: missing the donor_floor receipt.") + else: + floor = _mapping(floor, label=f"{stage}.donor_floor") + level = _finite_number(floor.get("floor"), label=f"{stage}.donor_floor.floor") + if level != float(declared_floor): + failures.append( + f"{stage}: the donor floor {level} is not the declared {declared_floor}." + ) + remaining = int(floor.get("remaining_negative_rows", -1)) + if remaining != 0: + failures.append( + f"{stage}: the donor floor left {remaining} negative consumption row(s)." + ) + for column, receipt in columns.items(): + if isinstance(receipt, Mapping) and "donor_min" in receipt: + lower = _finite_number( + receipt["donor_min"], label=f"{column}.donor_min" + ) + if column in _mapping(floor.get("columns", {}), label="floor") and ( + lower < level + ): + failures.append( + f"{stage}: {column!r} clip range starts at {lower}, below " + f"the donor floor {level}." + ) + details["donor_floor_rows_raised"] = int(floor.get("rows_raised", 0)) return ( _fail(stage, check, failures, details) if failures @@ -803,6 +843,121 @@ def _spi_support_channel_gate( ) +def _child_benefit_take_up_gate( + stage: str, + evidence: Mapping[str, object], + parameters: Mapping[str, object], +) -> GateResult: + """Claims meet HMRC's rates by child age and opt-outs its family share. + + The claimed share of eligible children is held overall against the + published rates at the frame's own age mix, and at each single year of age + with enough child rows to measure it; an age whose solved rate was clipped + (reporters alone exceed the published rate, or every family claiming + falls short of it) is reported, not held. The opted-out share of claiming + families is held to the published share, or may fall short of it only + where the receipt shows the charged families exhausted (microcosm#1063). + """ + + check = "child_benefit_take_up" + overall_tolerance = _finite_number( + parameters["maximum_claim_rate_deviation"], + label=f"{stage}.maximum_claim_rate_deviation", + ) + age_tolerance = _finite_number( + parameters["maximum_age_claim_rate_deviation"], + label=f"{stage}.maximum_age_claim_rate_deviation", + ) + minimum_age_rows = int(parameters["minimum_age_child_rows"]) + opt_out_tolerance = _finite_number( + parameters["maximum_opt_out_share_deviation"], + label=f"{stage}.maximum_opt_out_share_deviation", + ) + minimum_families = int(parameters["minimum_eligible_family_units"]) + failures: list[str] = [] + claims = _mapping(evidence.get("claims"), label=f"{stage}.claims") + opt_outs = _mapping(evidence.get("opt_outs"), label=f"{stage}.opt_outs") + families = int(claims.get("eligible_family_units", 0)) + details: dict[str, object] = {"eligible_family_units": families} + target = claims.get("target_rate") + realized = claims.get("realized_rate") + if families < minimum_families or target is None or realized is None: + failures.append(f"{stage}: the receipt has {families} eligible families.") + else: + target = _finite_number(target, label=f"{stage}.claims.target_rate") + realized = _finite_number(realized, label=f"{stage}.claims.realized_rate") + details["claim_rate"] = {"target": target, "realized": realized} + if abs(realized - target) > overall_tolerance: + failures.append( + f"{stage}: {realized:.4f} of eligible children are claimed for " + f"against {target:.4f} at the published rates by age (tolerance " + f"{overall_tolerance})." + ) + ages = claims.get("ages") + if not isinstance(ages, list) or not ages: + failures.append(f"{stage}: the receipt carries no claim rates by age.") + ages = [] + worst: tuple[float, int] | None = None + clipped: list[int] = [] + measured = 0 + for row in ages: + row = _mapping(row, label=f"{stage}.claims.ages[]") + age = int(row.get("age", -1)) + if row.get("clipped") is True: + clipped.append(age) + continue + if int(row.get("eligible_child_rows", 0)) < minimum_age_rows: + continue + published = _finite_number( + row.get("published_rate"), label=f"{stage}.age {age}.published_rate" + ) + achieved = _finite_number( + row.get("realized_rate"), label=f"{stage}.age {age}.realized_rate" + ) + measured += 1 + deviation = abs(achieved - published) + if worst is None or deviation > worst[0]: + worst = (deviation, age) + if deviation > age_tolerance: + failures.append( + f"{stage}: at age {age} {achieved:.4f} of eligible children are " + f"claimed for against the published {published:.4f} (tolerance " + f"{age_tolerance})." + ) + details["ages_measured"] = measured + details["clipped_ages"] = clipped + if worst is not None: + details["largest_age_deviation"] = {"age": worst[1], "deviation": worst[0]} + + share = opt_outs.get("realized_share") + target_share = _finite_number( + opt_outs.get("target_share"), label=f"{stage}.opt_outs.target_share" + ) + exhausted = opt_outs.get("pool_exhausted") is True + if share is None: + failures.append(f"{stage}: the receipt carries no opted-out share.") + else: + share = _finite_number(share, label=f"{stage}.opt_outs.realized_share") + details["opt_out_share"] = { + "target": target_share, + "realized": share, + "pool_exhausted": exhausted, + } + if share > target_share + opt_out_tolerance or ( + not exhausted and share < target_share - opt_out_tolerance + ): + failures.append( + f"{stage}: {share:.4f} of claiming families opted out against " + f"the published {target_share:.4f} (tolerance {opt_out_tolerance}, " + f"charged families exhausted: {exhausted})." + ) + return ( + _fail(stage, check, failures, details) + if failures + else _pass(stage, check, details) + ) + + def _pension_credit_take_up_gate( stage: str, evidence: Mapping[str, object], @@ -1145,16 +1300,82 @@ def _spi_income_band_donor_support_gate( evidence: Mapping[str, object], parameters: Mapping[str, object], ) -> GateResult: - """Every reserved band carries its donors at positive band-exact weight.""" + """Every reserved band carries its donors, and their mass was reallocated. + + The donors are a mass-conserving support channel (microcosm#1063): total + household mass is unchanged, no incumbent stratum gives up more than the + declared share, and no donor starts above the declared weight. On a full + frame (both scales one) each band seats its planned count, never fewer + than the minimum, and carries its published taxpayers; a build declaring + the full survey sample must be at full scale. + """ check = "spi_income_band_donor_support" - donors_per_band = int(parameters["donors_per_band"]) + minimum_donors = int(parameters["minimum_donors_per_band"]) + maximum_donor_weight = _finite_number( + parameters["maximum_donor_weight"], label=f"{stage}.maximum_donor_weight" + ) + minimum_funding_factor = _finite_number( + parameters["minimum_funding_factor"], label=f"{stage}.minimum_funding_factor" + ) + taxpayer_tolerance = _finite_number( + parameters["maximum_band_taxpayer_deviation"], + label=f"{stage}.maximum_band_taxpayer_deviation", + ) + tolerance = _finite_number( + parameters["maximum_relative_mass_deviation"], + label=f"{stage}.maximum_relative_mass_deviation", + ) expected_bands = [int(value) for value in parameters["band_lower_bounds"]] + failures: list[str] = [] + + for key, expected in ( + ("minimum_donors_per_band", float(minimum_donors)), + ("maximum_donor_weight", maximum_donor_weight), + ): + value = _finite_number(evidence.get(key), label=f"{stage}.{key}") + if value != expected: + failures.append( + f"{stage}: receipt {key} {value} differs from the gate's {expected}." + ) + scales: dict[str, float] = {} + for key in ("seating_scale", "mass_scale"): + scales[key] = _finite_number(evidence.get(key), label=f"{stage}.{key}") + if not 0.0 < scales[key] <= 1.0: + failures.append(f"{stage}: {key} {scales[key]} is outside (0, 1].") + full_seating = scales["seating_scale"] == 1.0 + full_scale = full_seating and scales["mass_scale"] == 1.0 + sample_fraction = _finite_number( + evidence.get("sample_fraction"), label=f"{stage}.sample_fraction" + ) + if sample_fraction == 1.0 and not full_scale: + failures.append( + f"{stage}: a full-sample build must seat and fund the donors at full " + f"scale, got seating_scale {scales['seating_scale']} and mass_scale " + f"{scales['mass_scale']}." + ) + + mass = _mapping(evidence.get("mass"), label=f"{stage}.mass") + old_total = _finite_number(mass.get("old_total"), label=f"{stage}.mass.old_total") + new_total = _finite_number(mass.get("new_total"), label=f"{stage}.mass.new_total") + if old_total <= 0.0 or new_total <= 0.0: + failures.append(f"{stage}: household mass must be positive before and after.") + deviation = abs(new_total - old_total) / max(abs(old_total), 1.0) + effective_tolerance = max(tolerance, _FLOAT_RELATIVE_TOLERANCE) + if deviation > effective_tolerance: + failures.append( + f"{stage}: household mass deviation {deviation} exceeds the effective " + f"tolerance {effective_tolerance}; the donors must be funded from the " + "incumbent households, not added." + ) + bands = evidence.get("bands") if not isinstance(bands, list | tuple): raise ValueError(f"{stage}.bands must be a list.") - failures: list[str] = [] seen: list[int] = [] + donor_total = 0 + donor_mass = 0.0 + heaviest_donor = 0.0 for row in bands: if not isinstance(row, Mapping): failures.append(f"{stage}: band row is not an object.") @@ -1168,8 +1389,22 @@ def _spi_income_band_donor_support_gate( row.get("donor_households"), label=f"{stage}.donor_households" ) ) + expected_donors = int( + _finite_number( + row.get("expected_donor_households"), + label=f"{stage}.expected_donor_households", + ) + ) carriers = int(_finite_number(row.get("carriers"), label=f"{stage}.carriers")) weight = _finite_number(row.get("donor_weight"), label=f"{stage}.donor_weight") + published = _finite_number( + row.get("published_taxpayers"), label=f"{stage}.published_taxpayers" + ) + donor_total += donors + donor_mass += weight * donors + heaviest_donor = max(heaviest_donor, weight) + if donors <= 0: + failures.append(f"{stage}: band from {lower} seats no donor.") if donors != carriers: failures.append( f"{stage}: band from {lower} has {donors} donors but {carriers} carriers." @@ -1178,26 +1413,77 @@ def _spi_income_band_donor_support_gate( failures.append( f"{stage}: band from {lower} donor weight {weight} is not positive." ) - published = _finite_number( - row.get("published_taxpayers"), label=f"{stage}.published_taxpayers" - ) - if abs(weight * donors - published) > 0.5 and donors > 0: - # A scaled rung stacks fewer donors than the declared count; its - # weighted taxpayers then fall short of the published band mass by - # construction, which the evidence still has to show honestly. - if donors == donors_per_band: - failures.append( - f"{stage}: band from {lower} weighted taxpayers " - f"{weight * donors} differ from the published {published}." - ) + if weight > maximum_donor_weight * (1.0 + _FLOAT_RELATIVE_TOLERANCE): + failures.append( + f"{stage}: band from {lower} donor weight {weight} exceeds the " + f"maximum {maximum_donor_weight}." + ) + if full_seating and (donors != expected_donors or donors < minimum_donors): + failures.append( + f"{stage}: band from {lower} seats {donors} donors at full scale; " + f"the plan seats {expected_donors} and the minimum is " + f"{minimum_donors}." + ) + # A scaled frame seats fewer donors or lighter ones, so its bands fall + # short of the published mass by construction; the receipt shows it. + if full_scale and abs(weight * donors - published) > taxpayer_tolerance: + failures.append( + f"{stage}: band from {lower} weighted taxpayers " + f"{weight * donors} differ from the published {published}." + ) if sorted(seen) != sorted(expected_bands): failures.append( f"{stage}: bands {sorted(seen)} differ from the declared {sorted(expected_bands)}." ) + donor_count = int( + _finite_number(evidence.get("donor_count"), label=f"{stage}.donor_count") + ) + if donor_count != donor_total: + failures.append( + f"{stage}: receipt donor_count {donor_count} differs from the bands' " + f"{donor_total}." + ) + + funding = evidence.get("funding") + if not isinstance(funding, list | tuple) or not funding: + raise ValueError(f"{stage}.funding must be a non-empty list.") + funded_mass = 0.0 + smallest_factor = 1.0 + for row in funding: + if not isinstance(row, Mapping): + failures.append(f"{stage}: funding row is not an object.") + continue + factor = _finite_number(row.get("factor"), label=f"{stage}.funding.factor") + funded_mass += _finite_number( + row.get("donor_mass"), label=f"{stage}.funding.donor_mass" + ) + smallest_factor = min(smallest_factor, factor) + if not ( + minimum_funding_factor * (1.0 - _FLOAT_RELATIVE_TOLERANCE) <= factor <= 1.0 + ): + failures.append( + f"{stage}: funding stratum {row.get('stratum')!r} factor {factor} " + f"is outside [{minimum_funding_factor}, 1]." + ) + reallocated = _finite_number( + evidence.get("reallocated_mass"), label=f"{stage}.reallocated_mass" + ) + for label, value in (("bands", donor_mass), ("funding strata", funded_mass)): + if abs(value - reallocated) > effective_tolerance * max(abs(reallocated), 1.0): + failures.append( + f"{stage}: the {label} carry donor mass {value}, the receipt " + f"reallocated {reallocated}." + ) details = { "bands_checked": len(bands), - "donors_per_band": donors_per_band, - "evidence_donors_per_band": evidence.get("donors_per_band"), + "donor_count": donor_count, + "seating_scale": scales["seating_scale"], + "mass_scale": scales["mass_scale"], + "reallocated_mass": reallocated, + "relative_mass_deviation": deviation, + "heaviest_donor_weight": heaviest_donor, + "smallest_funding_factor": smallest_factor, + "funding_strata": len(funding), } return ( _fail(stage, check, failures, details) @@ -1301,24 +1587,157 @@ def _cgt_imputation_summary_gate( ) +def _cgt_residential_split_gate( + stage: str, + evidence: Mapping[str, object], + parameters: Mapping[str, object], +) -> GateResult: + """The residential split conserved mass and left Table 8a as identities. + + The split carries the solved residential probability as weight + (microcosm#1063): every household with liable gainers becomes arms whose + weights are products of ``p`` and ``1 - p``, so the residential count and + gains at design weights are the solved expectations in every gain band. + The gate holds the total household mass to the declared deviation, the + solve to its targets, the realised masses to the expectations (an + arithmetic identity), the arm weights to their products, and the number + of liable gainers per household to the declared ceiling. + """ + + check = "cgt_residential_split" + tolerance = _finite_number( + parameters["maximum_relative_mass_deviation"], + label=f"{stage}.maximum_relative_mass_deviation", + ) + max_solve_error = _finite_number( + parameters["maximum_solve_relative_error"], + label=f"{stage}.maximum_solve_relative_error", + ) + max_identity_error = _finite_number( + parameters["maximum_identity_relative_error"], + label=f"{stage}.maximum_identity_relative_error", + ) + maximum_gainers = int(parameters["maximum_liable_gainers_per_household"]) + effective_tolerance = max(tolerance, _FLOAT_RELATIVE_TOLERANCE) + failures: list[str] = [] + details: dict[str, object] = {} + + mass = _mapping(evidence.get("mass"), label=f"{stage}.mass") + old_total = _finite_number(mass.get("old_total"), label=f"{stage}.mass.old_total") + new_total = _finite_number(mass.get("new_total"), label=f"{stage}.mass.new_total") + if old_total <= 0.0 or new_total <= 0.0: + failures.append(f"{stage}: household mass must be positive before and after.") + deviation = abs(new_total - old_total) / max(abs(old_total), 1.0) + details["relative_mass_deviation"] = deviation + if deviation > effective_tolerance: + failures.append( + f"{stage}: household mass deviation {deviation} exceeds the effective " + f"tolerance {effective_tolerance}." + ) + if evidence.get("arm_weights_exact") is not True: + failures.append(f"{stage}: the arm weights are not the declared products.") + by_k = _mapping( + evidence.get("households_by_liable_gainers"), + label=f"{stage}.households_by_liable_gainers", + ) + arms_expected = 0 + for key, count in by_k.items(): + k = int(key) + households = int(count) + if k < 1 or households < 0: + failures.append( + f"{stage}: households_by_liable_gainers[{key!r}] is invalid." + ) + continue + if k > maximum_gainers: + failures.append( + f"{stage}: {households} household(s) carry {k} liable gainers, " + f"above the declared maximum {maximum_gainers}." + ) + arms_expected += households * (2**k - 1) + arms_created = int(evidence.get("arms_created", -1)) + details["arms_created"] = arms_created + if arms_created != arms_expected: + failures.append( + f"{stage}: {arms_created} arms created, the households by liable " + f"gainers imply {arms_expected}." + ) + identities = _mapping(evidence.get("identities"), label=f"{stage}.identities") + + def number(key: str) -> float: + return _finite_number(identities.get(key), label=f"{stage}.identities.{key}") + + for measure in ("count", "gains"): + target = number(f"{measure}_target_individuals_basis") + if target <= 0.0: + failures.append(f"{stage}: residential {measure} target is not positive.") + continue + solve_error = number(f"{measure}_solve_relative_error") + identity_error = number(f"{measure}_identity_relative_error") + details[f"residential_{measure}_solve_relative_error"] = solve_error + details[f"residential_{measure}_identity_relative_error"] = identity_error + if solve_error > max_solve_error: + failures.append( + f"{stage}: residential {measure} solve error {solve_error} " + f"exceeds {max_solve_error}." + ) + if identity_error > max_identity_error: + failures.append( + f"{stage}: residential {measure} at design weights departs from " + f"its expectation by {identity_error}, above {max_identity_error}." + ) + bands = evidence.get("bands") + if not isinstance(bands, list) or not bands: + failures.append(f"{stage}: the receipt carries no gain bands.") + bands = [] + for position, band in enumerate(bands): + label = f"{stage}.bands[{position}]" + row = _mapping(band, label=label) + for measure in ("count", "gains"): + expected = _finite_number( + row.get(f"expected_{measure}"), label=f"{label}.expected_{measure}" + ) + achieved = _finite_number( + row.get(f"achieved_{measure}"), label=f"{label}.achieved_{measure}" + ) + if abs(achieved - expected) > max_identity_error * max(abs(expected), 1.0): + failures.append( + f"{stage}: residential {measure} in the gain band from " + f"{row.get('gain_lower_bound')} is {achieved} against the " + f"expectation {expected}." + ) + details["gain_bands"] = len(bands) + arm_weights = _mapping(evidence.get("arm_weights"), label=f"{stage}.arm_weights") + for key in ("residential_arms", "below_one_household", "minimum"): + details[f"arm_{key}"] = arm_weights.get(key) + concentration = _mapping( + evidence.get("concentration"), label=f"{stage}.concentration" + ) + for key in ( + "top_arms_share_of_achieved_gains", + "largest_liable_stake_share_of_gains_target", + ): + details[key] = concentration.get(key) + return ( + _fail(stage, check, failures, details) + if failures + else _pass(stage, check, details) + ) + + def _cgt_asset_type_summary_gate( stage: str, evidence: Mapping[str, object], parameters: Mapping[str, object], ) -> GateResult: - """The residential flag and the BADR claims realised the totals they were solved to. - - The stage solves the logistic exactly in expectation and realises it by - systematic sampling. This gate holds the solve to its targets, the - realised weighted count to within one carrier row's weight of the - expectation (``max_liable_weight``, the weighted systematic walk's - deterministic bound; about 3,300 people at full scale), and the realised - gains to the wider of the reviewed relative band and a multiple of the - Bernoulli sigma the stage reports, which overstates a systematic draw's - noise and so is a conservative envelope, so a tiny frame is judged by its - noise floor and a production frame by the band. Every liable gainer must - carry an asset type and the composition receipt must be finite - (microcosm#725). + """The residential arms carry Table 8a and the BADR claims their bands. + + The residential flag is no longer drawn here: ``cgt_residential_split`` + carries it as weight (microcosm#1063), so this gate holds the residential + count and gains the arms carry at design weights to the Table 8a targets + on the individuals basis within the solve tolerance, and requires every + gain band row of the receipt to be finite. Every liable gainer must carry + an asset type and the composition receipt must be finite (microcosm#725). Each Table 4.1 BADR band is held to deterministic bounds only (microcosm#1014): the solve to the band's targets; the realised count to @@ -1333,13 +1752,6 @@ def _cgt_asset_type_summary_gate( check = "cgt_asset_type_summary" failures: list[str] = [] residential = _mapping(evidence.get("residential"), label=f"{stage}.residential") - max_relative = _finite_number( - parameters["maximum_relative_deviation"], - label=f"{stage}.maximum_relative_deviation", - ) - max_sigma = _finite_number( - parameters["maximum_gains_sigma"], label=f"{stage}.maximum_gains_sigma" - ) max_solve_error = _finite_number( parameters["maximum_solve_relative_error"], label=f"{stage}.maximum_solve_relative_error", @@ -1349,39 +1761,45 @@ def _cgt_asset_type_summary_gate( def number(key: str) -> float: return _finite_number(residential.get(key), label=f"{stage}.residential.{key}") + if residential.get("source_stage") != "cgt_residential_split": + failures.append( + f"{stage}: the residential receipt does not name cgt_residential_split " + "as its source." + ) for measure in ("count", "gains"): target = number(f"{measure}_target_individuals_basis") - expected = number(f"expected_{measure}") + achieved = number(f"achieved_{measure}") if target <= 0.0: failures.append(f"{stage}: residential {measure} target is not positive.") continue - solve_error = abs(expected - target) / target - details[f"residential_{measure}_solve_relative_error"] = solve_error - if solve_error > max_solve_error: + error = abs(achieved - target) / target + details[f"residential_{measure}_relative_error"] = error + if error > max_solve_error: failures.append( - f"{stage}: residential {measure} solve error {solve_error} " - f"exceeds {max_solve_error}." + f"{stage}: residential {measure} on the arms is {achieved} against " + f"the target {target} (relative error {error}, tolerance " + f"{max_solve_error})." ) - count_gap = abs(number("achieved_count") - number("expected_count")) - count_bound = number("max_liable_weight") * (1.0 + 1e-9) - details["residential_count_gap"] = count_gap - if count_gap > count_bound: - failures.append( - f"{stage}: residential count gap {count_gap} exceeds one person " - f"({count_bound})." - ) - gains_target = number("gains_target_individuals_basis") - gains_gap = abs(number("achieved_gains") - number("expected_gains")) - gains_bound = max( - max_relative * gains_target, max_sigma * number("gains_bernoulli_sigma") - ) - details["residential_gains_gap"] = gains_gap - details["residential_gains_bound"] = gains_bound - if gains_target > 0.0 and gains_gap > gains_bound: - failures.append( - f"{stage}: residential gains gap {gains_gap} exceeds {gains_bound} " - f"(the wider of {max_relative} relative and {max_sigma} sigma)." - ) + bands = residential.get("bands") + if not isinstance(bands, list) or not bands: + failures.append(f"{stage}: residential receipt carries no gain bands.") + bands = [] + band_totals = {"achieved_count": 0.0, "achieved_gains": 0.0} + for position, band in enumerate(bands): + label = f"{stage}.residential.bands[{position}]" + row = _mapping(band, label=label) + for measure in band_totals: + band_totals[measure] += _finite_number( + row.get(measure), label=f"{label}.{measure}" + ) + for measure, total in band_totals.items(): + expected = number(measure) + if bands and abs(total - expected) > max_solve_error * max(abs(expected), 1.0): + failures.append( + f"{stage}: the gain bands' {measure} sums to {total}, not the " + f"residential receipt's {expected}." + ) + details["residential_gain_bands"] = len(bands) counts = _mapping(evidence.get("value_counts"), label=f"{stage}.value_counts") for value, rows in counts.items(): if not isinstance(rows, int) or isinstance(rows, bool) or rows < 0: @@ -1588,7 +2006,10 @@ def _cgt_incidence_anchor_gate( failures.append( f"{stage}: liable clone mass moved from {liable_before} to {liable_after}." ) - if not np.isclose(removed, transferred, rtol=1e-9, atol=0.0): + # Both masses are sums of the same weights; a build whose removed group is + # empty can still carry float-epsilon residue, so the comparison allows + # the absolute rounding tolerance the other mass identities use. + if not np.isclose(removed, transferred, rtol=1e-9, atol=1e-6): failures.append( f"{stage}: removed group mass {removed} disagrees with the transferred " f"mass {transferred}." @@ -1715,6 +2136,107 @@ def _binomial_tolerance(*, target: float, rows: int) -> float: return latent_attribute_tolerance(target=target, rows=rows) +#: Receipt counts the WAS wealth stage must report as zero (microcosm#1063): +#: the two tenure rules, and one count per accounting identity. +_WEALTH_TENURE_ZERO_KEYS = ( + "main_residence_mortgage_off_mortgaged_tenure_rows", + "main_residence_value_off_owner_tenure_rows", +) +_WEALTH_IDENTITY_ZERO_KEYS = ( + "property_wealth_violation_rows", + "corporate_wealth_violation_rows", + "gross_financial_wealth_violation_rows", + "net_financial_wealth_violation_rows", + "mortgage_debt_violation_rows", +) + + +def _wealth_coherence_gate( + stage: str, + evidence: Mapping[str, object], + parameters: Mapping[str, object], +) -> GateResult: + """The WAS wealth columns are coherent with tenure and with each other. + + The stage draws the main residence and the main-residence mortgage + inside their tenure stratum and derives every total from drawn + components (mortgage debt included: the main-residence mortgage plus the + mortgages on other property, which any tenure may hold); this gate + re-reads the receipt so the battery, not only the transform, holds that + no household off a mortgaged tenure carries a main-residence mortgage, + none off an owner tenure carries a main-residence value, every total + equals its components, and owners without a main-residence value are no + more common than on the donor beyond the reviewed excess. + """ + + check = "wealth_coherence" + failures: list[str] = [] + details: dict[str, object] = {} + blocks: dict[str, Mapping[str, object]] = {} + for block in ("tenure_coherence", "identities"): + value = evidence.get(block) + if not isinstance(value, Mapping): + failures.append(f"{stage}: receipt is missing {block}.") + continue + blocks[block] = value + for block, keys in ( + ("tenure_coherence", _WEALTH_TENURE_ZERO_KEYS), + ("identities", _WEALTH_IDENTITY_ZERO_KEYS), + ): + receipt = blocks.get(block) + if receipt is None: + continue + for key in keys: + value = receipt.get(key) + if isinstance(value, bool) or not isinstance(value, int | float): + failures.append(f"{stage}: {block} receipt is missing {key}.") + continue + details[key] = int(value) + if int(value) != 0: + failures.append(f"{stage}: {key} is {int(value)}, expected 0.") + tenure = blocks.get("tenure_coherence") + if tenure is not None: + maximum_excess = _finite_number( + parameters["maximum_owner_share_without_main_residence_excess"], + label=f"{stage}.maximum_owner_share_without_main_residence_excess", + ) + try: + share = _finite_number( + tenure.get("owner_share_without_main_residence_value"), + label=f"{stage}.owner_share_without_main_residence_value", + ) + donor_share = _finite_number( + tenure.get("donor_owner_share_without_main_residence_value"), + label=f"{stage}.donor_owner_share_without_main_residence_value", + ) + except ValueError as error: + failures.append(str(error)) + else: + details["owner_share_without_main_residence_value"] = share + details["donor_owner_share_without_main_residence_value"] = donor_share + details["maximum_owner_share_without_main_residence_excess"] = ( + maximum_excess + ) + if share - donor_share > maximum_excess: + failures.append( + f"{stage}: {share:.6g} of owner households carry no " + f"main-residence value against {donor_share:.6g} on the " + f"donor, above the reviewed excess {maximum_excess:.6g}." + ) + identities = blocks.get("identities") + if identities is not None: + details["capped_rows"] = { + str(key): int(value) + for key, value in identities.items() + if str(key).endswith("_rows") + and "capped" in str(key) + and isinstance(value, int | float) + } + if failures: + return _fail(stage, check, failures, details) + return _pass(stage, check, details) + + def _household_composition_gate( stage: str, evidence: Mapping[str, object], diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/student_loans.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/student_loans.py index ea06933ba..596985e00 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/student_loans.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/student_loans.py @@ -320,10 +320,12 @@ def assign_student_loan_plans( #: Household lineage columns the top-up receipt reports the taken rows on, #: when the frame carries them: the SPI support channel, the CGT support -#: split's copy flag and the incidence clone flag. +#: split's copy flag, the incidence clone flag and the residential split's +#: arm flag (microcosm#1063). _LINEAGE_CHANNEL_COLUMN = "household_support_channel" _LINEAGE_SUPPORT_COPY_COLUMN = "household_is_cgt_support_copy" _LINEAGE_CLONE_COLUMN = "household_is_capital_gains_clone" +_LINEAGE_RESIDENTIAL_ARM_COLUMN = "household_is_cgt_residential_clone" def _person_lineage(person: pd.DataFrame, household: pd.DataFrame) -> pd.DataFrame: @@ -335,6 +337,7 @@ def _person_lineage(person: pd.DataFrame, household: pd.DataFrame) -> pd.DataFra _LINEAGE_CHANNEL_COLUMN, _LINEAGE_SUPPORT_COPY_COLUMN, _LINEAGE_CLONE_COLUMN, + _LINEAGE_RESIDENTIAL_ARM_COLUMN, ): if column in by_household.columns: mapped = person["person_household_id"].map(by_household[column]) @@ -370,6 +373,7 @@ def _topped_up_lineage( for key, column in ( ("support_copy", _LINEAGE_SUPPORT_COPY_COLUMN), ("clone", _LINEAGE_CLONE_COLUMN), + ("residential_arm", _LINEAGE_RESIDENTIAL_ARM_COLUMN), ): if column in lineage.columns: flags = lineage[column].to_numpy(dtype=bool)[taken] diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/tenure_constants.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/tenure_constants.py new file mode 100644 index 000000000..c812e5a14 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/tenure_constants.py @@ -0,0 +1,46 @@ +"""UK tenure categories shared by the stages that group tenure four ways.""" + +from __future__ import annotations + +from collections.abc import Mapping +from types import MappingProxyType + +UK_TENURE_OWNED_OUTRIGHT = "owned_outright" +UK_TENURE_OWNED_WITH_MORTGAGE = "owned_with_mortgage" +UK_TENURE_PRIVATE_RENT = "private_rent" +UK_TENURE_SOCIAL_RENT = "social_rent" + +#: The four-way grouping in a fixed order: owner-occupiers without and with a +#: mortgage, private renters, and social renters (council and housing +#: association). +UK_TENURE_CATEGORIES: tuple[str, ...] = ( + UK_TENURE_OWNED_OUTRIGHT, + UK_TENURE_OWNED_WITH_MORTGAGE, + UK_TENURE_PRIVATE_RENT, + UK_TENURE_SOCIAL_RENT, +) +UK_OWNER_TENURE_CATEGORIES: tuple[str, ...] = ( + UK_TENURE_OWNED_OUTRIGHT, + UK_TENURE_OWNED_WITH_MORTGAGE, +) + +#: The engine's ``tenure_type`` member names onto the four-way grouping. +UK_TENURE_TYPE_TO_CATEGORY: Mapping[str, str] = MappingProxyType( + { + "OWNED_OUTRIGHT": UK_TENURE_OWNED_OUTRIGHT, + "OWNED_WITH_MORTGAGE": UK_TENURE_OWNED_WITH_MORTGAGE, + "RENT_PRIVATELY": UK_TENURE_PRIVATE_RENT, + "RENT_FROM_COUNCIL": UK_TENURE_SOCIAL_RENT, + "RENT_FROM_HA": UK_TENURE_SOCIAL_RENT, + } +) + +__all__ = [ + "UK_OWNER_TENURE_CATEGORIES", + "UK_TENURE_CATEGORIES", + "UK_TENURE_OWNED_OUTRIGHT", + "UK_TENURE_OWNED_WITH_MORTGAGE", + "UK_TENURE_PRIVATE_RENT", + "UK_TENURE_SOCIAL_RENT", + "UK_TENURE_TYPE_TO_CATEGORY", +] diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/terminal_gates.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/terminal_gates.py index 2cb3ac46c..46e0deb37 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/terminal_gates.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/terminal_gates.py @@ -62,6 +62,7 @@ "uk_default_degenerate_reviewed_exclusions", "uk_default_target_fit_reviewed_exclusions", "uk_degenerate_release_surface_gate", + "uk_export_candidate_columns", "uk_export_surface_gate", "uk_input_mass_parity_gate", "uk_qrf_tail_concentration_gate", @@ -73,7 +74,7 @@ "uk_cgt_projection_entrants_gate", ] -UK_CANDIDATE_DATASET_NAME = "microcosm_uk_2024" +UK_CANDIDATE_DATASET_NAME = "microcosm_uk_2024_25" # The label names the pinned reference artifact exactly: the 2024-25 line's # published enhanced_frs_2024_25.h5 (no separate "recalibrated" variant # exists at this vintage; the June report strings keep their own label). @@ -180,6 +181,9 @@ def __post_init__(self) -> None: # #1045: the CGT support split's family copy count. "household.cgt_support_copies", "household.cgt_support_copy_index", + # #1063: the residential split's arm flag and index. + "household.household_is_cgt_residential_clone", + "household.cgt_residential_clone_index", "household.clone_index", "household.constituency_code_oa", "household.consumer_debt", @@ -235,6 +239,7 @@ def __post_init__(self) -> None: "person.capital_gains_asset_type", "person.capital_gains_badr", "person.capital_gains_residential_property", + "person.cgt_residential_probability", "person.bus_in_london_trips", "person.bus_pass_eligible", "person.care_hours", @@ -275,6 +280,57 @@ def __post_init__(self) -> None: "person.would_claim_marriage_allowance", "person.would_claim_scp", "person.person_is_spi_income_band_carrier", + # microcosm#1063 c9 (ruling 2026-10-02): every declared stage output the + # enhanced-FRS incumbent never carried is allow-listed rather than + # dropped, so the release candidate ships the spine's own surface. The + # certifier rehearsal R5 on the 2026-09-30 build named them: the source + # and support-channel lineage of the three entities, the council tax + # family (#934), the reported-benefit inputs the take-up stages read + # (the five internal disability carriers are not among them: they leave + # at the release boundary, UK_RELEASE_EXPORT_DROPPED_COLUMNS), + # the SPI channel's hmrc_spi_* leaves (#717), the FRS education and + # housing fields, the two raw FRS codes the spine fences + # (ossben_identifiable_subset, srp_regular_code5) and + # other_investment_income. Only incapacity_benefit_reported is dropped + # (UK_REVIEWED_EXPORT_EXCLUSIONS). + "benunit.benunit_source_id", + "benunit.benunit_support_channel", + "benunit.benunit_support_clone_index", + "benunit.dependent_children", + "household.council_tax_rebate", + "household.council_tax_reported", + "household.council_tax_single_adult_raw", + "household.household_source_id", + "household.household_support_channel", + "household.household_support_clone_index", + "household.num_bedrooms", + "household.source_household_id", + "household.source_household_key", + "household.source_year", + "household.subrent", + "person.disabled_students_allowance_eligible_expenses", + "person.free_school_breakfasts", + "person.hmrc_spi_assessable_income", + "person.hmrc_spi_employed_income", + "person.hmrc_spi_employment_benefits", + "person.hmrc_spi_employment_expenses", + "person.hmrc_spi_incapacity_benefit_income", + "person.hmrc_spi_miscellaneous_employment_income", + "person.hmrc_spi_other_income", + "person.hmrc_spi_other_social_security_income", + "person.hmrc_spi_pay", + "person.hmrc_spi_state_pension_income", + "person.hmrc_spi_taxable_termination_pay", + "person.hmrc_spi_total_earned_income", + "person.hmrc_spi_total_investment_income", + "person.hmrc_spi_unemployment_benefit_income", + "person.is_in_approved_training", + "person.ossben_identifiable_subset", + "person.other_investment_income", + "person.person_source_id", + "person.person_support_channel", + "person.person_support_clone_index", + "person.srp_regular_code5", ) UK_KNOWN_MISSING_REFERENCE_EXPORT_COLUMNS: tuple[str, ...] = ( @@ -318,6 +374,27 @@ def uk_default_target_fit_reviewed_exclusions() -> Mapping[str, UKReviewedExclus ) +def uk_export_candidate_columns(frame: Any) -> set[str]: + """The ``entity.column`` surface a frame exports, as the gate reads it. + + Structural id columns are not exported content (microcosm#1063 c9: the + certifier rehearsal listed every id as an unreviewed extra), and the + frame's household weights live beside the tables, so the surface carries + ``household.household_weight`` explicitly, as the enhanced-FRS reference + does. + """ + + columns: set[str] = set() + for entity in frame.entities: + structural = _STRUCTURAL_COLUMNS.get(str(entity), frozenset()) + for column in frame.table(entity).columns: + if column in structural: + continue + columns.add(f"{entity}.{column}") + columns.add(f"household.{_WEIGHT_COLUMN}") + return columns + + def _entity_tables(dataset: Any) -> tuple[tuple[str, pd.DataFrame], ...]: if isinstance(dataset, Mapping): raw = tuple((entity, dataset.get(entity)) for entity in _STRUCTURAL_COLUMNS) diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/was_wealth.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/was_wealth.py index b18882eb4..eb0980c3d 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/was_wealth.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/was_wealth.py @@ -5,6 +5,7 @@ from collections.abc import Mapping, Sequence from dataclasses import dataclass, field from pathlib import Path +from types import MappingProxyType from typing import Any import numpy as np @@ -26,6 +27,15 @@ UKSupportClipResult, support_clip_to_donor_with_receipt, ) +from microcosm.build.uk_runtime.tenure_constants import ( + UK_OWNER_TENURE_CATEGORIES, + UK_TENURE_CATEGORIES, + UK_TENURE_OWNED_OUTRIGHT, + UK_TENURE_OWNED_WITH_MORTGAGE, + UK_TENURE_PRIVATE_RENT, + UK_TENURE_SOCIAL_RENT, + UK_TENURE_TYPE_TO_CATEGORY, +) from microcosm.frame import Frame from microcosm.frame.rules import assert_rules_engine_country @@ -33,6 +43,16 @@ WAS_DONOR_SHA256 = "18b3eb980c02c99f3d8a3254af859bee31682b2bdc11703877677292b3ce9374" WAS_DONOR_SIZE_BYTES = 39_073_613 +#: The tenure category column both sides carry (never a predictor itself: the +#: strata read it and the one-hot flags below enter the forests). +UK_WAS_TENURE_CATEGORY_COLUMN = "tenure_category" +#: One flag per tenure category but the first (owned outright is the base +#: level), built the same way on the donor (``Ten1R8``, ``DVPriRntR8``) and on +#: the recipient (the frame's ``tenure_type``), so housing-association and +#: council renters are matched to social-renter donors (microcosm#1063). +UK_WAS_TENURE_PREDICTORS = tuple( + f"tenure_{category}" for category in UK_TENURE_CATEGORIES[1:] +) UK_WAS_WEALTH_PREDICTORS = ( "household_net_income", "num_adults", @@ -43,7 +63,7 @@ "capital_income", "num_bedrooms", "council_tax", - "is_renting", + *UK_WAS_TENURE_PREDICTORS, "region", ) UK_WAS_ENGINE_PREDICTORS = ( @@ -54,13 +74,68 @@ "employment_income", "self_employment_income", "capital_income", - "is_renting", ) -#: Extra base predictor of the debt chain segment only (segments 1-3 are -#: unchanged so E5's fourteen columns stay byte-equal); declared in the manifest -#: as ``debt_segment_predictors`` and drift-asserted at run time. -UK_WAS_DEBT_SEGMENT_PREDICTORS = ("has_mortgage_tenure",) UK_WAS_DEBT_OUTPUT_COLUMNS = ("mortgage_debt", "consumer_debt") +#: Columns drawn only inside a tenure stratum and zero by rule outside it +#: (microcosm#1063): the main residence exists only for owner-occupiers, and +#: the mortgage on it only on a mortgaged tenure. Each maps to the tenure +#: categories of its stratum; the manifest declares the same mapping as +#: ``stratified_targets`` and the run asserts it. ``mortgage_debt`` itself is +#: not stratified: it is the main-residence mortgage plus the mortgages on +#: other property, which any tenure may hold (review of #1089, item 3). +UK_WAS_STRATIFIED_TARGETS: Mapping[str, tuple[str, ...]] = MappingProxyType( + { + "main_residence_value": UK_OWNER_TENURE_CATEGORIES, + "main_residence_mortgage": (UK_TENURE_OWNED_WITH_MORTGAGE,), + } +) +#: Components the chain draws but the stage does not emit: the remainder of a +#: published total over the components the engine reads, so every total is the +#: sum of its drawn parts rather than a draw of its own. +UK_WAS_OTHER_PROPERTY_COLUMN = "other_property_value" +UK_WAS_OTHER_FINANCIAL_COLUMN = "other_financial_assets" +#: The household's mortgages other than the one on its main residence +#: (``HMortGR8`` less ``TotMortR8`` on the donor): buy-to-let, second homes +#: and the rest, held on every tenure, drawn without a stratum. +UK_WAS_OTHER_MORTGAGE_COLUMN = "other_mortgage" +UK_WAS_INTERNAL_COMPONENT_COLUMNS = ( + UK_WAS_OTHER_PROPERTY_COLUMN, + UK_WAS_OTHER_FINANCIAL_COLUMN, + UK_WAS_OTHER_MORTGAGE_COLUMN, +) +#: Totals derived from drawn components, in derivation order. WAS round 8 +#: satisfies each identity on every donor row (checked when the donor is +#: cleaned): DVPropertyR8 is the sum of the property values, HFINWR8_SUM is +#: at least the listed financial assets, and net financial wealth is gross +#: less the financial liabilities. +UK_WAS_DERIVED_TOTALS: Mapping[str, tuple[str, ...]] = MappingProxyType( + { + "property_wealth": ( + "owned_land", + "main_residence_value", + "other_residential_property_value", + "non_residential_property_value", + UK_WAS_OTHER_PROPERTY_COLUMN, + ), + "corporate_wealth": ("corporate_wealth_excl_isa", "stocks_and_shares_isa"), + "gross_financial_wealth": ( + "savings", + "cash_isa", + "corporate_wealth", + UK_WAS_OTHER_FINANCIAL_COLUMN, + ), + # HMortGR8 is the main-residence mortgage (TotMortR8, zero off the + # mortgaged tenure) plus the mortgages on other property, held on any + # tenure; the engine's mortgage_debt is their sum. + "mortgage_debt": ("main_residence_mortgage", UK_WAS_OTHER_MORTGAGE_COLUMN), + } +) +#: ``net_financial_wealth = gross_financial_wealth - consumer_debt - +#: student_loan_balance`` (the liabilities are subtracted, so it is declared +#: apart from the sums above). +UK_WAS_NET_FINANCIAL_LIABILITIES = ("consumer_debt", "student_loan_balance") +#: Absolute tolerance, in pounds, of the donor identity checks. +UK_WAS_IDENTITY_TOLERANCE_GBP = 1.0 UK_WAS_WEALTH_OUTPUT_COLUMNS = ( "owned_land", "property_wealth", @@ -115,9 +190,26 @@ "employment_income": "person", "self_employment_income": "person", "capital_income": "person", - "is_renting": "household", } +#: WAS round-8 ``Ten1R8`` owner codes onto the shared tenure categories: 1 owns +#: outright, 2 is buying with a mortgage and 3 is part rent, part mortgage +#: (UKDS SN 7215 data dictionary). Codes 4 (rented) and 5 (rent-free) split on +#: ``DVPriRntR8``: 1 is a private renting household, anything else rents from a +#: council or housing association (the 1,726 rented rows with ``DVPriRntR8`` +#: 2 are exactly the ``LLordR8`` 1 and 2 rows of the pinned tab). Any other +#: code is unclassified, counted in the receipt and read as private rent, the +#: FRS spine's own fallback for an unknown tenure. +WAS_OWNER_TENURE_CODES: Mapping[int, str] = MappingProxyType( + { + 1: UK_TENURE_OWNED_OUTRIGHT, + 2: UK_TENURE_OWNED_WITH_MORTGAGE, + 3: UK_TENURE_OWNED_WITH_MORTGAGE, + } +) +WAS_RENTED_TENURE_CODES: tuple[int, ...] = (4, 5) +WAS_PRIVATE_RENT_CODE = 1 + #: UKDS negative sentinel codes observed in the round-8 household tab. #: Recoded to zero ONLY for columns whose domain cannot be negative and #: which the 2026-08-19 licensed audit found carrying them: vcarnr8 (2 rows @@ -125,7 +217,7 @@ #: unasked in the WAS household file; the predictor-quality question is #: registered for the end-of-workstream revisit on microcosm#145). #: DVPriRntR8's -9 is structural not-applicable (not a private renter), so -#: the is_renting == 1 mapping is already correct; genuinely negative +#: the private-renter reading (code 1) is already correct; genuinely negative #: domains (net financial wealth, self-employment losses, BHC income) are #: never recoded. Signed difference vs the incumbent, which trains on the #: raw sentinel values. @@ -168,6 +260,11 @@ "HFINWNTR8_exSLC_Sum": "net_financial_wealth_exsl", "HFINWR8_SUM": "gross_financial_wealth", "HMortGR8": "mortgage_debt", + # Diagnostics only (microcosm#1063): the main-residence mortgage and the + # debt on other property, so the receipt can say what the tenure rule on + # ``mortgage_debt`` leaves out. Neither is a predictor or an output. + "TotMortR8": "main_residence_mortgage", + "OthMortR8_sum": "other_property_mortgage", "Ten1R8": "tenure_code", "DVhvalueR8": "main_residence_value", "DVHseValR8_sum": "other_residential_property_value", @@ -182,15 +279,24 @@ @dataclass class UKWASWealthResult: - """Transformed frame and donor-support clip receipt.""" + """Transformed frame, donor-support clip receipt and coherence receipts.""" frame: Frame support_clip: UKSupportClipReceipt + #: Tenure coherence of the housing columns: the two structural-zero counts + #: the stage-health gate requires to be zero, and the donor comparisons. + tenure_coherence: Mapping[str, object] = field(default_factory=dict) + #: Rows on which a derived total departs from its components (zero by + #: construction unless the support clip moved a total) and the rows the + #: donor-range cap adjusted. + identities: Mapping[str, object] = field(default_factory=dict) def evidence(self) -> dict[str, object]: return { "stage": UK_WAS_WEALTH_STAGE_NAME, "support_clip": self.support_clip.evidence(), + "tenure_coherence": dict(self.tenure_coherence), + "identities": dict(self.identities), } @@ -240,7 +346,7 @@ def __call__(self, frame: Frame) -> Frame: ) ) ) - _assert_debt_segment_predictors(self.stage) + _assert_chain_declaration(self.stage) household_predictors = recipient_predictors(frame, self.engine) imputation = impute_was_wealth( donor, @@ -249,11 +355,22 @@ def __call__(self, frame: Frame) -> Frame: n_estimators=_qrf_n_estimators(self.stage), ) self.last_fit_weight_records = imputation.fit_weight_records - clip_result = support_clip_to_donor(imputation.draws, donor) + capped, capped_rows = cap_derived_totals_to_donor_range(imputation.draws, donor) + clip_result = support_clip_to_donor(capped, donor) household_draws = clip_result.clipped + identities = { + **wealth_identity_violations(household_draws), + **capped_rows, + } household_draws["num_vehicles"] = ( np.rint(household_draws["num_vehicles"]).clip(lower=0).astype("int64") ) + tenure_coherence = tenure_coherence_receipt( + donor=donor, + recipient_category=household_predictors[UK_WAS_TENURE_CATEGORY_COLUMN], + recipient_draws=household_draws, + recipient_weights=frame.weights_for("household").values, + ) person = frame.table("person").copy() household = frame.table("household").copy() for column in UK_WAS_WEALTH_HOUSEHOLD_OUTPUT_COLUMNS: @@ -281,6 +398,8 @@ def __call__(self, frame: Frame) -> Frame: self.last_result = UKWASWealthResult( frame=result, support_clip=clip_result.receipt, + tenure_coherence=tenure_coherence, + identities=identities, ) return result @@ -317,7 +436,10 @@ def clean_was_household_table(raw: pd.DataFrame) -> pd.DataFrame: for column in _SENTINEL_RECODE_COLUMNS: values = cleaned[column] cleaned[column] = values.where(~values.isin(_SENTINEL_CODES), 0) - cleaned["is_renting"] = cleaned["private_rent_code"] == 1 + # Private renting households, the flag the Lifetime ISA stage reads on + # both sides (``was_lisa``); this stage's own tenure predictors are the + # four-way flags below. + cleaned["is_renting"] = cleaned["private_rent_code"] == WAS_PRIVATE_RENT_CODE # Private pension wealth other than current-employment defined-benefit # entitlements (WAS total private pension wealth less DVValDBT_SCAPE; # defined-benefit-type components are valued at the SCAPE discount rate, @@ -344,20 +466,116 @@ def clean_was_household_table(raw: pd.DataFrame) -> pd.DataFrame: cleaned["consumer_debt"] = ( cleaned["gross_financial_wealth"] - cleaned["net_financial_wealth_exsl"] ).clip(lower=0) - cleaned["has_mortgage_tenure"] = cleaned["tenure_code"].isin({2, 3}) + cleaned[UK_WAS_TENURE_CATEGORY_COLUMN] = was_tenure_category( + cleaned["tenure_code"], cleaned["private_rent_code"] + ) + cleaned["tenure_code_unclassified"] = ~cleaned["tenure_code"].isin( + {*WAS_OWNER_TENURE_CODES, *WAS_RENTED_TENURE_CODES} + ) + for predictor in UK_WAS_TENURE_PREDICTORS: + cleaned[predictor] = cleaned[UK_WAS_TENURE_CATEGORY_COLUMN] == ( + predictor.removeprefix("tenure_") + ) + cleaned[UK_WAS_OTHER_PROPERTY_COLUMN] = _donor_remainder( + cleaned, + total="property_wealth", + components=UK_WAS_DERIVED_TOTALS["property_wealth"][:-1], + ) + cleaned[UK_WAS_OTHER_FINANCIAL_COLUMN] = _donor_remainder( + cleaned, + total="gross_financial_wealth", + components=UK_WAS_DERIVED_TOTALS["gross_financial_wealth"][:-1], + ) + cleaned[UK_WAS_OTHER_MORTGAGE_COLUMN] = _donor_remainder( + cleaned, + total="mortgage_debt", + components=UK_WAS_DERIVED_TOTALS["mortgage_debt"][:-1], + ) + _assert_donor_net_financial_identity(cleaned) cleaned["region"] = cleaned["region_code"].map(REGIONS) return cleaned[ [ *UK_WAS_WEALTH_PREDICTORS, "weight", + UK_WAS_TENURE_CATEGORY_COLUMN, + "tenure_code_unclassified", + "is_renting", "corporate_wealth_excl_isa", - "has_mortgage_tenure", + *UK_WAS_INTERNAL_COMPONENT_COLUMNS, + "main_residence_mortgage", + "other_property_mortgage", *UK_WAS_WEALTH_HOUSEHOLD_OUTPUT_COLUMNS, "student_loan_balance", ] ] +def was_tenure_category( + tenure_code: pd.Series, private_rent_code: pd.Series +) -> pd.Series: + """The donor's tenure category from ``Ten1R8`` and ``DVPriRntR8``.""" + + code = pd.to_numeric(tenure_code, errors="coerce") + private = pd.to_numeric(private_rent_code, errors="coerce") == WAS_PRIVATE_RENT_CODE + category = pd.Series(UK_TENURE_PRIVATE_RENT, index=tenure_code.index, dtype=object) + for owner_code, owner_category in WAS_OWNER_TENURE_CODES.items(): + category[code == owner_code] = owner_category + rented = code.isin(WAS_RENTED_TENURE_CODES) + category[rented & ~private] = UK_TENURE_SOCIAL_RENT + return category + + +def recipient_tenure_category(tenure_type: pd.Series) -> pd.Series: + """The recipient's tenure category from the frame's ``tenure_type``.""" + + names = tenure_type.map(_enum_name) + unknown = sorted(set(names) - set(UK_TENURE_TYPE_TO_CATEGORY)) + if unknown: + raise ValueError( + f"recipient tenure_type carries values outside the engine's enum: {unknown}." + ) + return names.map(UK_TENURE_TYPE_TO_CATEGORY) + + +def _donor_remainder( + cleaned: pd.DataFrame, *, total: str, components: Sequence[str] +) -> pd.Series: + """A published total less its listed components, refused when negative. + + The remainder becomes a drawn component, so a donor row whose components + exceed its total would make the total underivable from its parts. + """ + + remainder = cleaned[total] - cleaned[list(components)].sum(axis=1) + short = remainder < -UK_WAS_IDENTITY_TOLERANCE_GBP + if short.any(): + raise ValueError( + f"WAS donor {total} is below the sum of {list(components)} on " + f"{int(short.sum())} row(s); the total cannot be derived from its " + "components." + ) + return remainder.clip(lower=0.0) + + +def _assert_donor_net_financial_identity(cleaned: pd.DataFrame) -> None: + """``net = gross - consumer_debt - student_loan_balance`` on every donor row.""" + + derived = ( + cleaned["gross_financial_wealth"] + - cleaned["consumer_debt"] + - cleaned["student_loan_balance"] + ) + off = (derived - cleaned["net_financial_wealth"]).abs() > ( + UK_WAS_IDENTITY_TOLERANCE_GBP + ) + if off.any(): + raise ValueError( + "WAS donor net financial wealth is not gross financial wealth less " + f"consumer debt and the student loan balance on {int(off.sum())} " + "row(s)." + ) + + def recipient_predictors(frame: Frame, engine: object) -> pd.DataFrame: """Materialize the WAS predictor surface on recipient households. @@ -416,13 +634,13 @@ def recipient_predictors(frame: Frame, engine: object) -> pd.DataFrame: raise KeyError(f"recipient household table is missing {predictor!r}.") result[predictor] = household[predictor].to_numpy() result["region"] = result["region"].map(_enum_name).replace(REGION_REMAP) - result["is_renting"] = result["is_renting"].astype(bool) if "tenure_type" not in household.columns: raise KeyError("recipient household table is missing 'tenure_type'.") - result["has_mortgage_tenure"] = ( - household["tenure_type"].map(_enum_name) == "OWNED_WITH_MORTGAGE" - ) - return result.loc[:, (*UK_WAS_WEALTH_PREDICTORS, "has_mortgage_tenure")] + category = recipient_tenure_category(household["tenure_type"]) + for predictor in UK_WAS_TENURE_PREDICTORS: + result[predictor] = (category == predictor.removeprefix("tenure_")).to_numpy() + result[UK_WAS_TENURE_CATEGORY_COLUMN] = category.to_numpy() + return result.loc[:, (*UK_WAS_WEALTH_PREDICTORS, UK_WAS_TENURE_CATEGORY_COLUMN)] @dataclass(frozen=True) @@ -435,7 +653,17 @@ class UKWASWealthImputationResult: segment_seeds: tuple[int, ...] = () -def was_wealth_segment_seeds(seed: int, segments: int = 4) -> tuple[int, ...]: +#: The chain's segments, each with its own model and child seed: +#: 1 land; 2 main residence (owner stratum); 3 the other property components; +#: 4 pension and share-like wealth; 5 the financial components, vehicles and +#: the student loan balance; 6 mortgage debt (mortgaged stratum); 7 consumer +#: debt. Totals are derived between segments from the components drawn so far. +UK_WAS_CHAIN_SEGMENTS = 8 + + +def was_wealth_segment_seeds( + seed: int, segments: int = UK_WAS_CHAIN_SEGMENTS +) -> tuple[int, ...]: """Derive one independent RNG root per chain segment from the stage seed. :meth:`RegimeGatedQRF.start_chain` spawns its fit and draw streams from @@ -445,7 +673,8 @@ def was_wealth_segment_seeds(seed: int, segments: int = 4) -> tuple[int, ...]: licensed donor that coupling collapsed P(shares > 0 | property_wealth = 0) to 0.011 against 0.055 observed; the production child seeds recover 0.039 (hold-out receipt re-run with exactly this derivation). The declared stage - seed stays the root and the children are deterministic. + seed stays the root and the children are deterministic; ``spawn`` is + prefix-stable, so the first ``k`` seeds do not depend on ``segments``. """ return tuple( @@ -454,9 +683,6 @@ def was_wealth_segment_seeds(seed: int, segments: int = 4) -> tuple[int, ...]: ) -UK_WAS_CHAIN_SEGMENTS = 4 - - def impute_was_wealth( donor: pd.DataFrame, recipient_predictor_frame: pd.DataFrame, @@ -467,10 +693,16 @@ def impute_was_wealth( ) -> UKWASWealthImputationResult: """Fit segmented checkpointed QRF chains and draw WAS wealth outputs. - ``segments`` runs only the first *k* chain segments (a test seam: the - fourth, debt, segment draws from the fourth child seed, so the first three - segments' draws are identical whether or not it runs, and the E5 columns - stay byte-equal by construction). + The targets of ``UK_WAS_STRATIFIED_TARGETS`` are fitted on the donors of + their tenure stratum and drawn for the recipients of the same stratum + only; every other recipient holds zero by rule. The totals of + ``UK_WAS_DERIVED_TOTALS`` and ``net_financial_wealth`` are never drawn: + each is computed from its drawn components, so the accounting identities + the donor satisfies hold on every recipient. + + ``segments`` runs only the first *k* chain segments (a test seam: each + segment draws from its own child seed, so the earlier segments' draws are + identical whether or not the later ones run). """ from microcosm.fit import RegimeGatedQRF @@ -479,10 +711,25 @@ def impute_was_wealth( raise ValueError( f"segments must be between 1 and {UK_WAS_CHAIN_SEGMENTS}, got {segments!r}." ) + for table, side in ((donor, "donor"), (recipient_predictor_frame, "recipient")): + if UK_WAS_TENURE_CATEGORY_COLUMN not in table.columns: + raise KeyError( + f"WAS {side} table is missing {UK_WAS_TENURE_CATEGORY_COLUMN!r}." + ) + donor_category = donor[UK_WAS_TENURE_CATEGORY_COLUMN].astype(str).to_numpy() + recipient_category = ( + recipient_predictor_frame[UK_WAS_TENURE_CATEGORY_COLUMN].astype(str).to_numpy() + ) donor_encoded, recipient_encoded, encoded_predictors = encode_qrf_predictor_pair( donor, recipient_predictor_frame ) + # The tenure rule on the donor's conditioning copies: a later target is + # fitted on values the recipients can hold. The main-residence mortgage is + # zero off the mortgaged tenure on the pinned tab already; the mortgages + # on other property are carried on every tenure. + for target, categories in UK_WAS_STRATIFIED_TARGETS.items(): + donor_encoded.loc[~np.isin(donor_category, categories), target] = 0.0 segment_seeds = was_wealth_segment_seeds(seed) segment_models = iter( RegimeGatedQRF(n_estimators=n_estimators, seed=segment_seed) @@ -491,22 +738,44 @@ def impute_was_wealth( raw = pd.DataFrame(index=recipient_encoded.index) fit_records: list[FitWeightRecord] = [] - def run_segment(base_predictors: Sequence[str], targets: Sequence[str]) -> None: + def run_segment( + base_predictors: Sequence[str], + targets: Sequence[str], + *, + stratum: Sequence[str] | None = None, + ) -> None: model = next(segment_models) + if stratum is None: + donor_rows, recipient_rows = donor_encoded, recipient_encoded + else: + donor_rows = donor_encoded.loc[np.isin(donor_category, stratum)] + recipient_rows = recipient_encoded.loc[np.isin(recipient_category, stratum)] + if donor_rows.empty: + raise ValueError( + f"WAS donor has no household in the tenure stratum " + f"{list(stratum)} that {list(targets)} is fitted on." + ) state = model.start_chain( - donor_encoded, + donor_rows, list(base_predictors), list(targets), weights="weight", ) - segment_raw = pd.DataFrame(index=recipient_encoded.index) - recipient_base = pd.concat( - [recipient_encoded.loc[:, list(base_predictors)]], - axis=1, - ) + segment_raw = pd.DataFrame(index=recipient_rows.index) + recipient_base = recipient_rows.loc[:, list(base_predictors)] for target in targets: + raw[target] = 0.0 + if recipient_rows.empty: + # No recipient holds the target: nothing is fitted, and the + # audit still records the weight kind the chain resolved. + fit_records.append( + FitWeightRecord( + f"{UK_WAS_WEALTH_FIT_NAME}:{target}", state.weight_kind + ) + ) + continue result = model.fit_draw_next( - donor_encoded, + donor_rows, recipient_base, segment_raw, state=state, @@ -518,18 +787,42 @@ def run_segment(base_predictors: Sequence[str], targets: Sequence[str]) -> None: ) ) segment_raw[target] = result.raw_draw - raw[target] = result.raw_draw + raw.loc[recipient_rows.index, target] = np.asarray( + result.raw_draw, dtype=float + ) state = result.state + def derive_total(total: str) -> None: + raw[total] = raw.loc[:, list(UK_WAS_DERIVED_TOTALS[total])].sum(axis=1) + base = encoded_predictors - run_segment(base, ("owned_land", "property_wealth")) + run_segment(base, ("owned_land",)) if segments == 1: return _partial_result(raw, fit_records, segment_seeds) + recipient_encoded["owned_land"] = raw["owned_land"] + run_segment( + (*base, "owned_land"), + ("main_residence_value",), + stratum=UK_WAS_STRATIFIED_TARGETS["main_residence_value"], + ) + if segments == 2: + return _partial_result(raw, fit_records, segment_seeds) + recipient_encoded["main_residence_value"] = raw["main_residence_value"] + run_segment( + (*base, "owned_land", "main_residence_value"), + ( + "other_residential_property_value", + "non_residential_property_value", + UK_WAS_OTHER_PROPERTY_COLUMN, + ), + ) + derive_total("property_wealth") + if segments == 3: + return _partial_result(raw, fit_records, segment_seeds) donor_encoded["corporate_wealth"] = donor_encoded["corporate_wealth"].astype(float) donor_encoded["private_pension_wealth"] = donor_encoded[ "private_pension_wealth" ].astype(float) - recipient_encoded["owned_land"] = raw["owned_land"] recipient_encoded["property_wealth"] = raw["property_wealth"] # Private pension wealth is drawn first in the position the old folded # corporate_wealth (84.7% pension by donor mass) occupied; the share-like @@ -542,10 +835,8 @@ def run_segment(base_predictors: Sequence[str], targets: Sequence[str]) -> None: "stocks_and_shares_isa", ), ) - raw["corporate_wealth"] = ( - raw["corporate_wealth_excl_isa"] + raw["stocks_and_shares_isa"] - ) - if segments == 2: + derive_total("corporate_wealth") + if segments == 4: return _partial_result(raw, fit_records, segment_seeds) recipient_encoded["private_pension_wealth"] = raw["private_pension_wealth"] recipient_encoded["corporate_wealth"] = raw["corporate_wealth"] @@ -560,43 +851,79 @@ def run_segment(base_predictors: Sequence[str], targets: Sequence[str]) -> None: "corporate_wealth", ), ( - "gross_financial_wealth", - "net_financial_wealth", - "main_residence_value", - "other_residential_property_value", - "non_residential_property_value", "savings", + "cash_isa", + UK_WAS_OTHER_FINANCIAL_COLUMN, "num_vehicles", "student_loan_balance", - "cash_isa", ), ) - if segments == 3: + derive_total("gross_financial_wealth") + if segments == 5: return _partial_result(raw, fit_records, segment_seeds) prior_outputs = tuple( column for column in UK_WAS_WEALTH_OUTPUT_COLUMNS - if column not in UK_WAS_DEBT_OUTPUT_COLUMNS + if column not in (*UK_WAS_DEBT_OUTPUT_COLUMNS, "net_financial_wealth") ) for output in prior_outputs: recipient_encoded[output] = raw[output] run_segment( - (*base, *UK_WAS_DEBT_SEGMENT_PREDICTORS, *prior_outputs), - UK_WAS_DEBT_OUTPUT_COLUMNS, + (*base, *prior_outputs), + ("main_residence_mortgage",), + stratum=UK_WAS_STRATIFIED_TARGETS["main_residence_mortgage"], ) + if segments == 6: + return _partial_result(raw, fit_records, segment_seeds) + recipient_encoded["main_residence_mortgage"] = raw["main_residence_mortgage"] + # The mortgages on other property are held on every tenure (an outright + # owner's or a renter's buy-to-let), so they are drawn without a stratum + # and conditioned on the property the chain already placed. + run_segment( + (*base, *prior_outputs, "main_residence_mortgage"), + (UK_WAS_OTHER_MORTGAGE_COLUMN,), + ) + derive_total("mortgage_debt") + if segments == 7: + return _partial_result(raw, fit_records, segment_seeds) + recipient_encoded["mortgage_debt"] = raw["mortgage_debt"] + run_segment((*base, *prior_outputs, "mortgage_debt"), ("consumer_debt",)) + raw["net_financial_wealth"] = derived_net_financial_wealth(raw) return UKWASWealthImputationResult( - draws=raw.loc[:, UK_WAS_WEALTH_OUTPUT_COLUMNS], + draws=raw.loc[:, (*UK_WAS_WEALTH_OUTPUT_COLUMNS, *UK_WAS_DRAWN_ONLY_COLUMNS)], fit_weight_records=tuple(fit_records), segment_seeds=segment_seeds, ) +#: Drawn components the stage does not emit, kept beside the outputs so the +#: totals can be re-derived after the donor-range cap. +UK_WAS_DRAWN_ONLY_COLUMNS = ( + "corporate_wealth_excl_isa", + "main_residence_mortgage", + *UK_WAS_INTERNAL_COMPONENT_COLUMNS, +) + + +def derived_net_financial_wealth(draws: pd.DataFrame) -> pd.Series: + """Gross financial wealth less the financial liabilities.""" + + liabilities = sum( + draws[column].clip(lower=0.0) for column in UK_WAS_NET_FINANCIAL_LIABILITIES + ) + return draws["gross_financial_wealth"] - liabilities + + def _partial_result( raw: pd.DataFrame, fit_records: list[FitWeightRecord], segment_seeds: tuple[int, ...], ) -> UKWASWealthImputationResult: - produced = [column for column in UK_WAS_WEALTH_OUTPUT_COLUMNS if column in raw] + produced = [ + column + for column in (*UK_WAS_WEALTH_OUTPUT_COLUMNS, *UK_WAS_DRAWN_ONLY_COLUMNS) + if column in raw + ] return UKWASWealthImputationResult( draws=raw.loc[:, produced], fit_weight_records=tuple(fit_records), @@ -604,6 +931,112 @@ def _partial_result( ) +#: Components a derived total gives up, in order, when the sum of donor-valued +#: draws leaves the donor's range for the total. The remainder goes first; the +#: main residence and the share-like total are never reduced (each is within +#: its own donor range, which the total's range contains). +_UK_WAS_TOTAL_CAP_ORDER: Mapping[str, tuple[str, ...]] = MappingProxyType( + { + "property_wealth": ( + UK_WAS_OTHER_PROPERTY_COLUMN, + "non_residential_property_value", + "other_residential_property_value", + "owned_land", + ), + "corporate_wealth": ("corporate_wealth_excl_isa", "stocks_and_shares_isa"), + "gross_financial_wealth": ( + UK_WAS_OTHER_FINANCIAL_COLUMN, + "cash_isa", + "savings", + ), + "mortgage_debt": (UK_WAS_OTHER_MORTGAGE_COLUMN, "main_residence_mortgage"), + } +) + + +def cap_derived_totals_to_donor_range( + draws: pd.DataFrame, donor: pd.DataFrame +) -> tuple[pd.DataFrame, dict[str, int]]: + """Keep every derived total inside the donor's range without breaking it. + + Each component is a donor value, but their sum can pass the donor's + largest total. The excess is taken out of the components in the declared + order and the total recomputed, so the identity survives where a clip of + the total alone would break it. Returns the adjusted draws and the rows + adjusted per total (zero on a build whose joint draws stay in range). + """ + + capped = draws.copy() + fired: dict[str, int] = {} + + def reduce(total: str, excess: pd.Series) -> None: + remaining = excess.clip(lower=0.0) + for component in _UK_WAS_TOTAL_CAP_ORDER[total]: + take = np.minimum(remaining, capped[component].clip(lower=0.0)) + capped[component] = capped[component] - take + remaining = remaining - take + recomputed = capped.loc[:, list(UK_WAS_DERIVED_TOTALS[total])].sum(axis=1) + # The reduced components sum back to the maximum up to float + # rounding; a sum a few ulp above it would be clipped as a real + # excess by the support clip, so it lands on the maximum exactly + # (inside the identity tolerance). + maximum = float(pd.to_numeric(donor[total], errors="coerce").max()) + capped[total] = np.where( + (recomputed > maximum) & (recomputed - maximum <= 1e-6), + maximum, + recomputed, + ) + + for total, components in UK_WAS_DERIVED_TOTALS.items(): + # Recomputed first: a reduced share-like total moves gross financial + # wealth with it. + capped[total] = capped.loc[:, list(components)].sum(axis=1) + maximum = float(pd.to_numeric(donor[total], errors="coerce").max()) + excess = capped[total] - maximum + fired[f"{total}_capped_rows"] = int((excess > 0).sum()) + if fired[f"{total}_capped_rows"]: + reduce(total, excess) + net = pd.to_numeric(donor["net_financial_wealth"], errors="coerce") + capped["net_financial_wealth"] = derived_net_financial_wealth(capped) + above = capped["net_financial_wealth"] - float(net.max()) + fired["net_financial_wealth_capped_high_rows"] = int((above > 0).sum()) + if fired["net_financial_wealth_capped_high_rows"]: + reduce("gross_financial_wealth", above) + capped["net_financial_wealth"] = derived_net_financial_wealth(capped) + below = float(net.min()) - capped["net_financial_wealth"] + fired["net_financial_wealth_capped_low_rows"] = int((below > 0).sum()) + if fired["net_financial_wealth_capped_low_rows"]: + remaining = below.clip(lower=0.0) + for liability in UK_WAS_NET_FINANCIAL_LIABILITIES: + take = np.minimum(remaining, capped[liability].clip(lower=0.0)) + capped[liability] = capped[liability] - take + remaining = remaining - take + capped["net_financial_wealth"] = derived_net_financial_wealth(capped) + return capped, fired + + +def wealth_identity_violations(table: pd.DataFrame) -> dict[str, int]: + """Rows on which a total departs from its components, by identity.""" + + tolerance = UK_WAS_IDENTITY_TOLERANCE_GBP + violations: dict[str, int] = {} + for total, components in UK_WAS_DERIVED_TOTALS.items(): + available = [column for column in components if column in table.columns] + if len(available) == len(components): + gap = table[total] - table.loc[:, available].sum(axis=1) + violations[f"{total}_violation_rows"] = int((gap.abs() > tolerance).sum()) + else: + # The emitted frame carries no remainder column: the total must + # still cover the components it does carry. + gap = table[total] - table.loc[:, available].sum(axis=1) + violations[f"{total}_below_components_rows"] = int((gap < -tolerance).sum()) + gap = table["net_financial_wealth"] - derived_net_financial_wealth(table) + violations["net_financial_wealth_violation_rows"] = int( + (gap.abs() > tolerance).sum() + ) + return violations + + def encode_qrf_predictor_pair( donor: pd.DataFrame, recipient: pd.DataFrame, @@ -614,9 +1047,9 @@ def encode_qrf_predictor_pair( Mirrors the SPI stage's paired dummy encoding and the incumbent's dummy-encoded region. Donor rows with an unmapped region code (the - incumbent's absent GOR code 3) become all-zero dummy rows. The - predictor list defaults to the WAS wealth set; the E6 has-fuel bridge - passes its own. + incumbent's absent GOR code 3) become all-zero dummy rows. The tenure + category is a label for the strata, not a number, and is left out of the + encoded tables; its one-hot flags enter as 0/1 floats. """ numeric_predictors = tuple( @@ -633,10 +1066,10 @@ def encode_qrf_predictor_pair( dummies = dummies.reindex(sorted(dummies.columns), axis=1) def _encode(table: pd.DataFrame, block: pd.DataFrame) -> pd.DataFrame: - encoded = table.drop(columns=["region"]).copy() - if "is_renting" in encoded.columns: - encoded["is_renting"] = encoded["is_renting"].astype(bool).astype(float) - for column in UK_WAS_DEBT_SEGMENT_PREDICTORS: + encoded = table.drop( + columns=["region", UK_WAS_TENURE_CATEGORY_COLUMN], errors="ignore" + ).copy() + for column in ("is_renting", *UK_WAS_TENURE_PREDICTORS): if column in encoded.columns: encoded[column] = encoded[column].astype(bool).astype(float) for column in encoded.columns: @@ -658,6 +1091,146 @@ def _encode(table: pd.DataFrame, block: pd.DataFrame) -> pd.DataFrame: ) +def _weighted_share(mask: np.ndarray, weights: np.ndarray) -> float: + total = float(weights.sum()) + return float(weights[mask].sum() / total) if total > 0.0 else 0.0 + + +#: Housing columns whose share of positive values the receipt reports by +#: tenure on both sides. +UK_WAS_TENURE_RECEIPT_COLUMNS = ( + "main_residence_value", + "main_residence_mortgage", + UK_WAS_OTHER_MORTGAGE_COLUMN, + "mortgage_debt", + "property_wealth", + "other_residential_property_value", +) + + +def tenure_coherence_receipt( + *, + donor: pd.DataFrame, + recipient_category: pd.Series, + recipient_draws: pd.DataFrame, + recipient_weights: Sequence[float], +) -> dict[str, object]: + """Tenure coherence of the housing columns, donor against recipient. + + The first two counts are the structural zeros the stage-health gate + requires: no main-residence mortgage off a mortgaged tenure and no + main-residence value off an owner tenure. The rest compares the recipient + frame with the donor at each side's own weights, and records the mortgage + mass held off the mortgaged tenure on each side (the mortgages on other + property, which the chain carries on every tenure since the review of + #1089; on the donor, TotMortR8 and OthMortR8_sum are read for that record + only). + """ + + donor_category = donor[UK_WAS_TENURE_CATEGORY_COLUMN].astype(str).to_numpy() + donor_weights = pd.to_numeric(donor["weight"], errors="coerce").to_numpy(float) + category = np.asarray(recipient_category.astype(str)) + weights = np.asarray(recipient_weights, dtype=float) + owner = np.isin(category, UK_OWNER_TENURE_CATEGORIES) + mortgaged = category == UK_TENURE_OWNED_WITH_MORTGAGE + donor_owner = np.isin(donor_category, UK_OWNER_TENURE_CATEGORIES) + donor_mortgaged = donor_category == UK_TENURE_OWNED_WITH_MORTGAGE + + def values(table: pd.DataFrame, column: str) -> np.ndarray: + return pd.to_numeric(table[column], errors="coerce").fillna(0.0).to_numpy(float) + + main = values(recipient_draws, "main_residence_value") + debt = values(recipient_draws, "main_residence_mortgage") + total_debt = values(recipient_draws, "mortgage_debt") + donor_main = values(donor, "main_residence_value") + donor_debt = values(donor, "main_residence_mortgage") + donor_total_debt = values(donor, "mortgage_debt") + donor_debt_mass = float((donor_total_debt * donor_weights).sum()) + donor_off_tenure = float((donor_total_debt * donor_weights)[~donor_mortgaged].sum()) + off_tenure = float((total_debt * weights)[~mortgaged].sum()) + positive_share: dict[str, dict[str, dict[str, float]]] = {} + for column in UK_WAS_TENURE_RECEIPT_COLUMNS: + recipient_positive = values(recipient_draws, column) > 0.0 + donor_positive = values(donor, column) > 0.0 + positive_share[column] = { + tenure: { + "donor": _weighted_share( + donor_positive[donor_category == tenure], + donor_weights[donor_category == tenure], + ), + "recipient": _weighted_share( + recipient_positive[category == tenure], + weights[category == tenure], + ), + } + for tenure in UK_TENURE_CATEGORIES + } + return { + "main_residence_mortgage_off_mortgaged_tenure_rows": int( + ((debt > 0.0) & ~mortgaged).sum() + ), + # Carried, not a defect: the mortgages on other property that + # households off the mortgaged tenure hold. + "mortgage_debt_rows_off_mortgaged_tenure": int( + ((total_debt > 0.0) & ~mortgaged).sum() + ), + "mortgage_debt_mass_off_mortgaged_tenure": off_tenure, + "main_residence_value_off_owner_tenure_rows": int( + ((main > 0.0) & ~owner).sum() + ), + "owner_rows": int(owner.sum()), + "owner_rows_without_main_residence_value": int(((main <= 0.0) & owner).sum()), + "owner_share_without_main_residence_value": _weighted_share( + main[owner] <= 0.0, weights[owner] + ), + "donor_owner_share_without_main_residence_value": _weighted_share( + donor_main[donor_owner] <= 0.0, donor_weights[donor_owner] + ), + "mortgaged_share_without_main_residence_mortgage": _weighted_share( + debt[mortgaged] <= 0.0, weights[mortgaged] + ), + "donor_mortgaged_share_without_main_residence_mortgage": _weighted_share( + donor_debt[donor_mortgaged] <= 0.0, donor_weights[donor_mortgaged] + ), + # The main-residence mortgage against the main-residence value, read + # before the regional property uprating, which moves the value and + # leaves the debt alone. + "mortgaged_share_debt_above_main_residence_value": _weighted_share( + debt[mortgaged] > main[mortgaged], weights[mortgaged] + ), + "donor_mortgaged_share_debt_above_main_residence_value": _weighted_share( + donor_debt[donor_mortgaged] > donor_main[donor_mortgaged], + donor_weights[donor_mortgaged], + ), + "donor_mortgage_debt_mass": donor_debt_mass, + "donor_mortgage_debt_mass_off_mortgaged_tenure": donor_off_tenure, + "donor_mortgage_debt_share_off_mortgaged_tenure": ( + donor_off_tenure / donor_debt_mass if donor_debt_mass > 0.0 else 0.0 + ), + "donor_other_property_mortgage_mass_off_mortgaged_tenure": float( + (values(donor, "other_property_mortgage").clip(min=0.0) * donor_weights)[ + ~donor_mortgaged + ].sum() + ), + "donor_main_residence_mortgage_mass_off_mortgaged_tenure": float( + (values(donor, "main_residence_mortgage").clip(min=0.0) * donor_weights)[ + ~donor_mortgaged + ].sum() + ), + "donor_unclassified_tenure_rows": int( + donor["tenure_code_unclassified"].astype(bool).sum() + ), + "donor_rows_by_tenure": { + tenure: int((donor_category == tenure).sum()) + for tenure in UK_TENURE_CATEGORIES + }, + "recipient_rows_by_tenure": { + tenure: int((category == tenure).sum()) for tenure in UK_TENURE_CATEGORIES + }, + "positive_share_by_tenure": positive_share, + } + + def support_clip_to_donor( draws: pd.DataFrame, donor: pd.DataFrame ) -> UKSupportClipResult: @@ -751,17 +1324,40 @@ def _donor_artifact(stage: SourceStageSpec) -> Mapping[str, Any]: ) -def _assert_debt_segment_predictors(stage: SourceStageSpec) -> None: - """The chain op must declare exactly the debt segment's extra predictors.""" +def _assert_chain_declaration(stage: SourceStageSpec) -> None: + """The chain op must declare the strata and the derived totals the run uses.""" operation = next( op for op in stage.operations if op.kind == "fit_weighted_qrf_chain" ) - declared = tuple(operation.parameters.get("debt_segment_predictors", ())) - if declared != UK_WAS_DEBT_SEGMENT_PREDICTORS: + declared_strata = { + str(target): tuple(categories) + for target, categories in dict( + operation.parameters.get("stratified_targets", {}) + ).items() + } + if declared_strata != dict(UK_WAS_STRATIFIED_TARGETS): + raise ValueError( + "was_wealth stratified_targets drifted: manifest declares " + f"{declared_strata!r}, runtime uses {dict(UK_WAS_STRATIFIED_TARGETS)!r}." + ) + declared_totals = { + str(total): tuple(components) + for total, components in dict( + operation.parameters.get("derived_totals", {}) + ).items() + } + runtime_totals = { + **dict(UK_WAS_DERIVED_TOTALS), + "net_financial_wealth": ( + "gross_financial_wealth", + *(f"-{column}" for column in UK_WAS_NET_FINANCIAL_LIABILITIES), + ), + } + if declared_totals != runtime_totals: raise ValueError( - "was_wealth debt_segment_predictors drifted: manifest declares " - f"{declared!r}, runtime uses {UK_WAS_DEBT_SEGMENT_PREDICTORS!r}." + "was_wealth derived_totals drifted: manifest declares " + f"{declared_totals!r}, runtime uses {runtime_totals!r}." ) diff --git a/packages/microcosm-build/tests/engine/uk/test_uk_release_input_coverage.py b/packages/microcosm-build/tests/engine/uk/test_uk_release_input_coverage.py index c47e22f8d..67fe34579 100644 --- a/packages/microcosm-build/tests/engine/uk/test_uk_release_input_coverage.py +++ b/packages/microcosm-build/tests/engine/uk/test_uk_release_input_coverage.py @@ -451,8 +451,9 @@ def test_shipped_manifest_is_current(self) -> None: "cgt_support_split", "cgt_incidence_clone", "hmrc_cgt_gains_spine", - "hmrc_cgt_asset_type_spine", "cgt_incidence_anchor", + "cgt_residential_split", + "hmrc_cgt_asset_type_spine", "salary_sacrifice", "student_loans", "was_wealth", diff --git a/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py b/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py index 9b99b5882..c1b42f6b4 100644 --- a/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py +++ b/packages/microcosm-build/tests/engine_free/shared/test_country_spec.py @@ -945,6 +945,7 @@ def test_spi_spine_adds_no_country_package_resources(self) -> None: "ons_age_tail_band_populations.json", "ashe_employer_pension_contribution_rates.json", "ons_single_year_age_populations.json", + "hmrc_child_benefit_statistics.json", "lcfs_consumption_support_bounds.json", "etb_vat_support_bounds.json", "etb_services_support_bounds.json", @@ -1009,10 +1010,12 @@ def test_uk_source_manifest_contains_only_canonical_spine_stages(self) -> None: # #685, frs_relationships #791, hmrc_cgt_asset_type_spine #725, # cgt_incidence_anchor #970, nts_bus_travel #930, then # spi_income_band_donors (PolicyEngine/chronicle#280 lane), the SPI - # housing shell (#1012), was_lisa (#1003) and pension_credit_take_up - # (microcosm#1069) as the newest); the frs_hmrc_retained_leaves / - # hmrc_spi_income pair is retired (#901). - assert len(spec.sources.stages) == 36 + # housing shell (#1012), was_lisa (#1003), pension_credit_take_up + # (microcosm#1069) and child_benefit_take_up (microcosm#1063) as the + # newest); the frs_hmrc_retained_leaves / hmrc_spi_income pair is + # retired (#901). + # microcosm#1063 also adds cgt_residential_split after the anchor. + assert len(spec.sources.stages) == 38 assert not {"frs_hmrc_retained_leaves", "hmrc_spi_income"}.intersection( stage.stage for stage in spec.sources.stages ) @@ -1067,6 +1070,7 @@ def test_uk_package_loads(self) -> None: "ons_age_tail_band_populations.json", "ashe_employer_pension_contribution_rates.json", "ons_single_year_age_populations.json", + "hmrc_child_benefit_statistics.json", "lcfs_consumption_support_bounds.json", "etb_vat_support_bounds.json", "etb_services_support_bounds.json", @@ -1380,6 +1384,7 @@ def test_declares_the_full_june_battery(self, manifest) -> None: "uk_ledger_compile_parity_local_incumbent_2025", "uk_target_surface_local_default_2025", "uk_stage_was_wealth_support", + "uk_stage_was_wealth_coherence", "uk_stage_was_lisa_support", "uk_stage_nts_bus_travel_support", "uk_stage_nts_bus_travel_facts", @@ -1394,10 +1399,12 @@ def test_declares_the_full_june_battery(self, manifest) -> None: "uk_stage_spi_support_channel_mass", "uk_stage_hmrc_spi_income_spine_identity", "uk_stage_pension_credit_take_up", + "uk_stage_child_benefit_take_up", "uk_stage_cgt_incidence_clone_mass", "uk_stage_cgt_support_split_mass", "uk_stage_spi_income_band_donors_support", "uk_stage_hmrc_cgt_gains_spine_summary", + "uk_stage_cgt_residential_split_mass", "uk_stage_hmrc_cgt_asset_type_spine_summary", "uk_stage_cgt_incidence_anchor_composition", "uk_stage_salary_sacrifice_realization", @@ -1483,6 +1490,7 @@ def test_strict_absent_evidence_entries_are_declared(self, manifest) -> None: flagged = [g.id for g in manifest.gates if g.evidence_absent_blocks] assert flagged == [ "uk_stage_was_wealth_support", + "uk_stage_was_wealth_coherence", "uk_stage_was_lisa_support", "uk_stage_nts_bus_travel_support", "uk_stage_nts_bus_travel_facts", @@ -1497,10 +1505,12 @@ def test_strict_absent_evidence_entries_are_declared(self, manifest) -> None: "uk_stage_spi_support_channel_mass", "uk_stage_hmrc_spi_income_spine_identity", "uk_stage_pension_credit_take_up", + "uk_stage_child_benefit_take_up", "uk_stage_cgt_incidence_clone_mass", "uk_stage_cgt_support_split_mass", "uk_stage_spi_income_band_donors_support", "uk_stage_hmrc_cgt_gains_spine_summary", + "uk_stage_cgt_residential_split_mass", "uk_stage_hmrc_cgt_asset_type_spine_summary", "uk_stage_cgt_incidence_anchor_composition", "uk_stage_salary_sacrifice_realization", diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_atomic_identity_graph.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_atomic_identity_graph.py index b7937072b..d9851d9ae 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_atomic_identity_graph.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_atomic_identity_graph.py @@ -87,6 +87,9 @@ def lineage_frame(rows=LINEAGE): "household_is_capital_gains_clone": np.asarray([r[4] for r in rows]), "household_is_cgt_support_copy": np.asarray([r[5] != 0 for r in rows]), "cgt_support_copy_index": np.asarray([r[5] for r in rows], dtype=np.int64), + # microcosm#1063: a residential arm index (0 on every row here). + "household_is_cgt_residential_clone": np.zeros(len(rows), dtype=bool), + "cgt_residential_clone_index": np.zeros(len(rows), dtype=np.int64), } ) person = pd.DataFrame( @@ -324,5 +327,7 @@ def test_identity_inputs_are_the_declared_lineage_columns(): "household_is_cgt_support_copy", "cgt_support_copy_index", "household_is_capital_gains_clone", + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", CLONE, ) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_battery_bindings.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_battery_bindings.py index e2f4d5a2b..f4daf66f6 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_battery_bindings.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_battery_bindings.py @@ -59,7 +59,9 @@ #: target-fit register from the PolicyEngine/chronicle#280 lane; earliest expiry #: 2026-10-21, the same entries) so the suite never drifts across an approval #: or expiry boundary. Move it forward when a register gains a later approval. -CLOCK = date(2026, 9, 23) +# The committed registers are evaluated as of this date; the microcosm#1063 +# c9 input-mass and QRF-tail entries take force on 2026-10-02. +CLOCK = date(2026, 10, 4) VALIDATE_REFERENCE = ( "microcosm.build.uk_runtime.weighted_integrity." @@ -483,6 +485,133 @@ def test_nonnegative_binding_passes_clean_scheduled_columns(self) -> None: assert result.passed is True + def _salary_sacrifice_frames(self, *, carrier): + """The release frame without the pre-conversion pay carrier, the spine with it. + + The carrier leaves at the release boundary + (UK_RELEASE_EXPORT_DROPPED_COLUMNS), so a certifier checking the + salary_sacrifice stage's declared non-negative outputs on the release + candidate finds it only on the spine frame (microcosm#1063). + """ + + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_OUTPUT, + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) + + person, benunit, household = _tables() + person[SALSAC_OUTPUT] = [0.0, 10.0, 0.0, 5.0] + person["employee_pension_contributions"] = [1.0, 0.0, 2.0, 0.0] + release = uk_national_frame( + person=person.copy(), + benunit=benunit.copy(), + household=household.copy(), + time_period="2023", + ) + spine_person = person.copy() + spine_person[SALSAC_PRE_CONVERSION_PAY_COLUMN] = carrier + spine = uk_national_frame( + person=spine_person, + benunit=benunit.copy(), + household=household.copy(), + time_period="2023", + ) + return release, spine + + def test_nonnegative_binding_reads_export_dropped_columns_from_the_spine( + self, + ) -> None: + binding = UK_GATE_REGISTRY["nonnegative_columns"] + release, spine = self._salary_sacrifice_frames(carrier=[0.0, 30.0, 0.0, 20.0]) + # Without the spine frame the dropped carrier is a missing column. + missing = binding.evaluate( + EvidenceContext( + frame=release, artifacts={"build_stage_names": ("salary_sacrifice",)} + ), + {}, + ) + assert missing.passed is False + assert "salary_sacrifice_pre_conversion_pay" in missing.failures[0] + # With it the carrier is checked where it lives. + checked = binding.evaluate( + EvidenceContext( + frame=release, + artifacts={ + "build_stage_names": ("salary_sacrifice",), + "spine_frame": spine, + }, + ), + {}, + ) + assert checked.passed is True + # The spine's values are really checked, not just their presence. + _, negative_spine = self._salary_sacrifice_frames( + carrier=[0.0, 30.0, -1.0, 20.0] + ) + failing = binding.evaluate( + EvidenceContext( + frame=release, + artifacts={ + "build_stage_names": ("salary_sacrifice",), + "spine_frame": negative_spine, + }, + ), + {}, + ) + assert failing.passed is False + assert ( + "salary_sacrifice_pre_conversion_pay: 1 finite value(s) below zero" + in (failing.failures[0]) + ) + + def test_only_export_dropped_columns_are_read_from_the_spine(self) -> None: + from microcosm.build.uk_runtime.battery_bindings import ( + _export_dropped_columns_from_spine, + ) + from microcosm.build.uk_runtime.frs_disability import ( + UK_INTERNAL_DISABILITY_REPORTED_COLUMNS, + ) + + release, spine = self._salary_sacrifice_frames(carrier=[0.0, 30.0, 0.0, 20.0]) + spine_person = spine.table("person").copy() + spine_person["sic_industry_division"] = [1.0, 2.0, 3.0, 4.0] + spine_person[UK_INTERNAL_DISABILITY_REPORTED_COLUMNS[0]] = 0.0 + spine = uk_national_frame( + person=spine_person, + benunit=spine.table("benunit"), + household=spine.table("household"), + time_period="2023", + household_weights=spine.weights_for("household").values, + ) + found = _export_dropped_columns_from_spine( + EvidenceContext(frame=release, artifacts={"spine_frame": spine}), + [ + "salary_sacrifice_pre_conversion_pay", + UK_INTERNAL_DISABILITY_REPORTED_COLUMNS[0], + "sic_industry_division", + "never_declared", + ], + ) + # A column the export keeps is never answered from the spine, even + # when the spine carries it; one absent from the spine is not found. + assert set(found) == { + "salary_sacrifice_pre_conversion_pay", + UK_INTERNAL_DISABILITY_REPORTED_COLUMNS[0], + } + assert found["salary_sacrifice_pre_conversion_pay"].tolist() == [ + 0.0, + 30.0, + 0.0, + 20.0, + ] + assert ( + _export_dropped_columns_from_spine( + EvidenceContext(frame=release, artifacts={}), + ["salary_sacrifice_pre_conversion_pay"], + ) + == {} + ) + def test_nonnegative_binding_does_not_demand_unscheduled_stages(self) -> None: # This isolated stage declares no nonnegative outputs. Outputs of # unscheduled employment and income stages must not be demanded. @@ -1499,6 +1628,8 @@ def test_weight_gate_bindings_fold_support_families_before_evaluating() -> None: household["household_support_clone_index"] = np.zeros(4, dtype=np.int64) household["household_is_cgt_support_copy"] = [False, True, True, False] household["cgt_support_copy_index"] = np.asarray([0, 1, 2, 0], dtype=np.int64) + household["household_is_cgt_residential_clone"] = False + household["cgt_residential_clone_index"] = np.zeros(4, dtype=np.int64) frame = uk_national_frame( person=person, benunit=benunit, household=household, time_period="2023" ) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_asset_type.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_asset_type.py index 758646461..4e1709609 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_asset_type.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_asset_type.py @@ -21,6 +21,7 @@ CGT_BADR_ELIGIBLE_TYPES, CGT_BADR_GAINS_COLUMN, CGT_BUSINESS_STOCK_SIGNAL, + CGT_CLAIMANT_STATUSES, CGT_RESIDENTIAL_GAINS_COLUMN, CGT_RESIDENTIAL_STOCK_SIGNAL, CGT_STOCK_HOUSEHOLD_COLUMNS, @@ -39,6 +40,7 @@ UKCGTBADRParameters, assign_uk_cgt_asset_types, cgt_asset_type_operation_parameters, + claimant_status_share_targets, fit_type_weights, load_hmrc_cgt_asset_type_facts, solve_residential_logistic, @@ -48,6 +50,9 @@ UK_CGT_TAXABLE_INCOME_PROXY_COMPONENTS, UKCGTPolicyParameters, ) +from microcosm.build.uk_runtime.cgt_residential_split import ( + split_cgt_residential_households, +) from microcosm.build.uk_runtime.content_identity import uk_frame_content_identity from microcosm.build.uk_runtime.ledger_fact_vendoring import load_vendored_resource from microcosm.build.uk_runtime.national_frame import uk_national_frame @@ -197,6 +202,22 @@ def _frame(gains, *, weights=None, time_period: str = "2024", stocks=None) -> Fr ) +def _split(frame: Frame, facts: HMRCCGTAssetTypeFacts) -> Frame: + """The frame the asset-type stage sees: the residential split's arms.""" + + return split_cgt_residential_households( + frame, facts=facts, parameters=PARAMETERS + ).frame + + +def _assign(frame: Frame, facts, parameters, badr_parameters, **kwargs): + """Run the residential split, then the asset-type assignment.""" + + return assign_uk_cgt_asset_types( + _split(frame, facts), facts, parameters, badr_parameters, **kwargs + ) + + def _synthetic_gains(rows: int = 20_000, seed: int = 1) -> np.ndarray: rng = np.random.default_rng(seed) gains = np.where(rng.random(rows) < 0.8, np.exp(rng.normal(10.5, 1.8, rows)), 0.0) @@ -480,10 +501,21 @@ def _run(self, gains=None, weights=None, **kwargs): weights = np.full(gains.size, 60.0) if weights is None else weights facts = kwargs.pop("facts", None) or _synthetic_facts(gains, weights) frame = _frame(gains, weights=weights, stocks=kwargs.pop("stocks", None)) - result, summary = assign_uk_cgt_asset_types( + result, summary = _assign( frame, facts, PARAMETERS, kwargs.pop("badr", BADR_PARAMETERS), **kwargs ) - return gains, result.table("person"), summary.evidence() + person = result.table("person") + # The split adds the residential arms, so the gains are read back + # from the classified frame to stay aligned with its rows; the + # household stock signals ride along for the stock-conditioning cases. + household = result.table("household") + person = person.merge( + household[["household_id", *CGT_STOCK_HOUSEHOLD_COLUMNS]], + left_on="person_household_id", + right_on="household_id", + how="left", + ) + return person["capital_gains"].to_numpy(dtype=float), person, summary.evidence() def test_every_band_meets_its_targets_within_the_walk_bound(self) -> None: gains, person, evidence = self._run() @@ -628,10 +660,10 @@ def test_holders_are_drawn_more_often_while_every_total_holds(self) -> None: _, _, evidence = TestBADRClaims()._run(gains, stocks=stocks) residential = evidence["residential"] - assert residential["expected_count"] == pytest.approx( + assert residential["achieved_count"] == pytest.approx( residential["count_target_individuals_basis"], rel=1e-6 ) - assert residential["expected_gains"] == pytest.approx( + assert residential["achieved_gains"] == pytest.approx( residential["gains_target_individuals_basis"], rel=1e-6 ) assert residential["stock_share_liable"] == pytest.approx(0.5, abs=0.02) @@ -652,8 +684,9 @@ def test_holders_are_drawn_more_often_while_every_total_holds(self) -> None: for name in CGT_TYPE_STOCK_SIGNALS: shares = asset_type["stock_share_by_type"][name] assert shares["typed"] > shares["non_residential_liable"] + 0.1 - # The receipts restate the declared settings. - assert residential["stock_log_odds"] == CGT_STOCK_LOG_ODDS + # The receipts restate the declared settings; the residential log + # odds now live in the split stage's logistic receipt. + assert "stock_log_odds" not in residential assert residential["stock_signal"] == CGT_RESIDENTIAL_STOCK_SIGNAL assert badr["stock_log_odds"] == CGT_STOCK_LOG_ODDS assert badr["stock_signal"] == CGT_BUSINESS_STOCK_SIGNAL @@ -663,7 +696,7 @@ def test_the_open_top_band_leans_without_a_slope(self) -> None: gains = _synthetic_gains() stocks = self._stocks(gains.size) - _, person, evidence = TestBADRClaims()._run(gains, stocks=stocks) + gains, person, evidence = TestBADRClaims()._run(gains, stocks=stocks) top = evidence["badr"]["bands"][-1] assert top["qualifying_amount"] == "lifetime_limit" @@ -672,7 +705,9 @@ def test_the_open_top_band_leans_without_a_slope(self) -> None: assert top["expected_count"] == pytest.approx(top["count_target"], rel=1e-9) claimant = person[CGT_BADR_GAINS_COLUMN].to_numpy() > 0 at_limit = claimant & (gains >= BADR_PARAMETERS.lifetime_limit) - business = stocks["corporate_wealth"] > 0 + # The split's arms carry the stock columns, so read them back from + # the classified frame rather than the pre-split draw. + business = person["corporate_wealth"].to_numpy(dtype=float) > 0 assert business[at_limit].mean() > 0.65 def test_refuses_a_frame_without_a_stock_column(self) -> None: @@ -687,7 +722,7 @@ def test_refuses_a_frame_without_a_stock_column(self) -> None: ) with pytest.raises(ValueError, match=r"lacks \['corporate_wealth'\]"): - assign_uk_cgt_asset_types( + _assign( stripped, _synthetic_facts(gains, np.full(gains.size, 60.0)), PARAMETERS, @@ -701,7 +736,7 @@ def test_refuses_a_stock_signal_that_is_not_finite(self) -> None: frame = _frame(gains, stocks={"gross_financial_wealth": wealth}) with pytest.raises(ValueError, match="gross_financial_wealth must be finite"): - assign_uk_cgt_asset_types( + _assign( frame, _synthetic_facts(gains, np.full(gains.size, 60.0)), PARAMETERS, @@ -709,17 +744,87 @@ def test_refuses_a_stock_signal_that_is_not_finite(self) -> None: ) +class TestClaimantStatusTargets: + def test_the_two_compositions_restore_the_published_one(self) -> None: + targets = load_hmrc_cgt_asset_type_facts().non_residential_gains_shares() + share = 0.34 + + split = claimant_status_share_targets(targets, share) + + assert tuple(split) == CGT_CLAIMANT_STATUSES + eligible = sum(targets[name] for name in CGT_BADR_ELIGIBLE_TYPES) + for name in CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES: + claimants = split["claimants"][name] + if name in CGT_BADR_ELIGIBLE_TYPES: + assert claimants == pytest.approx(targets[name] / eligible) + else: + assert claimants == 0.0 + assert share * claimants + (1 - share) * split["non_claimants"][ + name + ] == pytest.approx(targets[name]) + assert sum(split["claimants"].values()) == pytest.approx(1.0) + assert sum(split["non_claimants"].values()) == pytest.approx(1.0) + + def test_without_claimants_everyone_takes_the_published_composition(self) -> None: + targets = load_hmrc_cgt_asset_type_facts().non_residential_gains_shares() + + split = claimant_status_share_targets(targets, 0.0) + + assert split["non_claimants"] == pytest.approx(dict(targets)) + + def test_refuses_claimants_holding_more_than_the_eligible_types_share( + self, + ) -> None: + targets = load_hmrc_cgt_asset_type_facts().non_residential_gains_shares() + + with pytest.raises(ValueError, match="negative share"): + claimant_status_share_targets(targets, 0.8) + with pytest.raises(ValueError, match=r"\[0, 1\)"): + claimant_status_share_targets(targets, 1.0) + + +class TestTypeDiffusion: + def test_realised_gains_track_the_fit_whatever_the_heaviest_rows_draw( + self, + ) -> None: + """An independent draw lets a few heavy rows decide the composition; + the walk places the largest gains first and every later row repays the + balance, so the realised shares stay within a few of the largest stakes + of the fitted ones.""" + + rng = np.random.default_rng(11) + rows = 400 + gains = np.exp(rng.normal(12.0, 2.5, rows)) + weights = rng.uniform(5.0, 60.0, rows) + raw = rng.uniform(0.05, 1.0, (rows, 5)) + raw[::9, [0, 2]] = 0.0 + probabilities = raw / raw.sum(axis=1, keepdims=True) + stakes = weights * gains + order = np.lexsort((np.arange(rows), -gains)) + + choice, balances = cgt_asset_type._diffuse_types(probabilities, stakes, order) + + # Nobody takes a type they may not hold. + assert (probabilities[np.arange(rows), choice] > 0.0).all() + expected = (stakes[:, None] * probabilities).sum(axis=0) + realised = np.asarray([stakes[choice == index].sum() for index in range(5)]) + np.testing.assert_allclose(balances, expected - realised, rtol=1e-9, atol=1.0) + assert np.abs(balances).max() <= 3.2 * stakes.max() + again, _ = cgt_asset_type._diffuse_types(probabilities, stakes, order) + assert (again == choice).all() + + class TestAssignment: def test_flags_and_types_every_liable_gainer_and_nobody_else(self) -> None: facts = load_hmrc_cgt_asset_type_facts() - gains = _synthetic_gains() - frame = _frame(gains) + split = _split(_frame(_synthetic_gains()), facts) result, summary = assign_uk_cgt_asset_types( - frame, facts, PARAMETERS, BADR_PARAMETERS + split, facts, PARAMETERS, BADR_PARAMETERS ) person = result.table("person") + gains = person["capital_gains"].to_numpy(dtype=float) types = person[CGT_ASSET_TYPE_COLUMN].to_numpy() residential = person[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy() liable = gains > PARAMETERS.annual_exempt_amount @@ -729,48 +834,79 @@ def test_flags_and_types_every_liable_gainer_and_nobody_else(self) -> None: assert not np.isin( types[liable], [CGT_ASSET_TYPE_NONE, CGT_ASSET_TYPE_SUB_AEA] ).any() + # A residential arm (the split's residential gain) is typed residential + # and nobody else is. flagged = types == CGT_ASSET_TYPE_RESIDENTIAL + assert (flagged == (residential > 0.0)).all() assert (residential[flagged] == gains[flagged]).all() - assert (residential[~flagged] == 0.0).all() # Untouched columns and weights, one conservation receipt. - assert (person["capital_gains"].to_numpy() == gains).all() + assert ( + residential + == split.table("person")[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy() + ).all() assert result.mass_log[-1].reason == UK_CGT_ASSET_TYPE_MASS_CONSERVATION_REASON assert result.mass_log[-1].old_total == result.mass_log[-1].new_total evidence = summary.evidence() assert evidence["stage"] == UK_CGT_ASSET_TYPE_STAGE_NAME residential_receipt = evidence["residential"] - assert residential_receipt["expected_count"] == pytest.approx( + # The arms carry Table 8a at design weights: the identities the split + # left, restated here by gain band. + assert residential_receipt["source_stage"] == "cgt_residential_split" + assert residential_receipt["achieved_count"] == pytest.approx( facts.residential_taxpayers_individuals_basis, rel=1e-6 ) - assert residential_receipt["expected_gains"] == pytest.approx( + assert residential_receipt["achieved_gains"] == pytest.approx( facts.residential_gains_individuals_basis, rel=1e-6 ) - # Systematic sampling lands within one person of the expected count. - assert ( - abs( - residential_receipt["achieved_count"] - - residential_receipt["expected_count"] - ) - <= 60.0 + assert residential_receipt["count_relative_error"] < 1e-6 + assert residential_receipt["gains_relative_error"] < 1e-6 + bands = residential_receipt["bands"] + assert len(bands) == 10 + assert sum(band["achieved_count"] for band in bands) == pytest.approx( + residential_receipt["achieved_count"] ) - assert residential_receipt["achieved_gains"] == pytest.approx( - residential_receipt["expected_gains"], rel=0.05 + assert sum(band["achieved_gains"] for band in bands) == pytest.approx( + residential_receipt["achieved_gains"] ) - assert residential_receipt["logistic_slope"] < 0 - shares = evidence["asset_type"]["achieved_gains_share"] + assert sum(band["achieved_rows"] for band in bands) == int(flagged.sum()) + assert "count_bound" not in residential_receipt + asset_types = evidence["asset_type"] + shares = asset_types["achieved_gains_share"] assert sum(shares.values()) == pytest.approx(1.0) - assert shares["unlisted_shares"] > 0.4 + # The realisation tracks the fit: the composition lands on Table 7 + # overall and on each claimant status's own targets, to within a + # couple of the largest stakes. + targets = facts.non_residential_gains_shares() + for name in CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES: + assert shares[name] == pytest.approx(targets[name], abs=0.02) + by_status = asset_types["composition_by_claimant_status"] + assert tuple(by_status) == CGT_CLAIMANT_STATUSES + for status in CGT_CLAIMANT_STATUSES: + report = by_status[status] + assert report["rows"] > 0 + tolerance = 3.2 * report["largest_stake"] / report["gains_mass"] + for name in CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES: + assert report["expected_gains_share"][name] == pytest.approx( + report["target_gains_share"][name], abs=1e-5 + ) + assert report["achieved_gains_share"][name] == pytest.approx( + report["target_gains_share"][name], abs=tolerance + ) + ineligible = set(CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES) - set( + CGT_BADR_ELIGIBLE_TYPES + ) + for name in ineligible: + assert by_status["claimants"]["target_gains_share"][name] == 0.0 + assert by_status["claimants"]["achieved_gains_share"][name] == 0.0 assert evidence["value_counts"][CGT_ASSET_TYPE_RESIDENTIAL] == int( flagged.sum() ) assert len(evidence["composition_by_band"]) == 10 assert evidence["facts"]["resource_sha256"] == facts.resource_sha256 - assert evidence["seeds"] == { - "residential_flag": 553, - "badr_flag": 555, - "asset_type": 554, - } + # The residential flag is carried as weight by the split and the main + # type is assigned by a deterministic walk: only the BADR draw seeds. + assert evidence["seeds"] == {"badr_flag": 555} badr = evidence["badr"] assert set(badr["invariants"].values()) == {0} assert badr["totals"]["gains_target"] == pytest.approx( @@ -781,12 +917,12 @@ def test_flags_and_types_every_liable_gainer_and_nobody_else(self) -> None: def test_is_deterministic_and_seed_sensitive(self) -> None: facts = load_hmrc_cgt_asset_type_facts() - frame = _frame(_synthetic_gains()) + frame = _split(_frame(_synthetic_gains()), facts) first, _ = assign_uk_cgt_asset_types(frame, facts, PARAMETERS, BADR_PARAMETERS) second, _ = assign_uk_cgt_asset_types(frame, facts, PARAMETERS, BADR_PARAMETERS) other, _ = assign_uk_cgt_asset_types( - frame, facts, PARAMETERS, BADR_PARAMETERS, asset_type_seed=999 + frame, facts, PARAMETERS, BADR_PARAMETERS, badr_seed=999 ) assert uk_frame_content_identity(first) == uk_frame_content_identity(second) @@ -797,17 +933,18 @@ def test_refuses_a_frame_that_cannot_hold_the_targets(self) -> None: frame = _frame([10_000.0, 20_000.0, 50_000.0], weights=[1.0, 1.0, 1.0]) with pytest.raises(ValueError, match="not below the liable taxpayer mass"): - assign_uk_cgt_asset_types(frame, facts, PARAMETERS, BADR_PARAMETERS) + _assign(frame, facts, PARAMETERS, BADR_PARAMETERS) def test_refuses_a_frame_already_classified(self) -> None: facts = load_hmrc_cgt_asset_type_facts() frame = _frame(_synthetic_gains()) - classified, _ = assign_uk_cgt_asset_types( - frame, facts, PARAMETERS, BADR_PARAMETERS - ) + classified, _ = _assign(frame, facts, PARAMETERS, BADR_PARAMETERS) with pytest.raises(ValueError, match="already carries"): assign_uk_cgt_asset_types(classified, facts, PARAMETERS, BADR_PARAMETERS) + # Without the split's arms the stage has nothing to type residential. + with pytest.raises(ValueError, match="run cgt_residential_split"): + assign_uk_cgt_asset_types(frame, facts, PARAMETERS, BADR_PARAMETERS) class TestStageContract: @@ -820,11 +957,8 @@ def test_manifest_operations_restate_the_reviewed_parameters(self) -> None: assert [operation.kind for operation in stage.operations] == list(expected) for operation in stage.operations: assert dict(operation.parameters) == expected[operation.kind] - assert tuple(stage.outputs) == ( - CGT_ASSET_TYPE_COLUMN, - CGT_RESIDENTIAL_GAINS_COLUMN, - CGT_BADR_GAINS_COLUMN, - ) + assert tuple(stage.outputs) == (CGT_ASSET_TYPE_COLUMN, CGT_BADR_GAINS_COLUMN) + assert "assign_residential_property_flag" not in expected roles = {artifact["role"]: artifact for artifact in stage.artifacts} assert roles["cgt_asset_type_facts"]["resource"] == HMRC_CGT_ASSET_TYPE_RESOURCE assert roles["cgt_asset_type_facts"]["runtime_sha256_required"] is True @@ -848,7 +982,6 @@ def test_transform_runs_from_the_resource_and_the_seam( stage = spec.sources.stage_map()[UK_CGT_ASSET_TYPE_STAGE_NAME] gains = _synthetic_gains(6_000) weights = np.full(gains.size, 300.0) - frame = _frame(gains, weights=weights) # The frame holds a few thousand gainers, so the Table 4.1 bands are # sized to it; the vendored Table 8 and Table 7 rows drive the rest. vendored = load_hmrc_cgt_asset_type_facts() @@ -861,6 +994,7 @@ def test_transform_runs_from_the_resource_and_the_seam( if name.startswith("table4_") }, ) + frame = _split(_frame(gains, weights=weights), facts) resolved: list[str] = [] def load_facts(): @@ -949,7 +1083,7 @@ def test_synthetic_facts_type_check_without_the_feed() -> None: ) frame = _frame(gains, weights=np.full(3_000, 5.0)) - _, summary = assign_uk_cgt_asset_types(frame, facts, PARAMETERS, BADR_PARAMETERS) + _, summary = _assign(frame, facts, PARAMETERS, BADR_PARAMETERS) assert summary.evidence()["residential"]["count_target_individuals_basis"] == ( pytest.approx(1_900.0) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_residential_split.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_residential_split.py new file mode 100644 index 000000000..f36cbf30d --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_residential_split.py @@ -0,0 +1,401 @@ +"""The CGT residential split: the residential flag carried as weight (microcosm#1063).""" + +from __future__ import annotations + +import dataclasses +from dataclasses import replace + +import numpy as np +import pandas as pd +import pytest + +from microcosm.build.country_spec import load_country_spec +from microcosm.build.source_manifest import SourceOperationSpec +from microcosm.build.uk_runtime import cgt_residential_split +from microcosm.build.uk_runtime.cgt_asset_type import ( + CGT_RESIDENTIAL_GAINS_COLUMN, + CGT_STOCK_HOUSEHOLD_COLUMNS, + CGT_STOCK_PERSON_COLUMNS, + load_hmrc_cgt_asset_type_facts, +) +from microcosm.build.uk_runtime.cgt_imputation import ( + UK_CGT_TAXABLE_INCOME_PROXY_COMPONENTS, + UKCGTPolicyParameters, +) +from microcosm.build.uk_runtime.cgt_residential_split import ( + CGT_RESIDENTIAL_CLONE_INDEX_COLUMN, + CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS, + CGT_RESIDENTIAL_PROBABILITY_COLUMN, + CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON, + CGT_RESIDENTIAL_SPLIT_STAGE_NAME, + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + UKCGTResidentialSplitStageTransform, + _assert_cgt_residential_split_stage_parameters, + cgt_residential_split_operation_parameters, + residential_arm_factors, + split_cgt_residential_households, +) +from microcosm.build.uk_runtime.national_frame import uk_national_frame +from microcosm.frame import WeightKind + +PARAMETERS = UKCGTPolicyParameters( + personal_allowance=12_570.0, + personal_allowance_taper_threshold=100_000.0, + personal_allowance_taper_rate=0.5, + annual_exempt_amount=3_000.0, + instant="2024-06-01", + source="test", +) + + +def _frame(gains, *, weights=None, people_per_household=1, stocks=None): + """Households of ``people_per_household`` persons, one gain per person.""" + + gains = np.asarray(gains, dtype=float) + rows = gains.size + households = rows // people_per_household + assert households * people_per_household == rows + household_of = np.repeat(np.arange(households, dtype="int64"), people_per_household) + person = pd.DataFrame( + { + "person_id": np.arange(rows, dtype="int64"), + "person_household_id": household_of, + "person_benunit_id": household_of, + "capital_gains": gains, + "age": np.full(rows, 45, dtype="int64"), + } + ) + for column in (*UK_CGT_TAXABLE_INCOME_PROXY_COMPONENTS, *CGT_STOCK_PERSON_COLUMNS): + person[column] = 0.0 + household = pd.DataFrame( + { + "household_id": np.arange(households, dtype="int64"), + "household_weight": ( + np.full(households, 60.0) + if weights is None + else np.asarray(weights, dtype=float) + ), + "region": np.full(households, "LONDON", dtype=object), + } + ) + for column in CGT_STOCK_HOUSEHOLD_COLUMNS: + household[column] = 0.0 + for column, values in (stocks or {}).items(): + table = person if column in CGT_STOCK_PERSON_COLUMNS else household + table[column] = np.asarray(values, dtype=float) + benunit = pd.DataFrame({"benunit_id": np.arange(households, dtype="int64")}) + return uk_national_frame( + person=person, benunit=benunit, household=household, time_period="2024" + ) + + +def _facts(frame, *, share: float = 0.3): + """Vendored facts with the Table 8 totals sized to the frame's liable mass.""" + + person = frame.table("person") + household = frame.table("household") + weights = pd.Series( + frame.weights_for("household").values, index=household["household_id"] + ) + person_weight = person["person_household_id"].map(weights).to_numpy(dtype=float) + gains = person["capital_gains"].to_numpy(dtype=float) + liable = gains > PARAMETERS.annual_exempt_amount + count = share * float(person_weight[liable].sum()) + total = share * float((person_weight[liable] * gains[liable]).sum()) + return dataclasses.replace( + load_hmrc_cgt_asset_type_facts(), + table8a_taxpayers_total=count, + table8a_gains_total=total, + table8b_individuals_taxpayers=count, + table8b_individuals_gains=total, + table8b_all_taxpayers=count, + table8b_all_gains=total, + ) + + +def _gains(rows: int = 3_000, seed: int = 1) -> np.ndarray: + rng = np.random.default_rng(seed) + gains = np.where(rng.random(rows) < 0.7, np.exp(rng.normal(10.5, 1.8, rows)), 0.0) + gains[:20] = -500.0 + return gains + + +def test_arm_factors_are_the_subset_products_and_sum_to_one() -> None: + assert residential_arm_factors(np.asarray([0.2])).tolist() == pytest.approx( + [0.8, 0.2] + ) + # Bit m of the arm index marks gainer m: arm 1 = first residential only, + # arm 2 = second only, arm 3 = both. + assert residential_arm_factors(np.asarray([0.2, 0.5])).tolist() == pytest.approx( + [0.4, 0.1, 0.4, 0.1] + ) + three = residential_arm_factors(np.asarray([0.1, 0.2, 0.3])) + assert three.size == 8 and three.sum() == pytest.approx(1.0) + assert three[7] == pytest.approx(0.1 * 0.2 * 0.3) + with pytest.raises(ValueError, match=r"lie in \[0, 1\]"): + residential_arm_factors(np.asarray([1.2])) + with pytest.raises(ValueError, match="at least one liable gainer"): + residential_arm_factors(np.asarray([])) + + +def test_split_carries_table_8a_as_identities_and_conserves_mass_bitwise() -> None: + gains = _gains() + frame = _frame(gains) + facts = _facts(frame) + + result = split_cgt_residential_households(frame, facts=facts, parameters=PARAMETERS) + + evidence = result.evidence() + identities = evidence["identities"] + # The solve puts the expectations on the targets; the arms realise the + # expectations exactly, in every gain band. + assert identities["count_solve_relative_error"] < 1e-6 + assert identities["gains_solve_relative_error"] < 1e-6 + assert identities["count_identity_relative_error"] < 1e-12 + assert identities["gains_identity_relative_error"] < 1e-12 + for band in evidence["bands"]: + assert band["achieved_count"] == pytest.approx( + band["expected_count"], rel=1e-12 + ) + assert band["achieved_gains"] == pytest.approx( + band["expected_gains"], rel=1e-12 + ) + # Mass: bitwise total, one record at declared factor one, every arm the + # product of its household's weight and the probabilities. + assert float(result.frame.weights_for("household").total) == float( + frame.weights_for("household").total + ) + record = result.frame.mass_log[-1] + assert record.reason == CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON + assert record.declared_factor == 1.0 and record.old_total == record.new_total + assert evidence["arm_weights_exact"] is True + household = result.frame.table("household") + person = result.frame.table("person") + weights = pd.Series( + result.frame.weights_for("household").values, index=household["household_id"] + ) + arms = household[HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE].to_numpy(dtype=bool) + liable = gains > PARAMETERS.annual_exempt_amount + assert int(arms.sum()) == int(liable.sum()) == evidence["arms_created"] + assert evidence["households_by_liable_gainers"] == {"1": int(liable.sum())} + multiplier = cgt_residential_split.id_multiplier_for_values( + frame.table("person")["person_id"], + frame.table("person")["person_household_id"], + frame.table("person")["person_benunit_id"], + frame.table("benunit")["benunit_id"], + frame.table("household")["household_id"], + ) + probability = person.set_index("person_id")[CGT_RESIDENTIAL_PROBABILITY_COLUMN] + for source in np.flatnonzero(liable): + p = float(probability.loc[source]) + assert 0.0 < p < 1.0 + assert weights.loc[source] == 60.0 * (1.0 - p) + assert weights.loc[source + multiplier] == 60.0 * p + assert ( + household.set_index("household_id").loc[ + source + multiplier, CGT_RESIDENTIAL_CLONE_INDEX_COLUMN + ] + == 1 + ) + # The residential arm's gainer carries the whole gain; nobody else does. + residential = person[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy() + on_arm = person["person_id"].to_numpy() >= multiplier + assert (residential[on_arm] == person["capital_gains"].to_numpy()[on_arm]).all() + assert (residential[~on_arm] == 0.0).all() + assert (probability.loc[np.flatnonzero(~liable)] == 0.0).all() + + +def test_households_with_several_liable_gainers_become_product_arms() -> None: + # Three households of two persons: both liable, one liable, none liable. + frame = _frame( + [50_000.0, 80_000.0, 90_000.0, 0.0, 1_000.0, 0.0], + weights=[10.0, 20.0, 30.0], + people_per_household=2, + ) + result = split_cgt_residential_households( + frame, facts=_facts(frame), parameters=PARAMETERS + ) + evidence = result.evidence() + assert evidence["households_by_liable_gainers"] == {"1": 1, "2": 1} + assert evidence["arms_created"] == 3 + 1 + household = result.frame.table("household").set_index("household_id") + person = result.frame.table("person") + weights = pd.Series( + result.frame.weights_for("household").values, index=household.index + ) + probability = person.set_index("person_id")[CGT_RESIDENTIAL_PROBABILITY_COLUMN] + p0, p1 = float(probability.loc[0]), float(probability.loc[1]) + multiplier = 10 + assert weights.loc[0] == pytest.approx(10.0 * (1 - p0) * (1 - p1)) + assert weights.loc[0 + multiplier] == pytest.approx(10.0 * p0 * (1 - p1)) + assert weights.loc[0 + 2 * multiplier] == pytest.approx(10.0 * (1 - p0) * p1) + assert weights.loc[0 + 3 * multiplier] == pytest.approx(10.0 * p0 * p1) + assert household.loc[[10, 20, 30], CGT_RESIDENTIAL_CLONE_INDEX_COLUMN].tolist() == [ + 1, + 2, + 3, + ] + by_person = person.set_index("person_id")[CGT_RESIDENTIAL_GAINS_COLUMN] + # Arm 1: gainer 0 residential; arm 2: gainer 1; arm 3: both. + assert by_person.loc[[0 + multiplier, 1 + multiplier]].tolist() == [50_000.0, 0.0] + assert by_person.loc[[0 + 2 * multiplier, 1 + 2 * multiplier]].tolist() == [ + 0.0, + 80_000.0, + ] + assert by_person.loc[[0 + 3 * multiplier, 1 + 3 * multiplier]].tolist() == [ + 50_000.0, + 80_000.0, + ] + # The unsplit household is untouched and carries the exact-total correction. + assert household.loc[2, HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE] is np.False_ + assert float(result.frame.weights_for("household").total) == 60.0 + assert weights.loc[2] == pytest.approx(30.0) + + +def test_split_refuses_too_many_gainers_an_already_split_or_classified_frame() -> None: + gains = [10_000.0] * (CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS + 1) + frame = _frame(gains, people_per_household=len(gains)) + with pytest.raises(ValueError, match="more than"): + split_cgt_residential_households( + frame, facts=_facts(frame), parameters=PARAMETERS + ) + frame = _frame(_gains(500)) + facts = _facts(frame) + once = split_cgt_residential_households(frame, facts=facts, parameters=PARAMETERS) + with pytest.raises(ValueError, match="already ran"): + split_cgt_residential_households(once.frame, facts=facts, parameters=PARAMETERS) + person = frame.table("person").assign(**{CGT_RESIDENTIAL_GAINS_COLUMN: 0.0}) + classified = uk_national_frame( + person=person, + benunit=frame.table("benunit"), + household=frame.table("household"), + time_period="2024", + household_weights=frame.weights_for("household").values, + ) + with pytest.raises(ValueError, match="already ran"): + split_cgt_residential_households(classified, facts=facts, parameters=PARAMETERS) + + +def test_split_is_stable_under_person_order_and_follows_the_stock_signal() -> None: + gains = _gains(1_500) + frame = _frame(gains) + facts = _facts(frame) + first = split_cgt_residential_households(frame, facts=facts, parameters=PARAMETERS) + reversed_person = frame.table("person").iloc[::-1].reset_index(drop=True) + second = split_cgt_residential_households( + uk_national_frame( + person=reversed_person, + benunit=frame.table("benunit"), + household=frame.table("household"), + time_period="2024", + household_weights=frame.weights_for("household").values, + ), + facts=facts, + parameters=PARAMETERS, + ) + + def arms(result): + household = result.frame.table("household") + return pd.Series( + result.frame.weights_for("household").values, + index=household["household_id"].to_numpy(), + ).sort_index() + + # Bitwise, not close: the E8 identity receipt compares two orders of the + # same persons with ``Series.equals`` (microcosm#1063). + pd.testing.assert_series_equal(arms(first), arms(second), check_exact=True) + shuffled_person = ( + frame.table("person") + .iloc[np.random.default_rng(1063).permutation(gains.size)] + .reset_index(drop=True) + ) + third = split_cgt_residential_households( + uk_national_frame( + person=shuffled_person, + benunit=frame.table("benunit"), + household=frame.table("household"), + time_period="2024", + household_weights=frame.weights_for("household").values, + ), + facts=facts, + parameters=PARAMETERS, + ) + pd.testing.assert_series_equal(arms(first), arms(third), check_exact=True) + assert third.frame.table("household")["household_id"].tolist() == ( + first.frame.table("household")["household_id"].tolist() + ) + # A gainer who shows the stock the flag implies is more likely residential. + stocks = {"property_income": np.where(np.arange(gains.size) % 2 == 0, 1.0, 0.0)} + shifted = split_cgt_residential_households( + _frame(gains, stocks=stocks), facts=facts, parameters=PARAMETERS + ) + evidence = shifted.evidence() + assert ( + evidence["stock"]["expected_share_residential"] + > evidence["stock"]["share_liable"] + ) + assert evidence["identities"]["count_solve_relative_error"] < 1e-6 + + +def test_evidence_is_json_plain_and_the_stage_transform_runs_from_the_seam() -> None: + import json + + spec = load_country_spec("uk") + assert spec.sources is not None + stage = spec.sources.stage_map()[CGT_RESIDENTIAL_SPLIT_STAGE_NAME] + assert stage.grain == "household" + assert tuple(stage.outputs) == UKCGTResidentialSplitStageTransform.output_columns() + assert stage.rewrites == () + frame = _frame(_gains(800)) + transform = UKCGTResidentialSplitStageTransform( + stage=stage, facts=_facts(frame), parameters=PARAMETERS + ) + result = transform(frame) + assert result.weights_for("household").kind is WeightKind.IMPORTANCE + evidence = transform.checkpoint_metadata()["evidence"] + assert evidence["stage"] == CGT_RESIDENTIAL_SPLIT_STAGE_NAME + json.dumps(evidence) + assert evidence["maximum_liable_gainers_per_household"] == ( + CGT_RESIDENTIAL_MAXIMUM_LIABLE_GAINERS + ) + + +def test_operation_dictionary_restates_the_reviewed_design() -> None: + operations = dict(cgt_residential_split_operation_parameters()) + assert list(operations) == [ + "verify_vendored_fact_resource", + "split_liable_gainers_by_residential_probability", + ] + split = operations["split_liable_gainers_by_residential_probability"] + assert split["declared_factor"] == 1.0 and split["conservation"] == "exact_total" + assert split["reason"] == CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON + assert "no draw" in split["realization"] + assert split["flag_column"] == HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE + assert split["index_column"] == CGT_RESIDENTIAL_CLONE_INDEX_COLUMN + assert split["probability_column"] == CGT_RESIDENTIAL_PROBABILITY_COLUMN + assert split["output_column"] == CGT_RESIDENTIAL_GAINS_COLUMN + + +@pytest.mark.parametrize("parameter", ["realization", "declared_factor", "flag_column"]) +def test_drift_assert_covers_every_reviewed_parameter(parameter: str) -> None: + spec = load_country_spec("uk") + assert spec.sources is not None + stage = spec.sources.stage_map()[CGT_RESIDENTIAL_SPLIT_STAGE_NAME] + _assert_cgt_residential_split_stage_parameters(stage) + operations = list(stage.operations) + operation = operations[1] + operations[1] = SourceOperationSpec( + operation.kind, {**operation.parameters, parameter: "__drift__"} + ) + with pytest.raises(ValueError): + _assert_cgt_residential_split_stage_parameters( + replace(stage, operations=tuple(operations)) + ) + with pytest.raises(ValueError, match="Expected stage"): + _assert_cgt_residential_split_stage_parameters( + replace(stage, stage="cgt_support_split") + ) + with pytest.raises(ValueError, match="exactly its flag"): + _assert_cgt_residential_split_stage_parameters( + replace(stage, outputs=stage.outputs[:-1]) + ) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_structure.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_structure.py index b36203d16..d59d7b5d5 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_structure.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_cgt_structure.py @@ -641,6 +641,37 @@ def test_anchor_leaves_a_group_already_below_its_target_untouched() -> None: assert result.trimmed_households == 1 +def test_anchor_sums_do_not_depend_on_the_row_order() -> None: + """A pairwise float sum moves in its last bits when the rows are permuted, + which moved the anchor's bisection onto a neighbouring scale and the + anchored weights by a few ulps between two orderings of the same + households (the E8 identity receipt, microcosm#1063).""" + + from microcosm.build.uk_runtime.cgt_structure import ( + _order_free_sum, + _solve_group_factors, + ) + + # The case the helper exists for: a float sum that depends on the order. + awkward = np.array([1e16, 1.0, -1e16, 1.0]) + assert float(awkward.sum()) != float(awkward[[0, 2, 1, 3]].sum()) + assert _order_free_sum(awkward) == _order_free_sum(awkward[[0, 2, 1, 3]]) == 2.0 + + rng = np.random.default_rng(1063) + weights = rng.uniform(0.1, 3_000.0, 20_000) * 10.0 ** rng.integers(-3, 3, 20_000) + propensity = rng.uniform(0.001, 0.2, 20_000) + order = rng.permutation(weights.size) + assert _order_free_sum(weights) == _order_free_sum(weights[order]) + + target = 0.4 * _order_free_sum(weights) + scale, factors = _solve_group_factors(weights, propensity, target) + permuted_scale, permuted_factors = _solve_group_factors( + weights[order], propensity[order], target + ) + assert permuted_scale == scale + np.testing.assert_array_equal(permuted_factors, factors[order]) + + def test_anchor_is_deterministic_under_person_permutation() -> None: permutation = np.random.default_rng(5).permutation( (len(ANCHOR_PATTERN) + SUPPORT_COPY_COUNT) * 4 diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_child_benefit_take_up.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_child_benefit_take_up.py new file mode 100644 index 000000000..d5da1cc4e --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_child_benefit_take_up.py @@ -0,0 +1,406 @@ +"""Child Benefit claims and opt-outs after the SPI chain (microcosm#1063).""" + +from __future__ import annotations + +from dataclasses import replace + +import numpy as np +import pandas as pd +import pytest + +from microcosm.build.country_spec import load_country_spec +from microcosm.build.source_manifest import SourceStageSpec +from microcosm.build.uk_runtime.child_benefit_take_up import ( + CHILD_BENEFIT_ENGINE_VARIABLES, + CHILD_BENEFIT_STATISTICS_RESOURCE, + CHILD_BENEFIT_TAKE_UP_STAGE_NAME, + UKChildBenefitChargeThresholds, + UKChildBenefitStatistics, + UKChildBenefitTakeUpStageTransform, + _assert_stage_parameters, + assign_child_benefit_claims, + assign_child_benefit_opt_outs, + child_benefit_take_up_operation_parameters, + load_child_benefit_statistics, + redraw_child_benefit_take_up, + solve_child_benefit_claim_rates, +) +from microcosm.build.uk_runtime.frs_take_up import UK_TAKE_UP_SIGNAL_OUTPUTS +from microcosm.build.uk_runtime.ledger_fact_vendoring import load_vendor_selections +from microcosm.build.uk_runtime.national_frame import uk_national_frame +from microcosm.frame import WeightKind + +THRESHOLDS = UKChildBenefitChargeThresholds( + phase_out_start=60_000.0, phase_out_end=80_000.0 +) + + +def _statistics(rates: dict[int, float] | None = None) -> UKChildBenefitStatistics: + return UKChildBenefitStatistics( + claim_rates=rates or {0: 0.7, 1: 0.8, 2: 0.9}, + all_ages_claim_rate=0.8, + families_registered=1_000.0, + families_in_payment=900.0, + families_opted_out=100.0, + children_in_payment=1_500.0, + children_opted_out=150.0, + ) + + +def _families(count: int = 30_000, *, reporter_share: float = 0.3, seed: int = 5): + """Families of one or two children aged 0 to 2, one row per child.""" + + rng = np.random.default_rng(seed) + eldest = rng.integers(0, 3, size=count) + second = rng.random(count) < 0.5 + younger = np.where(second, rng.integers(0, 3, size=count), -1) + younger = np.where(younger > eldest, eldest, younger) + reporter = rng.random(count) < reporter_share + units = np.r_[np.arange(count), np.flatnonzero(second)] + ages = np.r_[eldest, younger[second]] + return eldest, reporter, units, ages, rng.random(count) + + +def test_solved_rates_reproduce_the_published_rate_at_every_age() -> None: + # Given families whose children's ages differ from the family's key age, + # when the rates are solved from the eldest age down, then the expected + # claimed share of the children of every age is the published one. + eldest, reporter, units, ages, _ = _families() + rates = {0: 0.7, 1: 0.8, 2: 0.9} + weights = np.ones(len(units)) + residual, rows = solve_child_benefit_claim_rates( + child_ages=ages, + child_key_ages=eldest[units], + child_reporter=reporter[units], + child_weights=weights, + claim_rates=rates, + ) + assert [row["age"] for row in rows] == [0, 1, 2] + assert not any(row["clipped"] for row in rows) + for age, rate in rates.items(): + of_age = ages == age + claimed = reporter[units][of_age].sum() + sum( + residual[key] * (of_age & ~reporter[units] & (eldest[units] == key)).sum() + for key in residual + ) + assert claimed / of_age.sum() == pytest.approx(rate, abs=1e-12) + # The eldest age has no older sibling: its rate is the plain residual. + top = ages == 2 + assert residual[2] == pytest.approx( + (0.9 * top.sum() - reporter[units][top].sum()) / (top & ~reporter[units]).sum() + ) + + +def test_solved_rates_are_weighted() -> None: + # One age, a reporter child of weight 1 and a non-reporter of weight 3: + # r = (0.5 * 4 - 1) / 3. + residual, rows = solve_child_benefit_claim_rates( + child_ages=np.asarray([0, 0]), + child_key_ages=np.asarray([0, 0]), + child_reporter=np.asarray([True, False]), + child_weights=np.asarray([1.0, 3.0]), + claim_rates={0: 0.5}, + ) + assert residual == {0: pytest.approx(1 / 3)} + assert rows[0]["eligible_children"] == 4.0 and rows[0]["reporter_children"] == 1.0 + + +def test_a_rate_the_draw_cannot_reach_is_clipped_and_receipted() -> None: + # Reporters alone exceed the published rate at age 0; at age 1 every + # family claiming still falls short of it. + residual, rows = solve_child_benefit_claim_rates( + child_ages=np.asarray([0, 0, 1, 1]), + child_key_ages=np.asarray([0, 0, 1, 1]), + child_reporter=np.asarray([True, False, False, False]), + child_weights=np.asarray([9.0, 1.0, 1.0, 1.0]), + claim_rates={0: 0.5, 1: 1.0}, + ) + assert residual == {0: 0.0, 1: 1.0} + by_age = {row["age"]: row for row in rows} + assert by_age[0]["clipped"] is True + assert by_age[0]["residual_rate_unclipped"] == pytest.approx(-4.0) + assert by_age[1]["clipped"] is False + with pytest.raises(ValueError, match=r"aged \[3\]"): + solve_child_benefit_claim_rates( + child_ages=np.asarray([3]), + child_key_ages=np.asarray([3]), + child_reporter=np.asarray([False]), + child_weights=np.asarray([1.0]), + claim_rates={0: 0.5}, + ) + + +def test_claims_keep_reporters_and_realise_the_published_rates() -> None: + eldest, reporter, units, ages, draws = _families() + # Every sixth unit has no eligible child. + childless = np.arange(len(eldest)) % 6 == 0 + keep = ~childless[units] + eldest = np.where(childless, -1, eldest) + claims, receipt = assign_child_benefit_claims( + child_units=units[keep], + child_ages=ages[keep], + eldest=eldest, + reporter=reporter, + weights=np.ones(len(eldest)), + draws=draws, + statistics=_statistics(), + ) + assert claims[reporter & ~childless].all() + # A unit without an eligible child is not a claiming family, reporter or not. + assert not claims[childless].any() + assert receipt["eligible_family_units"] == int((~childless).sum()) + for row in receipt["ages"]: + assert row["realized_rate"] == pytest.approx(row["published_rate"], abs=0.02) + assert receipt["realized_rate"] == pytest.approx(receipt["target_rate"], abs=0.01) + assert receipt["clipped_ages"] == [] + assert receipt["published_all_ages_rate"] == 0.8 + + +def test_opt_outs_take_the_published_share_from_fully_charged_families_first() -> None: + count = 20_000 + rng = np.random.default_rng(3) + claims = rng.random(count) < 0.9 + reporter = claims & (rng.random(count) < 0.4) + income = rng.choice([30_000.0, 70_000.0, 90_000.0], size=count, p=[0.6, 0.2, 0.2]) + weights = np.ones(count) + arguments = { + "claims": claims, + "reporter": reporter, + "highest_income": income, + "children": np.full(count, 2.0), + "weights": weights, + "draws": rng.random(count), + "thresholds": THRESHOLDS, + } + opt_out, receipt = assign_child_benefit_opt_outs( + statistics=_statistics(), **arguments + ) + # Only claiming, non-reporting families at or above full withdrawal. + assert (claims & ~reporter & (income >= 80_000.0))[opt_out].all() + assert receipt["target_share"] == pytest.approx(0.1) + assert receipt["realized_share"] == pytest.approx(0.1, abs=0.01) + assert receipt["pool_exhausted"] is False + assert [pool["pool"] for pool in receipt["pools"]] == ["fully_charged", "taper"] + assert receipt["pools"][1]["rate"] == 0.0 + assert receipt["children_per_opted_out_family"] == 2.0 + assert receipt["published_children_per_opted_out_family"] == 1.5 + + # A target heavier than the fully charged pool spills into the taper, and + # one heavier than both pools is recorded as a shortfall, never drawn from + # families outside the charge. + spill = UKChildBenefitStatistics( + **{**_statistics().__dict__, "families_opted_out": 150.0} + ) + opt_out, receipt = assign_child_benefit_opt_outs(statistics=spill, **arguments) + assert receipt["pools"][0]["rate"] == 1.0 + assert 0.0 < receipt["pools"][1]["rate"] < 1.0 + assert receipt["pool_exhausted"] is False + short = UKChildBenefitStatistics( + **{**_statistics().__dict__, "families_opted_out": 400.0} + ) + opt_out, receipt = assign_child_benefit_opt_outs(statistics=short, **arguments) + assert receipt["pool_exhausted"] is True and receipt["pool_shortfall_families"] > 0 + assert (opt_out == (claims & ~reporter & (income > 60_000.0))).all() + + +def test_statistics_and_thresholds_refuse_unusable_values() -> None: + with pytest.raises(ValueError, match=r"must lie in \[0, 1\]"): + _statistics({0: 1.2}) + with pytest.raises(ValueError, match="positive part of the registered"): + UKChildBenefitStatistics( + **{**_statistics().__dict__, "families_opted_out": 2_000.0} + ) + with pytest.raises(ValueError, match="0 < start < end"): + UKChildBenefitChargeThresholds(phase_out_start=80_000.0, phase_out_end=60_000.0) + + +class _StubEngine: + country = "uk" + + def __init__(self) -> None: + self.calls: list[tuple[tuple[str, ...], str]] = [] + + def materialize(self, frame, variables, period): + self.calls.append((tuple(variables), str(period))) + person = frame.table("person") + return { + "is_child_or_qualifying_young_person_for_child_benefit": ( + person["age"].to_numpy() < 16 + ), + "adjusted_net_income": person["stub_income"].to_numpy(dtype=float), + } + + +def _frame(): + # Unit 1: a reporter with a child. Unit 2: fully charged, with two + # children. Unit 3: a low-income family. Unit 4: no child, reports. + person = pd.DataFrame( + { + "person_id": [11, 12, 21, 22, 23, 31, 32, 41], + "person_benunit_id": [1, 1, 2, 2, 2, 3, 3, 4], + "person_household_id": [1, 1, 2, 2, 2, 3, 3, 4], + "age": [40, 2, 45, 1, 2, 30, 0, 50], + "child_benefit_reported": [1_300.0, 0, 0, 0, 0, 0, 0, 900.0], + "stub_income": [20_000.0, 0, 95_000.0, 0, 0, 15_000.0, 0, 10_000.0], + } + ) + benunit = pd.DataFrame( + { + "benunit_id": [1, 2, 3, 4], + "would_claim_child_benefit": [False, False, True, True], + "child_benefit_opts_out": [True, False, True, True], + } + ) + household = pd.DataFrame({"household_id": [1, 2, 3, 4], "region": ["WALES"] * 4}) + return uk_national_frame( + person=person, + benunit=benunit, + household=household, + household_weights=np.asarray([10.0, 10.0, 10.0, 10.0]), + weight_kind=WeightKind.IMPORTANCE, + time_period="2024", + ) + + +def test_redraw_reads_the_engine_once_and_rewrites_only_the_two_flags() -> None: + engine = _StubEngine() + frame = _frame() + # Every family claims and half of the claiming families opt out, so the + # one fully charged family is certain to. + statistics = UKChildBenefitStatistics( + **{ + **_statistics({0: 1.0, 1: 1.0, 2: 1.0}).__dict__, + "families_opted_out": 500.0, + } + ) + + result = redraw_child_benefit_take_up( + frame, engine=engine, statistics=statistics, thresholds=THRESHOLDS + ) + + assert engine.calls == [(CHILD_BENEFIT_ENGINE_VARIABLES, "2024")] + after = result.frame.table("benunit").set_index("benunit_id") + # The reporter is paid; the fully charged family claims and opts out, so + # it is not; the low-income family is paid; the unit without a child + # keeps its early draw and is never opted out. + assert after["would_claim_child_benefit"].to_dict() == { + 1: True, + 2: False, + 3: True, + 4: True, + } + assert after["child_benefit_opts_out"].to_dict() == { + 1: False, + 2: True, + 3: False, + 4: False, + } + for entity in ("person", "household"): + pd.testing.assert_frame_equal( + result.frame.table(entity), frame.table(entity), check_exact=True + ) + for name in CHILD_BENEFIT_ENGINE_VARIABLES: + assert name not in result.frame.table("person") + assert name not in result.frame.table("benunit") + evidence = result.evidence() + assert evidence["claims"]["eligible_family_units"] == 3 + assert evidence["claims"]["eligible_children"] == 40.0 + assert evidence["opt_outs"]["opted_out_families"] == 10.0 + assert evidence["opt_outs"]["opted_out_children"] == 20.0 + assert evidence["in_payment"]["families"] == 20.0 + assert evidence["in_payment"]["children"] == 20.0 + assert evidence["reporters_without_eligible_child"] == { + "units": 1, + "weighted_units": 10.0, + } + assert evidence["changed_units"] == { + "would_claim_child_benefit": 1, + "child_benefit_opts_out": 4, + } + + +def _stage() -> SourceStageSpec: + spec = load_country_spec("uk") + assert spec.sources is not None + return spec.sources.stage_map()[CHILD_BENEFIT_TAKE_UP_STAGE_NAME] + + +def test_committed_manifest_declares_the_stage_its_resource_and_rewrites() -> None: + stage = _stage() + + assert stage.grain == "benunit" + assert stage.outputs == () + assert stage.rewrites == ("would_claim_child_benefit", "child_benefit_opts_out") + assert [artifact["resource"] for artifact in stage.artifacts] == [ + CHILD_BENEFIT_STATISTICS_RESOURCE + ] + _assert_stage_parameters(stage) + assert { + operation.kind: dict(operation.parameters) for operation in stage.operations + } == child_benefit_take_up_operation_parameters() + transform = UKChildBenefitTakeUpStageTransform( + stage=stage, + engine=_StubEngine(), + statistics=_statistics({0: 1.0, 1: 1.0, 2: 1.0}), + thresholds=THRESHOLDS, + ) + transform(_frame()) + assert transform.checkpoint_metadata()["evidence"]["stage"] == ( + CHILD_BENEFIT_TAKE_UP_STAGE_NAME + ) + # The stage runs straight after the Pension Credit redraw. + roster = [entry.stage for entry in load_country_spec("uk").sources.stages] + assert roster[roster.index("pension_credit_take_up") + 1] == stage.stage + + +def test_stage_parameters_refuse_drift() -> None: + stage = _stage() + operations = [] + for operation in stage.operations: + payload = {"kind": operation.kind, **dict(operation.parameters)} + if operation.kind == "assign_opt_out_by_charge_income": + payload["pools"] = ["taper", "fully_charged"] + operations.append(payload) + drifted = SourceStageSpec.from_mapping({**stage.__dict__, "operations": operations}) + with pytest.raises(ValueError, match=r"operation\(s\) \['assign_opt_out"): + _assert_stage_parameters(drifted) + with pytest.raises(ValueError, match="expected 'child_benefit_take_up'"): + _assert_stage_parameters(replace(stage, stage="frs_take_up")) + with pytest.raises(ValueError, match="exactly the rewrites"): + _assert_stage_parameters(replace(stage, rewrites=("child_benefit_opts_out",))) + + +def test_vendored_statistics_are_the_august_2025_release() -> None: + statistics = load_child_benefit_statistics() + + # Table 15, May 2025: single years 0 to 19 and the all-ages rate. + assert sorted(statistics.claim_rates) == list(range(20)) + assert statistics.all_ages_claim_rate == pytest.approx(0.866) + assert statistics.claim_rates[0] == pytest.approx(0.688) + assert statistics.claim_rates[15] == pytest.approx(0.966) + assert statistics.claim_rates[19] == pytest.approx(0.497) + # August 2025 caseload: registered families are those in payment plus + # those opted out. + assert statistics.families_registered == 7_552_330 + assert statistics.families_opted_out == 684_635 + assert statistics.families_in_payment == 6_867_695 + assert ( + statistics.families_in_payment + statistics.families_opted_out + == statistics.families_registered + ) + assert statistics.children_in_payment == 11_730_730 + assert statistics.children_opted_out == 999_680 + assert statistics.opt_out_share == pytest.approx(684_635 / 7_552_330) + entry = next( + resource + for resource in load_vendor_selections()["resources"] + if resource["resource"] == CHILD_BENEFIT_STATISTICS_RESOURCE + ) + assert entry["consumers"] == ["uk_runtime.child_benefit_take_up"] + + +def test_terminal_signal_gate_leaves_the_child_benefit_flags_to_the_stage_gate() -> ( + None +): + outputs = {output for _, output, _ in UK_TAKE_UP_SIGNAL_OUTPUTS} + assert not outputs & {"would_claim_child_benefit", "child_benefit_opts_out"} diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_education.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_education.py index 2067ff1c4..0ba3fe2db 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_education.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_education.py @@ -80,6 +80,15 @@ def test_training_qyp_ema_and_benefits_in_own_right() -> None: result = derive_frs_education(person, raw) assert result["is_in_approved_training"].tolist() == [True, False, False] + # FRS 2024-25 renames the variable to train2 and recodes it: 9 is "None + # of these" and 10 the Kickstart Scheme (microcosm#1063 c9). + recoded = derive_frs_education( + person, raw.drop(columns=["train"]).assign(train2=[9, 10, 1]) + ) + assert recoded["is_in_approved_training"].tolist() == [False, True, True] + assert derive_frs_education(person, raw.drop(columns=["train"]))[ + "is_in_approved_training" + ].tolist() == [False, False, False] assert result["age_started_or_accepted_current_education_or_training"].tolist() == [ 18, 1000, diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_spine.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_spine.py index ce94fbd8d..f50bc194f 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_spine.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_frs_spine.py @@ -2336,14 +2336,14 @@ def test_e8_manifest_seeds_all_reach_the_build_sidecar_harvester() -> None: "uc_deduction_type_random_draw": 0, } assert "cgt_support_split" not in declared - assert declared["spi_income_band_donors"] == { - "stack_income_band_donor_households": 3 - } + assert declared["spi_income_band_donors"] == {"spi_income_band_donor_draw": 3} + assert declared["hmrc_spi_income_spine"]["spi_income_band_donor_leaf_draw"] == 44 assert declared["hmrc_cgt_gains_spine"] == {"within_band_draws": 552} + # The residential flag is carried as weight by cgt_residential_split + # (microcosm#1063), which seeds nothing; only the BADR draw remains. + assert "cgt_residential_split" not in declared assert declared["hmrc_cgt_asset_type_spine"] == { - "assign_residential_property_flag": 553, "assign_badr_qualifying_gains": 555, - "assign_main_asset_type": 554, } # The #970 incidence anchor is deterministic and consumes no seed. assert "cgt_incidence_anchor" not in declared diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_graph.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_graph.py index 32cccf01b..d1d5b4981 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_graph.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_graph.py @@ -44,11 +44,12 @@ def test_uk_spine_graph_contains_every_manifest_stage() -> None: # #791 frs_relationships, #725 hmrc_cgt_asset_type_spine, #970 # cgt_incidence_anchor, #930 nts_bus_travel and the income-anchor lane's # (PolicyEngine/chronicle#280) spi_income_band_donors stages, the #1012 - # spi_housing_shell, the #1003 was_lisa and the microcosm#1069 - # pension_credit_take_up; the frs_hmrc_retained_leaves / hmrc_spi_income - # certified-pair alternatives are retired (#901), so the manifest roster is - # the graph roster. - assert len(expected) == 36 + # spi_housing_shell, the #1003 was_lisa, the microcosm#1069 + # pension_credit_take_up and the microcosm#1063 child_benefit_take_up; the + # frs_hmrc_retained_leaves / hmrc_spi_income certified-pair alternatives + # are retired (#901), so the manifest roster is the graph roster. + # microcosm#1063 also adds cgt_residential_split after the anchor. + assert len(expected) == 38 assert {"frs_hmrc_retained_leaves", "hmrc_spi_income"}.isdisjoint(expected) assert set(expected) <= ids assert {"frs_hmrc_retained_leaves", "hmrc_spi_income"}.isdisjoint(ids) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_identity_stability_receipts.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_identity_stability_receipts.py index fb7d8fa64..af618acc9 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_identity_stability_receipts.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_identity_stability_receipts.py @@ -120,6 +120,22 @@ def test_a_complete_artifact_receipts_green(self) -> None: "source_household_key", ] + def test_the_income_band_donors_sit_at_their_own_clone_index(self) -> None: + # The donors carry the synthetic flag of the SPI support copy but are + # stacked at clone index 2; reading every synthetic household as index + # 1 failed the receipt on every spine built since the donor stage. + tool = _load_tool() + frame = _frame() + household = frame.table("household") + household["household_is_spi_income_band_donor"] = [False, True] + household["household_support_clone_index"] = [0, 2] + receipt = tool.e7_identity_receipt(frame, permutation_seed=7) + assert receipt["matches_stored_columns"] is True + + household["household_support_clone_index"] = [0, 1] + receipt = tool.e7_identity_receipt(frame, permutation_seed=7) + assert receipt["matches_stored_columns"] is False + def test_an_artifact_without_the_e7_layer_is_refused(self) -> None: # Previously this returned a green receipt over an empty comparison. tool = _load_tool() @@ -459,6 +475,147 @@ def test_cloned_households_are_excluded_from_the_recompute(self) -> None: assert receipt["matches_stored_columns"] is True +class TestPreSalarySacrificeRestoration: + """A receipt of a stage before salary_sacrifice bands on pre-conversion pay. + + The conversion rewrites a converted record's pay and employee contribution + in place after the CGT stages banded on them (microcosm#1069 c9); the + stage keeps the pre-conversion pay on its carrier and the tool reverses + the rewrite before recomputing (microcosm#1063). + """ + + @staticmethod + def _converted(frame): + """The split-then-cloned scenario after a conversion on household 6. + + Household 6 is the band-37,700 family the split divides (wealth 20 over + household 5's 10). Its carrier sacrifices enough pay to fall into band + 0, the way the stage rewrites a converted record: pay lowered, + employee contribution zeroed and moved whole into the salary-sacrifice + column, the pre-conversion pay kept on the carrier. + """ + + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_OUTPUT, + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) + + person = frame.table("person").copy() + person["employee_pension_contributions"] = 0.0 + person[SALSAC_OUTPUT] = 0.0 + person[SALSAC_PRE_CONVERSION_PAY_COLUMN] = 0.0 + members = person.loc[person["person_household_id"] == 6] + carrier = members.sort_values( + ["age", "person_id"], ascending=[False, True] + ).index[0] + pay = float(person.loc[carrier, "employment_income"]) + sacrificed = pay - float(BAND_INCOMES[0]) + assert sacrificed > 0.0 + person.loc[carrier, "employment_income"] = pay - sacrificed + person.loc[carrier, SALSAC_OUTPUT] = sacrificed + person.loc[carrier, SALSAC_PRE_CONVERSION_PAY_COLUMN] = pay + rewritten = uk_national_frame( + person=person, + benunit=frame.table("benunit").copy(), + household=frame.table("household").copy(), + time_period="2024", + weight_kind=WeightKind.IMPORTANCE, + household_weights=frame.weights_for("household").values, + mass_log=frame.mass_log, + ) + return rewritten, int(person.loc[carrier, "person_id"]), pay, sacrificed + + def test_the_support_split_receipt_bands_on_the_restored_pay(self) -> None: + tool = _load_tool() + _, frame = _split_then_cloned() + rewritten, _, _, _ = self._converted(frame) + + def receipt(target): + problems: dict[str, object] = {} + tool._e8_support_split( + target, + problems, + distribution=support_distribution(_SPLIT_TOP_CELLS), + parameters=PARAMETERS, + permutation_seed=7, + ) + return problems + + # On the stored pay the rule moves household 6 to band 0, where the + # 500-wealth households outrank it, and divides household 5 instead. + assert receipt(rewritten) == { + "support_split_selection_stored": {"missing": 1, "extra": 1} + } + assert receipt(tool._pre_salary_sacrifice_frame(rewritten)) == {} + + def test_the_restoration_reverses_the_conversion_exactly(self) -> None: + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_OUTPUT, + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) + + tool = _load_tool() + _, frame = _split_then_cloned() + rewritten, carrier_id, pay, sacrificed = self._converted(frame) + restored = tool._pre_salary_sacrifice_frame(rewritten) + person = restored.table("person").set_index("person_id") + assert person.loc[carrier_id, "employment_income"] == pay + assert person.loc[carrier_id, "employee_pension_contributions"] == sacrificed + assert person.loc[carrier_id, SALSAC_OUTPUT] == 0.0 + others = person.drop(index=carrier_id) + before = rewritten.table("person").set_index("person_id").drop(index=carrier_id) + pd.testing.assert_frame_equal(others, before, check_exact=True) + # The carrier itself is untouched, the weights and mass log pass through. + assert (person[SALSAC_PRE_CONVERSION_PAY_COLUMN] > 0).sum() == 1 + assert restored.weights_for("household").values.tolist() == ( + rewritten.weights_for("household").values.tolist() + ) + assert restored.mass_log == rewritten.mass_log + # Scoping to a stage before salary_sacrifice restores; a later stage + # keeps the stored pay; an artifact without the carrier passes through. + split_view = tool._frame_as_stage_saw(rewritten, "cgt_support_split") + assert ( + split_view.table("person") + .set_index("person_id") + .loc[carrier_id, "employment_income"] + == pay + ) + later_view = tool._frame_as_stage_saw(rewritten, "student_loans") + assert ( + later_view.table("person") + .set_index("person_id") + .loc[carrier_id, "employment_income"] + == pay - sacrificed + ) + assert tool._pre_salary_sacrifice_frame(frame) is frame + + def test_a_carrier_disagreeing_with_the_columns_is_refused(self) -> None: + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) + + tool = _load_tool() + _, frame = _split_then_cloned() + rewritten, carrier_id, _, _ = self._converted(frame) + person = rewritten.table("person").copy() + # A positive employee contribution on a record the carrier calls + # converted is not the state the conversion leaves. + person.loc[ + person["person_id"] == carrier_id, "employee_pension_contributions" + ] = 1.0 + tampered = uk_national_frame( + person=person, + benunit=rewritten.table("benunit"), + household=rewritten.table("household"), + time_period="2024", + weight_kind=WeightKind.IMPORTANCE, + household_weights=rewritten.weights_for("household").values, + mass_log=rewritten.mass_log, + ) + with pytest.raises(ValueError, match=SALSAC_PRE_CONVERSION_PAY_COLUMN): + tool._pre_salary_sacrifice_frame(tampered) + + class TestRosterOrderedScoping: """E5/E6 run after the SPI stack and before the CGT layers.""" diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_ladder_rowwise_clone.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_ladder_rowwise_clone.py index 6faf0cfb7..65b7d23ea 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_ladder_rowwise_clone.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_ladder_rowwise_clone.py @@ -564,6 +564,8 @@ def test_driver_ladder_spine_lineage_plan_manifest_parity( "household_is_capital_gains_clone", "household_is_cgt_support_copy", "cgt_support_copy_index", + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", ], "distinct_source_households": 3, "distinct_by_support_channel": {"frs": 1, "spi": 2}, @@ -571,6 +573,7 @@ def test_driver_ladder_spine_lineage_plan_manifest_parity( "household_is_spi_synthetic": 2, "household_is_capital_gains_clone": 1, "household_is_cgt_support_copy": 1, + "household_is_cgt_residential_clone": 0, }, } diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_lcfs_consumption.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_lcfs_consumption.py index a104bad6a..712c841df 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_lcfs_consumption.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_lcfs_consumption.py @@ -143,6 +143,47 @@ def test_recipient_has_fuel_is_conditioned_on_vehicle_count_and_deterministic() assert second.tolist() == first.tolist() +def test_negative_donor_consumption_is_floored_with_a_receipt() -> None: + """microcosm#1063 c9: a diary spend netting refunds below zero is raised to + the declared floor before the clip ranges and the imputation see it.""" + from microcosm.build.uk_runtime.lcfs_consumption import ( + CONSUMPTION_VARIABLE_RENAMES, + floor_negative_donor_consumption, + ) + + columns = list(CONSUMPTION_VARIABLE_RENAMES.values()) + donor = pd.DataFrame({column: [10.0, 20.0, 30.0] for column in columns}) + donor.loc[1, "housing_water_and_electricity_consumption"] = -14_165.0 + donor.loc[2, "housing_water_and_electricity_consumption"] = -1.0 + + floored, receipt = floor_negative_donor_consumption(donor) + + assert floored["housing_water_and_electricity_consumption"].tolist() == [ + 10.0, + 0.0, + 0.0, + ] + for column in columns: + if column != "housing_water_and_electricity_consumption": + assert floored[column].tolist() == [10.0, 20.0, 30.0] + assert receipt["floor"] == 0.0 + assert receipt["rows_raised"] == 2 + assert receipt["remaining_negative_rows"] == 0 + housing = receipt["columns"]["housing_water_and_electricity_consumption"] + assert housing == { + "rows_raised": 2, + "negative_mass": -14_166.0, + "minimum_before": -14_165.0, + } + assert receipt["columns"]["food_and_non_alcoholic_beverages_consumption"] == { + "rows_raised": 0, + "negative_mass": 0.0, + "minimum_before": 10.0, + } + with pytest.raises(ValueError, match="missing consumption column"): + floor_negative_donor_consumption(donor.drop(columns=[columns[0]])) + + def test_support_clip_exempts_raked_energy_columns() -> None: donor = pd.DataFrame( {column: [1.0, 5.0] for column in UK_LCFS_CONSUMPTION_TARGET_COLUMNS} diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_measure_simulation.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_measure_simulation.py index 77d9f0a5f..f18cfba39 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_measure_simulation.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_measure_simulation.py @@ -576,7 +576,7 @@ def test_exclusion_applier_warns_within_week_of_expiry(): "hmrc.salary_sacrifice.": 3, "_1_000_000_to_inf": 10, "slc.": 5, - "dwp/uc_payment_dist/": 17, + "dwp/uc_payment_dist/": 19, "obr.universal_credit_": 2, # microcosm#890 E1 (2026-09-11): the all-road-users obr.fuel_duties row # is signed out of the reference surface (target_reference_signed_ @@ -635,7 +635,7 @@ def test_exclusion_applier_warns_within_week_of_expiry(): def test_packaged_exclusions_load(): exclusions = load_uk_calibration_measure_exclusions() names = [entry["name"] for entry in exclusions] - assert len(names) == len(set(names)) == 60 + assert len(names) == len(set(names)) == 62 band_h_region_cells = [ entry for entry in exclusions @@ -683,11 +683,13 @@ def test_packaged_exclusions_load(): assert entry["expires_on"] == "2026-11-26", entry["name"] assert not [n for n in names if n.startswith("ons.household_composition.")] - # The 2026-09-03 tranche is #762's A16: five unreachable national rows, + # The 2026-09-03 tranche was #762's A16: five unreachable national rows, # a one-month window, each row tracked on its spine-defect issue. Two of # them (housing benefit, JSA) were re-adjudicated on 2026-09-16 with the - # mechanism receipts (#882) and moved to the 2026-12-08 clock; the other - # three still lapse on 2026-10-03. + # mechanism receipts (#882) and moved to the 2026-12-08 clock; the + # savings-interest row moved to the 2026-09-22 concept clock; the two + # plan-2 borrower rows were renewed one month on 2026-10-02 (microcosm#1063 + # stack), so nothing remains on the 2026-09-03 clock. a16_issues = { "ons.savings_interest_income": "microcosm#866", "obr.housing_benefit": "microcosm#867", @@ -695,12 +697,32 @@ def test_packaged_exclusions_load(): "slc.borrowers.plan_2_above_threshold": "microcosm#868", "dwp.jsa_claimants": "microcosm#869", } - a16 = [e for e in exclusions if e["approved_on"] == "2026-09-03"] - assert sorted(e["name"] for e in a16) == sorted(_A16_UNREACHABLE_ROWS[:2]) - for entry in a16: - assert entry["expires_on"] == "2026-10-03", entry["name"] + assert [e for e in exclusions if e["approved_on"] == "2026-09-03"] == [] + today = [e for e in exclusions if e["approved_on"] == "2026-10-02"] + # The single-row UC payment cell joined the uk-data#452 class on the + # #1063 stack arm (one supporting household, lost to a draw reshuffle). + thin = [ + e + for e in today + if e["name"] + in ( + "dwp/uc_payment_dist/SINGLE_annual_payment_28_800_to_30_000", + "dwp/uc_payment_dist/COUPLE_NO_CHILDREN_annual_payment_22_800_to_24_000", + ) + ] + assert len(thin) == 2 + for entry in thin: + assert entry["tracking"] == "uk-data#452" + assert entry["expires_on"] == "2026-11-26" + assert "support" in entry["reason"] + renewed = [e for e in today if e not in thin] + assert sorted(e["name"] for e in renewed) == sorted(_A16_UNREACHABLE_ROWS[:2]) + for entry in renewed: + assert entry["expires_on"] == "2026-11-03", entry["name"] assert entry["tracking"] == a16_issues[entry["name"]], entry["name"] assert "A16" in entry["adjudication"], entry["name"] + assert "renewed one month on 2026-10-02" in entry["adjudication"], entry["name"] + assert "microcosm#1063" in entry["adjudication"], entry["name"] for name in _A16_READJUDICATED_ROWS: entry = next(e for e in exclusions if e["name"] == name) assert entry["approved_on"] == "2026-09-16", name diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_national_graph.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_national_graph.py index 110b0ba1b..0b5d20947 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_national_graph.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_national_graph.py @@ -35,6 +35,9 @@ register_uk_national_kernels, uk_national_graph, ) +from microcosm.build.uk_runtime.measure_simulation import ( + load_uk_calibration_measure_exclusions, +) from microcosm.build.uk_runtime.national_calibration import ( CalibrationFrameAdapter, national_calibration_mass_reason, @@ -45,6 +48,14 @@ uk_national_target_loss_weights, ) from microcosm.build.uk_runtime.national_frame import write_uk_national_frame +from microcosm.build.uk_runtime.terminal_gates import ( + uk_default_degenerate_reviewed_exclusions, + uk_default_target_fit_reviewed_exclusions, +) +from microcosm.build.uk_runtime.weighted_integrity import ( + uk_default_input_mass_reviewed_exclusions, + uk_default_qrf_tail_reviewed_exclusions, +) from microcosm.calibrate import calibrate from microcosm.calibrate.artifacts import decode_problem from microcosm.frame import WeightKind @@ -140,6 +151,53 @@ def _registry_for_run(): return kernels +def _review_date() -> str: + """The latest approval across the committed exclusion registers. + + The seam battery evaluates the real registers on the graph's review + date, and a reviewed entry approved after that date is refused as not + yet in force (the South East target-fit deferral, approved 2026-10-02, + failed a review at 2026-09-29; microcosm#1063). Reviewing at the latest + approval keeps every entry in force without pinning a calendar date + that the next approval would break; an expired entry still fails, as + the real build's review would, which is the register's discipline. + """ + + def records(value): + if hasattr(value, "approved_on"): + yield value + elif isinstance(value, dict) and "approved_on" in value: + yield value + elif hasattr(value, "values"): + for item in value.values(): + yield from records(item) + elif isinstance(value, (list, tuple)): + for item in value: + yield from records(item) + + def field(record, name): + return record[name] if isinstance(record, dict) else getattr(record, name) + + entries = [ + record + for register in ( + uk_default_target_fit_reviewed_exclusions(), + uk_default_degenerate_reviewed_exclusions(), + uk_default_input_mass_reviewed_exclusions(), + uk_default_qrf_tail_reviewed_exclusions(), + load_uk_calibration_measure_exclusions(), + ) + for record in records(register) + ] + latest_approval = max(str(field(record, "approved_on")) for record in entries) + earliest_expiry = min(str(field(record, "expires_on")) for record in entries) + assert latest_approval < earliest_expiry, (latest_approval, earliest_expiry) + return latest_approval + + +REVIEW_DATE = _review_date() + + def _run(tmp_path, monkeypatch, config=None, *, endpoint=None): pytest.importorskip("tables") frame = seam._frame() @@ -150,7 +208,7 @@ def _run(tmp_path, monkeypatch, config=None, *, endpoint=None): config or _config(), spine=bound_spine_graph(frame), spine_population="uk.full.spine_checkpoint", - review_date="2026-09-29", + review_date=REVIEW_DATE, ) graph = national.graph if endpoint is not None: @@ -186,7 +244,7 @@ def test_national_graph_composes_the_bound_checkpoint_line(): _config(), spine=bound_spine_graph(frame), spine_population="uk.full.spine_checkpoint", - review_date="2026-09-29", + review_date=REVIEW_DATE, ) compiled = compile_graph(national.graph) assert list(compiled.order) == [ @@ -243,7 +301,7 @@ def test_national_graph_refuses_a_population_without_bound_provenance(): _config(), spine=unbound, spine_population="uk.full.spine_checkpoint", - review_date="2026-09-29", + review_date=REVIEW_DATE, ) @@ -413,7 +471,7 @@ def test_national_problem_refuses_an_empty_register(tmp_path, monkeypatch): _config(), spine=bound_spine_graph(frame), spine_population="uk.full.spine_checkpoint", - review_date="2026-09-29", + review_date=REVIEW_DATE, ) with pytest.raises(Exception, match="selects no target"): run_graph( @@ -477,7 +535,7 @@ def test_national_gate_node_reads_its_scope_from_the_posture(): _config(), spine=bound_spine_graph(seam._frame()), spine_population="uk.full.spine_checkpoint", - review_date="2026-09-29", + review_date=REVIEW_DATE, ) node = national.graph.node(NATIONAL_GATES_NODE) assert json.loads(node.params["gate_scope"]) == list( diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_national_sampling.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_national_sampling.py index ea8bcaacc..48a940208 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_national_sampling.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_national_sampling.py @@ -249,6 +249,9 @@ def _spine_family_frame( "household_is_spi_synthetic": [row["spi"] for row in rows], "household_is_capital_gains_clone": [row["cgt"] for row in rows], "household_is_cgt_support_copy": [row["band_donor"] for row in rows], + "household_is_cgt_residential_clone": [ + row.get("residential_arm", False) for row in rows + ], "region": [row["region"] for row in rows], "household_weight": [row["weight"] for row in rows], } diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_regional_uprating.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_regional_uprating.py index a9eba67a6..100036c32 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_regional_uprating.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_regional_uprating.py @@ -129,3 +129,40 @@ def test_regional_property_uprating_factor_excludes_spi_support_rows() -> None: assert uprated["property_wealth"].tolist() == pytest.approx( [200.0, 600.0, 3_000.0, 50.0] ) + + +def test_regional_property_uprating_keeps_the_total_the_sum_of_its_components(): + """The factor scales the total and every component of it the frame carries + (microcosm#1063), so the total an owner holds stays the sum of its parts; + a renter's other property is untouched, as before.""" + + household = pd.DataFrame( + { + "household_id": [1, 2], + "region": ["LONDON", "LONDON"], + "main_residence_value": [100.0, 0.0], + "owned_land": [10.0, 4.0], + "other_residential_property_value": [30.0, 5.0], + "non_residential_property_value": [10.0, 1.0], + "property_wealth": [150.0, 10.0], + } + ) + resource = {"values": [{"region": "LONDON", "avg_house_price": 300.0}]} + + uprated = uprate_household_property_by_region(household, resource) + + components = [ + "main_residence_value", + "owned_land", + "other_residential_property_value", + "non_residential_property_value", + ] + assert uprated.loc[0, components].tolist() == [300.0, 30.0, 90.0, 30.0] + assert uprated.loc[0, "property_wealth"] == pytest.approx(450.0) + assert uprated.loc[1, [*components, "property_wealth"]].tolist() == [ + 0.0, + 4.0, + 5.0, + 1.0, + 10.0, + ] diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_certification.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_certification.py index 72090682a..63d459ae5 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_certification.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_certification.py @@ -165,19 +165,52 @@ def test_compose_green_certification(green_certification_inputs): green_certification_inputs["certification_path"].read_text(encoding="utf-8") ) assert written["attestation"]["signature"] - # The certification's own signature verifies under the release key. + # The certification's own signature verifies under the release key, over + # the gate battery's canonical bytes (the form every signed UK report + # and the data contract's verifier share). import hmac as hmac_module - from microcosm.build.logbook import canonical_json_bytes + from microcosm.build.gate_battery import canonical_report_bytes unsigned = json.loads(json.dumps(written)) unsigned["attestation"]["signature"] = None recomputed = hmac_module.new( - base64.b64decode(_TEST_KEY), canonical_json_bytes(unsigned), hashlib.sha256 + base64.b64decode(_TEST_KEY), canonical_report_bytes(unsigned), hashlib.sha256 ).hexdigest() assert recomputed == written["attestation"]["signature"] +def test_compose_verifies_parts_the_battery_signed_over_floats( + green_certification_inputs, +): + """A real battery report carries floats such as 1.0 and 1e-05. + + The battery signs over its own canonical bytes, which render them as + Python does; the logbook's #628 form renders them as 1 and 0.00001, so + a verifier using it refused every real spine report while agreeing with + the float-free synthetic fixtures (the 3 October 2026 certifier run on + microcosm#1063's final build). The seam part gets such floats here and + must still compose. + """ + + seam_path = green_certification_inputs["seam_report_path"] + payload = json.loads(seam_path.read_text(encoding="utf-8")) + payload["aggregate_admin_measurement"] = { + "whole_ratio": 1.0, + "tiny_share": 1e-05, + "plain": 0.5, + } + _FIXTURES.resign_uk_gate_report(payload) + seam_path.write_text( + json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8" + ) + green_certification_inputs["build_record"]["artifacts"]["terminal_gate_json"][ + "sha256" + ] = _sha(seam_path) + certification = compose_uk_release_certification(**green_certification_inputs) + assert certification["attestation"]["signature"] + + def test_compose_refuses_tampered_part_signature(green_certification_inputs): seam_path = green_certification_inputs["seam_report_path"] payload = json.loads(seam_path.read_text(encoding="utf-8")) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_input_coverage.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_input_coverage.py index 9dd4cae4d..5d0ecbaac 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_release_input_coverage.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_release_input_coverage.py @@ -1,6 +1,13 @@ """Tests split from packages/microcosm-build/tests/test_uk_release_input_coverage.py.""" # ruff: noqa: F403, F405 +from importlib.resources import files + +from microcosm.build.uk_runtime.release_input_coverage import ( + UK_FAMILY_MASS_CHANGE_SEMANTICS, + _parse_family_coverage, + _valid_mass_record, +) from test_support.microcosm_build.uk_release_input_coverage import * @@ -127,3 +134,48 @@ def test_binding_hands_the_spine_frame_to_the_build_state_half(self) -> None: ) assert result.passed, result.failures assert result.details["family_build_state_frame"] == "spine" + + +class TestFamilyMassSemantics: + """Every family conserves household mass (microcosm#1063 item 3).""" + + def _payload(self) -> dict[str, dict[str, object]]: + resource = files("microcosm.build.uk").joinpath( + "release_input_coverage_manifest.json" + ) + return json.loads(resource.read_text(encoding="utf-8"))["family_coverage"] + + def test_parser_refuses_the_retired_mass_increasing_semantics(self) -> None: + payload = self._payload() + families = _parse_family_coverage(payload, resource="test") + assert {family["mass_change_semantics"] for family in families.values()} == { + UK_FAMILY_MASS_CHANGE_SEMANTICS + } + assert "spi_income_band_donors" in families + payload["spi_income_band_donors"] = { + **payload["spi_income_band_donors"], + "mass_change_semantics": "mass_increasing_support", + } + with pytest.raises(ValueError, match="the only semantics is 'mass_conserving'"): + _parse_family_coverage(payload, resource="test") + + def test_a_record_that_adds_mass_never_satisfies_a_family(self) -> None: + conserving = MassChangeRecord( + entity="household", + old_total=2_000.0, + new_total=2_000.0, + declared_factor=1.0, + reason="reviewed", + ) + assert _valid_mass_record(conserving) + # The pre-#1063 band-donor record: mass added, no declared factor. + assert not _valid_mass_record( + MassChangeRecord( + entity="household", + old_total=2_000.0, + new_total=2_450.0, + declared_factor=None, + reason="reviewed", + ) + ) + assert not _valid_mass_record({**conserving.__dict__, "entity": "person"}) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_salary_sacrifice.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_salary_sacrifice.py index 9747dbee0..8ce6171fa 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_salary_sacrifice.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_salary_sacrifice.py @@ -15,8 +15,10 @@ from microcosm.build.uk_runtime.national_frame import uk_national_frame from microcosm.build.uk_runtime.salary_sacrifice import ( SALSAC_OUTPUT, + SALSAC_PRE_CONVERSION_PAY_COLUMN, SALSAC_RATE_CAP, SALSAC_STAGE_TARGET, + UKSalarySacrificeStageTransform, _assert_salary_sacrifice_stage_parameters, impute_salary_sacrifice, load_salary_sacrifice_anchor, @@ -54,6 +56,7 @@ def _frame( clones=None, donors=None, weights=None, + employment_income=None, ): n = len(asked) ids = np.arange(1, n + 1, dtype="int64") @@ -66,7 +69,11 @@ def _frame( "person_benunit_id": ids, "person_household_id": ids, "age": np.linspace(25, 55, n), - "employment_income": np.full(n, 30_000.0), + "employment_income": ( + np.full(n, 30_000.0) + if employment_income is None + else np.asarray(employment_income, dtype=float) + ), "salary_sacrifice_asked": asked, SALSAC_OUTPUT: salary_sacrifice_values, "employee_pension_contributions": employee_pension, @@ -135,6 +142,44 @@ def test_qrf_preserves_asked_rows_and_excludes_nonbase_training_rows( assert result.prediction_rows == 1 +def test_qrf_predicts_only_for_people_not_asked_who_are_paid( + monkeypatch: pytest.MonkeyPatch, +) -> None: + # Given people not asked the question, with and without employment + # income, one of them carrying a stale amount. + _FakeQRF.training_frames = [] + _Fitted.value = 900.0 + monkeypatch.setattr(salary_sacrifice, "QRF", _FakeQRF) + frame = _frame( + asked=[1, 1, 0, 0, 0], + salary_sacrifice_values=[10.0, 20.0, 0.0, 0.0, 55.0], + employee_pension=[0.0] * 5, + employment_income=[30_000.0, 0.0, 25_000.0, 0.0, 0.0], + weights=np.asarray([1.0, 1.0, 1.0, 2.0, 3.0]), + ) + + result = impute_salary_sacrifice(frame) + + # Then the paid person not asked is predicted for, the unpaid ones are + # zero, and an asked person keeps what they reported whatever their pay. + assert result.frame.table("person")[SALSAC_OUTPUT].tolist() == [ + 10.0, + 20.0, + 900.0, + 0.0, + 0.0, + ] + assert result.prediction_rows == 1 + assert result.not_asked_without_pay_rows == 2 + assert result.not_asked_without_pay_mass == 5.0 + qrf = result.evidence()["qrf"] + assert qrf["not_asked_without_pay_rows"] == 2 + assert qrf["not_asked_without_pay_mass"] == 5.0 + assert "employment_income > 0" in qrf["target_population"] + # The headcount the conversion tops up starts from true users only. + assert result.pre_headcount == 3.0 + + def test_conversion_moves_full_pension_zeros_source_and_records_cap( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -166,9 +211,17 @@ def test_conversion_moves_full_pension_zeros_source_and_records_cap( 30_000.0 - person.loc[converted, SALSAC_OUTPUT], ) assert (person.loc[~converted, "employment_income"] == 30_000.0).all() + # The converted record's pay before the conversion stays on the carrier, + # zero for everyone else, so the rewrite reverses exactly (microcosm#1063). + assert (person.loc[converted, SALSAC_PRE_CONVERSION_PAY_COLUMN] == 30_000.0).all() + assert (person.loc[~converted, SALSAC_PRE_CONVERSION_PAY_COLUMN] == 0.0).all() + assert SALSAC_PRE_CONVERSION_PAY_COLUMN in ( + UKSalarySacrificeStageTransform.output_columns() + ) evidence = result.evidence()["headcount_receipt"] assert evidence["target"] == SALSAC_STAGE_TARGET assert evidence["converted_rows"] == result.converted_rows + assert evidence["pre_conversion_pay_column"] == SALSAC_PRE_CONVERSION_PAY_COLUMN def test_anchor_is_self_consistent() -> None: diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_source_stages.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_source_stages.py index f3da2e590..42b6c737a 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_source_stages.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_source_stages.py @@ -69,6 +69,9 @@ PENSION_CREDIT_TAKE_UP_STAGE_NAMES = [ "pension_credit_take_up", ] +CHILD_BENEFIT_TAKE_UP_STAGE_NAMES = [ + "child_benefit_take_up", +] E9_STAGE_NAMES = [ "uc_deduction_attributes", ] @@ -76,8 +79,11 @@ "cgt_support_split", "cgt_incidence_clone", "hmrc_cgt_gains_spine", - "hmrc_cgt_asset_type_spine", "cgt_incidence_anchor", + # microcosm#1063: the residential split carries the flag as weight and + # the asset-type stage types its arms, so both run after the anchor. + "cgt_residential_split", + "hmrc_cgt_asset_type_spine", "salary_sacrifice", "student_loans", ] @@ -95,6 +101,7 @@ *UC_REPORTER_REDRAW_STAGE_NAMES, *UC_COHERENCE_STAGE_NAMES, *PENSION_CREDIT_TAKE_UP_STAGE_NAMES, + *CHILD_BENEFIT_TAKE_UP_STAGE_NAMES, *E9_STAGE_NAMES, *E8_STAGE_NAMES, ] @@ -164,6 +171,7 @@ def test_e5_and_e6_follow_e7_and_precede_the_uc_rewrites(self) -> None: *UC_REPORTER_REDRAW_STAGE_NAMES, *UC_COHERENCE_STAGE_NAMES, *PENSION_CREDIT_TAKE_UP_STAGE_NAMES, + *CHILD_BENEFIT_TAKE_UP_STAGE_NAMES, *E9_STAGE_NAMES, ] @@ -236,12 +244,14 @@ def test_country_stage_plan_assembles_spine_plan(self) -> None: "uc_reporter_redraw": _identity, "uc_capital_coherence": _identity, "pension_credit_take_up": _identity, + "child_benefit_take_up": _identity, "uc_deduction_attributes": _identity, "cgt_support_split": _identity, "cgt_incidence_clone": _identity, "hmrc_cgt_gains_spine": _identity, - "hmrc_cgt_asset_type_spine": _identity, "cgt_incidence_anchor": _identity, + "cgt_residential_split": _identity, + "hmrc_cgt_asset_type_spine": _identity, "salary_sacrifice": _identity, "student_loans": _identity, "age_tail": _identity, @@ -448,7 +458,10 @@ def test_e8_outputs_and_rewrites_are_backed_by_runtime_constants(self) -> None: CGT_SUPPORT_COPY_INDEX_COLUMN, HOUSEHOLD_IS_CGT_SUPPORT_COPY, ) - from microcosm.build.uk_runtime.salary_sacrifice import SALSAC_OUTPUT + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_OUTPUT, + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) stages = load_country_spec("uk").sources.stage_map() @@ -469,6 +482,7 @@ def test_e8_outputs_and_rewrites_are_backed_by_runtime_constants(self) -> None: SALSAC_OUTPUT, "employee_pension_contributions", "employment_income", + SALSAC_PRE_CONVERSION_PAY_COLUMN, ) assert stages["student_loans"].outputs == ("student_loan_plan",) @@ -869,9 +883,15 @@ def test_every_e4_stochastic_operation_declares_integer_seed(self) -> None: }: assert isinstance(operation.parameters.get("seed"), int) - def test_e5_debt_segment_predictors_lockstep(self) -> None: + def test_e5_tenure_strata_and_derived_totals_lockstep(self) -> None: from microcosm.build.uk_runtime.was_wealth import ( - UK_WAS_DEBT_SEGMENT_PREDICTORS, + UK_WAS_DERIVED_TOTALS, + UK_WAS_DRAWN_ONLY_COLUMNS, + UK_WAS_INTERNAL_COMPONENT_COLUMNS, + UK_WAS_NET_FINANCIAL_LIABILITIES, + UK_WAS_STRATIFIED_TARGETS, + UK_WAS_TENURE_PREDICTORS, + UK_WAS_WEALTH_OUTPUT_COLUMNS, UK_WAS_WEALTH_PREDICTORS, ) @@ -880,15 +900,35 @@ def test_e5_debt_segment_predictors_lockstep(self) -> None: qrf = stages["was_wealth"].operations[2] assert qrf.kind == "fit_weighted_qrf_chain" - assert tuple(qrf.parameters["debt_segment_predictors"]) == ( - UK_WAS_DEBT_SEGMENT_PREDICTORS + assert tuple(qrf.parameters["predictors"]) == UK_WAS_WEALTH_PREDICTORS + # Tenure enters every segment as the four-way flags (microcosm#1063); + # the engine's is_renting, which omits housing-association renters, is + # no predictor of the stage. + assert set(UK_WAS_TENURE_PREDICTORS) <= set(qrf.parameters["predictors"]) + assert "is_renting" not in qrf.parameters["predictors"] + assert "debt_segment_predictors" not in qrf.parameters + assert { + target: tuple(categories) + for target, categories in qrf.parameters["stratified_targets"].items() + } == dict(UK_WAS_STRATIFIED_TARGETS) + declared = { + total: tuple(components) + for total, components in qrf.parameters["derived_totals"].items() + } + assert declared.pop("net_financial_wealth") == ( + "gross_financial_wealth", + *(f"-{column}" for column in UK_WAS_NET_FINANCIAL_LIABILITIES), ) - # The extra predictor belongs to the debt segment only: the shared - # base list, and so E5's first three segments, are unchanged. - assert not set(UK_WAS_DEBT_SEGMENT_PREDICTORS) & set( - qrf.parameters["predictors"] + assert declared == dict(UK_WAS_DERIVED_TOTALS) + assert tuple(qrf.parameters["internal_components"]) == ( + UK_WAS_INTERNAL_COMPONENT_COLUMNS + ) + # The chain order names every output and every drawn-only column (the + # internal components, the dropped share-like component and the + # main-residence mortgage) exactly once. + assert sorted(qrf.parameters["chain_order"]) == sorted( + {*UK_WAS_WEALTH_OUTPUT_COLUMNS, *UK_WAS_DRAWN_ONLY_COLUMNS} ) - assert tuple(qrf.parameters["predictors"]) == UK_WAS_WEALTH_PREDICTORS def test_e5_qrf_operation_declares_integer_seed(self) -> None: spec = load_country_spec("uk") diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_band_donors.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_band_donors.py index bfb8b0c58..4631e5e4c 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_band_donors.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_band_donors.py @@ -2,6 +2,8 @@ from __future__ import annotations +import importlib.util + import numpy as np import pandas as pd import pytest @@ -15,18 +17,26 @@ SPI_INCOME_BAND_DONOR_CLONE_INDEX, SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN, SPI_INCOME_BAND_DONOR_LOWER_BOUNDS, + SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON, SPI_INCOME_BAND_DONOR_SEED, - SPI_INCOME_BAND_DONORS_PER_BAND, + SPI_INCOME_BAND_DONOR_TAPE_SEED, + SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT, + SPI_INCOME_BAND_MINIMUM_DONORS, UKSPIIncomeBandDonorStageTransform, _assert_band_donor_stage_parameters, income_band_lower_bound, load_hmrc_itl_band_taxpayers, + spi_income_band_donor_operation_parameters, + spi_income_band_donor_plan, + spi_income_band_donor_propensity, + spi_income_band_donor_seats, spi_income_band_propensity, stack_spi_income_band_donors, ) from microcosm.build.uk_runtime.spi_income import ( SPI_DONOR_REQUIRED_COLUMNS, SPI_INCOME_QRF_OUTPUT_COLUMNS, + SPIDonorAgeModel, _resample_band_donor_leaves, prepare_spi_donor_table, ) @@ -37,6 +47,13 @@ support_channel_column, ) from microcosm.frame import WeightKind +from test_support.paths import paths_for + +_IDENTITY_TOOL_PATH = ( + paths_for("microcosm-build").repository + / "tools" + / "verify_uk_identity_stability.py" +) BANDS = SPI_INCOME_BAND_DONOR_LOWER_BOUNDS TAXPAYERS = { @@ -45,6 +62,55 @@ 1_000_000: 20_000.0, 2_000_000: 10_000.0, } +#: Band masses a 200-household synthetic frame can fund at full scale. +SMALL_TAXPAYERS = { + 200_000: 40.0, + 500_000: 12.0, + 1_000_000: 6.0, + 2_000_000: 3.0, +} + + +def _age_model(state_pension_age: int = 66) -> SPIDonorAgeModel: + """A flat single-year population, so the engine-free lane needs no engine.""" + + return SPIDonorAgeModel( + populations={ + (sex, age): 100.0 for sex in ("MALE", "FEMALE") for age in range(0, 91) + }, + state_pension_age=state_pension_age, + source="fixture", + ) + + +def _propensity() -> pd.DataFrame: + donor = prepare_spi_donor_table(_raw_tape(), seed=1) + return spi_income_band_propensity(donor, lower_bounds=BANDS) + + +def _stack_small(frame, **overrides): + arguments = { + "propensity": _propensity(), + "band_taxpayers": SMALL_TAXPAYERS, + "minimum_donors": 3, + "maximum_donor_weight": 10.0, + "seed": SPI_INCOME_BAND_DONOR_SEED, + "taxpayer_period": 2024, + } + arguments.update(overrides) + return stack_spi_income_band_donors(frame, **arguments) + + +def _donor_source_ids(result) -> dict[int, float]: + household = result.frame.table("household") + donors = household.loc[household[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR]] + return dict( + zip( + donors["household_source_id"].astype(int), + donors[SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN], + strict=True, + ) + ) def _raw_tape(rows: int = 240, *, seed: int = 0) -> pd.DataFrame: @@ -153,22 +219,61 @@ def test_propensity_table_is_fact_weighted_band_share_per_cell() -> None: spi_income_band_propensity(thin, lower_bounds=BANDS) -def test_stack_reserves_band_exact_positive_rows_with_one_carrier_each() -> None: - donor = prepare_spi_donor_table(_raw_tape(), seed=1) - propensity = spi_income_band_propensity(donor, lower_bounds=BANDS) - frame = _support_frame() - result = stack_spi_income_band_donors( - frame, - propensity=propensity, - band_taxpayers=TAXPAYERS, - donors_per_band=5, - seed=SPI_INCOME_BAND_DONOR_SEED, - taxpayer_period=2024, +def test_plan_keeps_every_donor_at_or_below_the_maximum_weight() -> None: + # Given the 2024-25 Table 2.5 taxpayers, when the plan is read, then a + # band seats the minimum unless that would put a donor above the cap. + plan = spi_income_band_donor_plan(TAXPAYERS) + assert {band: count for band, (count, _) in plan.items()} == { + 200_000: 1_436, + 500_000: 244, + 1_000_000: SPI_INCOME_BAND_MINIMUM_DONORS, + 2_000_000: SPI_INCOME_BAND_MINIMUM_DONORS, + } + for band, (count, weight) in plan.items(): + assert weight <= SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT + assert weight * count == pytest.approx(TAXPAYERS[band], abs=1e-6) + with pytest.raises(ValueError, match="positive published taxpayers"): + spi_income_band_donor_plan({**TAXPAYERS, 2_000_000: 0.0}) + with pytest.raises(ValueError, match="minimum_donors"): + spi_income_band_donor_plan(TAXPAYERS, minimum_donors=0) + with pytest.raises(ValueError, match="maximum_donor_weight"): + spi_income_band_donor_plan(TAXPAYERS, maximum_donor_weight=0.0) + + +def test_seats_scale_only_when_the_plan_outgrows_the_frame() -> None: + plan = spi_income_band_donor_plan(TAXPAYERS) + expected = sum(count for count, _ in plan.values()) + # Given a frame the plan fits in, the scale is one and the seats the plan's. + scale, seats = spi_income_band_donor_seats( + plan, eligible_households=40_000, maximum_donor_household_share=0.15 ) + assert scale == 1.0 + assert seats == {band: count for band, (count, _) in plan.items()} + # Given a small frame, every band is scaled alike and none is dropped. + scale, seats = spi_income_band_donor_seats( + plan, eligible_households=60, maximum_donor_household_share=0.15 + ) + assert scale == pytest.approx(0.15 * 60 / expected) + assert seats == {200_000: 6, 500_000: 1, 1_000_000: 1, 2_000_000: 1} + with pytest.raises(ValueError, match="no eligible base household"): + spi_income_band_donor_seats(plan, eligible_households=0) + + +def test_stack_reserves_band_rows_and_conserves_every_region_mass() -> None: + frame = _support_frame(200) + result = _stack_small(frame) household = result.frame.table("household") person = result.frame.table("person") donors = household.loc[household[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR]] - assert len(donors) == 5 * len(BANDS) + # 40 / 10 = 4 donors in the first band; the others seat the minimum. + assert donors[ + SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN + ].value_counts().to_dict() == { + 200_000.0: 4, + 500_000.0: 3, + 1_000_000.0: 3, + 2_000_000.0: 3, + } assert donors[HOUSEHOLD_IS_SPI_SYNTHETIC_COLUMN].all() assert ( donors[support_channel_column("household")] == SPI_SYNTHETIC_SUPPORT_CHANNEL @@ -176,9 +281,7 @@ def test_stack_reserves_band_exact_positive_rows_with_one_carrier_each() -> None assert ( donors["household_support_clone_index"] == SPI_INCOME_BAND_DONOR_CLONE_INDEX ).all() - assert sorted(donors[SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN].unique()) == list( - BANDS - ) + assert donors["household_source_id"].is_unique assert ( household.loc[ ~household[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR], @@ -190,73 +293,136 @@ def test_stack_reserves_band_exact_positive_rows_with_one_carrier_each() -> None assert len(carriers) == len(donors) assert set(carriers["person_household_id"]) == set(donors["household_id"]) assert (carriers["age"] >= 16).all() - weights = result.frame.weights_for("household").values[-len(donors) :] - assert (weights > 0.0).all() + + # Equal weights within a band, carrying the band's published taxpayers. + weights = pd.Series( + result.frame.weights_for("household").values, index=household.index + ) by_band = {int(row["lower_bound"]): row for row in result.band_rows} for band in BANDS: - assert by_band[band]["donor_households"] == 5 - assert by_band[band]["carriers"] == 5 - assert by_band[band]["donor_weight"] == pytest.approx(TAXPAYERS[band] / 5) - assert by_band[band]["weighted_taxpayers"] == pytest.approx(TAXPAYERS[band]) - assert result.frame.mass_log[-1].new_total == pytest.approx( - frame.weights_for("household").total + sum(TAXPAYERS.values()) - ) - assert result.frame.mass_log[-1].declared_factor is None - # The base and ordinary synthetic rows keep their weights. - assert np.allclose( - result.frame.weights_for("household").values[: len(frame.table("household"))], - frame.weights_for("household").values, - ) - + band_weights = weights[ + donors.index[donors[SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN] == band] + ] + assert band_weights.nunique() == 1 + assert band_weights.iloc[0] <= 10.0 + assert band_weights.sum() == pytest.approx(SMALL_TAXPAYERS[band]) + assert by_band[band]["weighted_taxpayers"] == pytest.approx( + SMALL_TAXPAYERS[band] + ) + assert by_band[band]["donor_households"] == by_band[band]["carriers"] + assert ( + by_band[band]["donor_households"] + == by_band[band]["expected_donor_households"] + ) -def test_stack_is_permutation_stable_and_refuses_a_second_stack() -> None: - donor = prepare_spi_donor_table(_raw_tape(), seed=1) - propensity = spi_income_band_propensity(donor, lower_bounds=BANDS) - first = stack_spi_income_band_donors( - _support_frame(), - propensity=propensity, - band_taxpayers=TAXPAYERS, - donors_per_band=3, - seed=3, - taxpayer_period=2024, + # The mass is reallocated, not created: the total is bit-equal, every + # region keeps its mass, and the receipt declares factor one. + incoming = frame.weights_for("household") + assert float(weights.sum()) == float(incoming.total) + before = ( + pd.Series(incoming.values) + .groupby(frame.table("household")["region"].to_numpy()) + .sum() ) - second = stack_spi_income_band_donors( - _support_frame(reverse=True), - propensity=propensity, - band_taxpayers=TAXPAYERS, - donors_per_band=3, - seed=3, - taxpayer_period=2024, + after = weights.groupby(household["region"].to_numpy()).sum() + assert after.to_dict() == pytest.approx(before.to_dict(), rel=1e-12) + receipt = result.frame.mass_log[-1] + assert receipt.declared_factor == 1.0 + assert receipt.old_total == receipt.new_total == float(incoming.total) + assert receipt.reason == SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON + evidence = result.evidence() + assert evidence["seating_scale"] == 1.0 and evidence["mass_scale"] == 1.0 + assert evidence["reallocated_mass"] == pytest.approx(sum(SMALL_TAXPAYERS.values())) + assert evidence["mass"]["old_total"] == evidence["mass"]["new_total"] + + # Every incumbent of a region is scaled by that region's one factor, and + # the zero-weight support copies stay at zero. + incumbents = len(frame.table("household")) + ratio = pd.Series(weights.to_numpy()[:incumbents] / np.asarray(incoming.values)) + regions = pd.Series(frame.table("household")["region"].to_numpy()) + positive = np.asarray(incoming.values) > 0.0 + factors = {row["stratum"]: row["factor"] for row in evidence["funding"]} + for region, factor in factors.items(): + assert 0.5 <= factor < 1.0 + rows = (regions == region).to_numpy() & positive + assert ratio[rows].to_numpy() == pytest.approx(factor, rel=1e-9) + assert (weights.to_numpy()[:incumbents][~positive] == 0.0).all() + + +def test_stack_scales_seats_then_mass_on_a_frame_too_small_for_the_plan() -> None: + # Given the published masses on a 60-household frame, the plan's 1,920 + # donors do not fit: the seats shrink to the declared share of the frame. + frame = _support_frame() + result = stack_spi_income_band_donors( + frame, propensity=_propensity(), band_taxpayers=TAXPAYERS, taxpayer_period=2024 ) - pick = lambda result: set( # noqa: E731 - result.frame.table("household").loc[ - lambda t: t[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR], "household_id" - ] + evidence = result.evidence() + assert evidence["expected_donor_count"] == 1_920 + assert evidence["donor_count"] == 9 + assert evidence["seating_scale"] == pytest.approx(0.15 * 60 / 1_920) + assert evidence["mass_scale"] == 1.0 + assert float(result.frame.weights_for("household").total) == float( + frame.weights_for("household").total ) - assert pick(first) == pick(second) - with pytest.raises(ValueError, match="already stacked"): - stack_spi_income_band_donors( - first.frame, - propensity=propensity, - band_taxpayers=TAXPAYERS, - donors_per_band=3, - seed=3, - taxpayer_period=2024, + # When the seated mass would take more than the declared share of a + # region, every band's weight is scaled alike and the total still holds. + heavy = _stack_small( + _support_frame(), + band_taxpayers={band: value * 400.0 for band, value in SMALL_TAXPAYERS.items()}, + maximum_donor_weight=4_000.0, + ) + evidence = heavy.evidence() + assert 0.0 < evidence["mass_scale"] < 1.0 + assert min(row["factor"] for row in evidence["funding"]) == pytest.approx(0.5) + for row in evidence["bands"]: + assert row["donor_weight"] == pytest.approx( + row["planned_donor_weight"] * evidence["mass_scale"] ) + assert evidence["mass"]["old_total"] == evidence["mass"]["new_total"] -def test_stack_refuses_zero_band_weight_and_missing_support_channel() -> None: - donor = prepare_spi_donor_table(_raw_tape(), seed=1) - propensity = spi_income_band_propensity(donor, lower_bounds=BANDS) - with pytest.raises(ValueError, match="positive initial weight"): - stack_spi_income_band_donors( - _support_frame(), - propensity=propensity, - band_taxpayers={**TAXPAYERS, 2_000_000: 0.0}, - donors_per_band=3, - seed=3, - taxpayer_period=2024, +def test_stack_is_keyed_on_identity_and_refuses_a_second_stack() -> None: + first = _stack_small(_support_frame(200)) + # The same households are seated in the same bands whatever the row order. + second = _stack_small(_support_frame(200, reverse=True)) + assert _donor_source_ids(first) == _donor_source_ids(second) + # A different seed seats different households. + other = _stack_small(_support_frame(200), seed=SPI_INCOME_BAND_DONOR_SEED + 1) + assert set(_donor_source_ids(other)) != set(_donor_source_ids(first)) + with pytest.raises(ValueError, match="already stacked"): + _stack_small(first.frame) + + +def test_stack_follows_weight_times_propensity_not_the_sample() -> None: + # Given a tape whose band propensities are equal across the two regions, + # seats follow household weight: the heavier half of the frame is seated + # more often than its half share of the rows across many seeds. + frame = _support_frame(200) + weight = pd.Series( + frame.weights_for("household").values, + index=frame.table("household")["household_id"].to_numpy(), + ) + heavy = weight > weight[weight > 0.0].median() + heavy_seats = seats = 0 + propensity = _propensity() + for seed in range(40): + result = _stack_small(frame, propensity=propensity, seed=seed) + ids = list(_donor_source_ids(result)) + heavy_seats += int(heavy.loc[ids].sum()) + seats += len(ids) + assert heavy_seats / seats > 0.53 + + +def test_stack_refuses_bad_inputs_and_a_missing_support_channel() -> None: + propensity = _propensity() + with pytest.raises(ValueError, match="positive published taxpayers"): + _stack_small( + _support_frame(), band_taxpayers={**SMALL_TAXPAYERS, 2_000_000: 0.0} ) + with pytest.raises(ValueError, match="maximum_reallocation_share"): + _stack_small(_support_frame(), maximum_reallocation_share=1.0) + with pytest.raises(ValueError, match="maximum_donor_household_share"): + _stack_small(_support_frame(), maximum_donor_household_share=0.0) raw = uk_national_frame( person=pd.DataFrame( { @@ -276,8 +442,8 @@ def test_stack_refuses_zero_band_weight_and_missing_support_channel() -> None: stack_spi_income_band_donors( raw, propensity=propensity, - band_taxpayers=TAXPAYERS, - donors_per_band=1, + band_taxpayers=SMALL_TAXPAYERS, + minimum_donors=1, seed=3, taxpayer_period=2024, ) @@ -402,6 +568,61 @@ def test_resample_cuts_the_pools_on_the_total_income_a_record_realises() -> None assert receipt["pool_basis"].startswith("uprated total income") +def test_resample_draw_is_keyed_on_the_carrier_not_on_the_row_order() -> None: + # Given twelve carriers in one band, when the person rows are reversed or + # a carrier is removed, then every remaining carrier draws the same record. + donor = prepare_spi_donor_table(_raw_tape(), seed=1) + # The synthetic tape's records are identical within a band; a small + # per-record amount tells them apart without moving any out of its band. + donor["dividend_income"] = donor["dividend_income"] + np.arange(len(donor)) + ids = np.arange(1, 13) + household = pd.DataFrame( + { + "household_id": ids, + "region": "LONDON", + SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN: 200_000.0, + } + ) + person = pd.DataFrame( + { + "person_id": ids * 10, + "person_source_id": ids, + "person_household_id": ids, + PERSON_IS_SPI_INCOME_BAND_CARRIER: True, + } + ) + for column in SPI_INCOME_QRF_OUTPUT_COLUMNS: + person[column] = 0.0 + arguments = { + "household": household, + "uprating_factors": dict.fromkeys(SPI_INCOME_QRF_OUTPUT_COLUMNS, 1.0), + "lower_bounds": BANDS, + "regional_pool_minimum": 5, + "seed": 44, + } + + def draw(frame: pd.DataFrame) -> pd.Series: + updated, receipt = _resample_band_donor_leaves( + frame, + donor, + spi_people=pd.Series(True, index=frame.index), + **arguments, + ) + assert receipt["draw_key"] == "person_source_id" + assert receipt["salt"] == "spi_income_band_donor_leaf_draw" + return updated.set_index("person_source_id")["dividend_income"] + + first = draw(person) + assert first.nunique() > 1 + reversed_rows = draw(person.iloc[::-1].reset_index(drop=True)) + assert reversed_rows.sort_index().equals(first.sort_index()) + fewer = draw(person.iloc[3:].reset_index(drop=True)) + assert fewer.sort_index().equals(first.sort_index().iloc[3:]) + # The key is the lineage id, not the frame's person id. + renumbered = draw(person.assign(person_id=person["person_id"] + 7)) + assert renumbered.sort_index().equals(first.sort_index()) + + def test_resample_refuses_missing_carriers_or_undeclared_band() -> None: donor = prepare_spi_donor_table(_raw_tape(), seed=1) household = pd.DataFrame( @@ -448,27 +669,74 @@ def test_resample_refuses_missing_carriers_or_undeclared_band() -> None: def test_packaged_stage_binds_to_the_reviewed_constants_and_refuses_drift() -> None: spec = load_country_spec("uk") stage = spec.sources.stage_map()["spi_income_band_donors"] - assert _assert_band_donor_stage_parameters( - stage, seed=SPI_INCOME_BAND_DONOR_SEED - ) == (SPI_INCOME_BAND_DONORS_PER_BAND) - drifted = SourceStageSpec.from_mapping( - { - **stage.__dict__, - "operations": [ - { - **dict(stage.operations[0].parameters), - "kind": stage.operations[0].kind, - "donors_per_band": 119, - } - ], - } + _assert_band_donor_stage_parameters(stage, seed=SPI_INCOME_BAND_DONOR_SEED) + declared = dict(stage.operations[0].parameters) + assert ( + declared + == spi_income_band_donor_operation_parameters()[ + "stack_income_band_donor_households" + ] ) - with pytest.raises(ValueError, match="donors_per_band"): - _assert_band_donor_stage_parameters(drifted, seed=SPI_INCOME_BAND_DONOR_SEED) + assert declared["declared_factor"] == 1.0 + assert declared["conservation"] == "exact_total" + assert declared["reason"] == SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON + for key, value in ( + ("minimum_donors_per_band", 119), + ("maximum_donor_weight", 500.0), + ("funding_strata", ["region", "tenure_type"]), + ("declared_factor", 1.01), + ): + drifted = SourceStageSpec.from_mapping( + { + **stage.__dict__, + "operations": [ + {**declared, "kind": stage.operations[0].kind, key: value} + ], + } + ) + with pytest.raises(ValueError, match=key): + _assert_band_donor_stage_parameters( + drifted, seed=SPI_INCOME_BAND_DONOR_SEED + ) with pytest.raises(ValueError, match=r"parameter\(s\) \['seed'\]"): _assert_band_donor_stage_parameters(stage, seed=SPI_INCOME_BAND_DONOR_SEED + 1) +def test_tape_is_prepared_as_the_income_stage_prepares_it() -> None: + # The declared tape seed is the income stage's stage-1 seed, and that + # stage declares the age draw the donor stage shares. + stages = load_country_spec("uk").sources.stage_map() + income = {op.kind: op for op in stages["hmrc_spi_income_spine"].operations} + assert ( + income["fit_weighted_qrf_stage1"].parameters["seed"] + == SPI_INCOME_BAND_DONOR_TAPE_SEED + ) + assert "draw_spi_donor_ages_by_population" in income + declared = stages["spi_income_band_donors"].operations[0].parameters + assert declared["tape_preparation_seed"] == SPI_INCOME_BAND_DONOR_TAPE_SEED + + # The stage's propensity table is the band shares of the tape prepared + # with that seed and the age model: a State Pension recipient the tape + # publishes no age band for (a composite) sits at or over State Pension + # age, never in a working-age band. + raw = _raw_tape() + composite = raw["AGERANGE"] == -1.0 + raw.loc[composite, "SRP"] = 9_000.0 + raw.loc[composite, ["TEI", "TI"]] += 9_000.0 + model = _age_model() + table = spi_income_band_donor_propensity(raw, age_model=model) + prepared = prepare_spi_donor_table( + raw, seed=SPI_INCOME_BAND_DONOR_TAPE_SEED, age_model=model + ) + pd.testing.assert_frame_equal( + table, spi_income_band_propensity(prepared, lower_bounds=BANDS) + ) + assert (prepared.loc[composite.to_numpy(), "age"] >= 66).all() + # The synthetic tape's non-composite rows are all aged 45 to 54, so the + # composites are the only mass in the 65-to-74 band. + assert set(table["age_band"]) == {45, 65} + + def test_vendored_table_2_5_counts_feed_the_build_year_weights() -> None: counts = load_hmrc_itl_band_taxpayers(2024) assert counts == { @@ -488,17 +756,31 @@ def test_stage_transform_runs_on_a_synthetic_tape_and_receipts() -> None: "put2223uk.tab", stage=stage, seed=SPI_INCOME_BAND_DONOR_SEED, - sample_fraction=4 / SPI_INCOME_BAND_DONORS_PER_BAND, + sample_fraction=0.01, donor_table=_raw_tape(), band_taxpayers=TAXPAYERS, + age_model=_age_model(), ) - result = transform(_support_frame()) + frame = _support_frame() + result = transform(frame) assert result.weights_for("household").kind is WeightKind.IMPORTANCE + assert float(result.weights_for("household").total) == float( + frame.weights_for("household").total + ) evidence = transform.checkpoint_metadata()["evidence"] assert evidence["stage"] == "spi_income_band_donors" - assert evidence["donors_per_band"] == 4 + assert evidence["minimum_donors_per_band"] == SPI_INCOME_BAND_MINIMUM_DONORS + assert evidence["maximum_donor_weight"] == SPI_INCOME_BAND_MAXIMUM_DONOR_WEIGHT + # The survey-side sample fraction is provenance only: the seats follow + # the frame, not the fraction. + assert evidence["sample_fraction"] == 0.01 + assert evidence["donor_count"] == 9 and evidence["seating_scale"] < 1.0 assert [row["lower_bound"] for row in evidence["bands"]] == list(BANDS) - assert sum(evidence["carrier_region_counts"].values()) == 4 * len(BANDS) + assert sum(evidence["carrier_region_counts"].values()) == 9 + assert evidence["draw"] == { + "seed": SPI_INCOME_BAND_DONOR_SEED, + "salt": "spi_income_band_donor_draw", + } assert transform.output_columns() == ( HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR, SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN, @@ -524,3 +806,116 @@ def test_resample_receipts_a_frame_without_the_donor_stage() -> None: ) assert receipt["carriers"] == 0 and "did not run" in receipt["skipped"] assert updated.equals(person) + + +def _identity_tool(): + spec = importlib.util.spec_from_file_location( + "verify_uk_identity_stability", _IDENTITY_TOOL_PATH + ) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _reweighted(frame, weights): + return uk_national_frame( + person=frame.table("person"), + benunit=frame.table("benunit"), + household=frame.table("household"), + time_period="2024", + weight_kind=WeightKind.IMPORTANCE, + household_weights=weights, + mass_log=frame.mass_log, + ) + + +def test_e8_receipt_reconstructs_the_seating_from_ids() -> None: + # Given a stacked frame, when the E8 block reruns the stage on the + # pre-donor frame it reconstructs from the artifact, then the seats, the + # carriers and the band weights are the stored ones in any row order. + tool = _identity_tool() + propensity = _propensity() + stacked = stack_spi_income_band_donors( + _support_frame(200), + propensity=propensity, + band_taxpayers=TAXPAYERS, + taxpayer_period=2024, + ).frame + problems: dict[str, object] = {} + receipt = tool._e8_band_donors( + stacked, + problems, + propensity=propensity, + band_taxpayers=TAXPAYERS, + permutation_seed=123, + ) + assert problems == {} + assert receipt["seating_recomputed"] is True + assert receipt["donor_households"] == 27 + assert 0.0 < receipt["seating_scale"] < 1.0 + assert receipt["mass"]["declared_factor"] == 1.0 + assert [row["lower_bound"] for row in receipt["bands"]] == list(BANDS) + assert set(receipt["funding"]) == {"LONDON", "SOUTH_EAST"} + + # Without the tape's propensities the layer is checked for structure and + # mass only, and the receipt says so. + problems = {} + receipt = tool._e8_band_donors( + stacked, problems, propensity=None, band_taxpayers=None, permutation_seed=123 + ) + assert problems == {} and receipt["seating_recomputed"] is False + assert "pass --spi-tab" in receipt["seating_scope"] + + +def test_e8_receipt_reports_a_seating_the_ids_do_not_reproduce() -> None: + tool = _identity_tool() + propensity = _propensity() + frame = _support_frame(200) + stacked = stack_spi_income_band_donors( + frame, propensity=propensity, band_taxpayers=TAXPAYERS, taxpayer_period=2024 + ).frame + arguments = { + "propensity": propensity, + "band_taxpayers": TAXPAYERS, + "permutation_seed": 123, + } + # A seating drawn under another seed is not the one the ids reproduce. + other = stack_spi_income_band_donors( + frame, + propensity=propensity, + band_taxpayers=TAXPAYERS, + seed=SPI_INCOME_BAND_DONOR_SEED + 1, + taxpayer_period=2024, + ).frame + problems: dict[str, object] = {} + tool._e8_band_donors(other, problems, **arguments) + assert "band_donor_seats_stored" in problems + assert "band_donor_carriers_stored" in problems + + # Unequal weights within a band are a stored-layer defect. + household = stacked.table("household") + weights = np.asarray(stacked.weights_for("household").values, dtype=float).copy() + first_donor = int(np.flatnonzero(household[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR])[0]) + weights[first_donor] *= 1.5 + problems = {} + tool._e8_band_donors(_reweighted(stacked, weights), problems, **arguments) + assert problems["band_donor_layer"] == {"bands_with_unequal_donor_weights": 1} + + # A frame without the stage's mass record, or without the layer, refuses. + problems = {} + tool._e8_band_donors( + uk_national_frame( + person=stacked.table("person"), + benunit=stacked.table("benunit"), + household=household, + time_period="2024", + weight_kind=WeightKind.IMPORTANCE, + household_weights=stacked.weights_for("household").values, + ), + problems, + **arguments, + ) + assert problems["band_donor_mass_record"] == "missing" + with pytest.raises(ValueError, match="donor layer is absent"): + tool._e8_band_donors(frame, {}, **arguments) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_pension_age.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_pension_age.py index e940dbe94..d2952d3d8 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_pension_age.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_spi_pension_age.py @@ -189,6 +189,7 @@ def _band_resample_inputs(carrier_age: float): donor.loc[30:, spi_income.SPI_HMRC_STATE_PENSION_INCOME_COLUMN] = 10_000.0 person = pd.DataFrame( { + "person_id": [1], "person_household_id": [1], "age": [carrier_age], spi_income.SPI_INCOME_BAND_CARRIER_COLUMN: [True], diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_spine_acceptance_receipt.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_spine_acceptance_receipt.py index 8b01ce745..7e9ac477d 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_spine_acceptance_receipt.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_spine_acceptance_receipt.py @@ -111,6 +111,19 @@ def _apply_pending_roster_transformations( # the post-SPI incomes, so it runs right after UC capital coherence. assert "pension_credit_take_up" not in roster roster.insert(roster.index("uc_capital_coherence") + 1, "pension_credit_take_up") + # microcosm#1063 re-mint pending: the Child Benefit redraw reads the same + # post-SPI incomes, right after the Pension Credit redraw. + assert "child_benefit_take_up" not in roster + roster.insert(roster.index("pension_credit_take_up") + 1, "child_benefit_take_up") + # microcosm#1063 re-mint pending: the residential split carries the flag + # as weight after the anchor, and the asset-type stage types its arms. + assert "cgt_residential_split" not in roster + roster.remove("hmrc_cgt_asset_type_spine") + anchor = roster.index("cgt_incidence_anchor") + roster[anchor + 1 : anchor + 1] = [ + "cgt_residential_split", + "hmrc_cgt_asset_type_spine", + ] return tuple(roster) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_stage_health.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_stage_health.py index d779c64de..5c2256852 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_stage_health.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_stage_health.py @@ -697,6 +697,90 @@ def test_cgt_summary_minimum_rows_parameter_is_live() -> None: ).passed +def test_support_clip_gate_holds_the_declared_donor_floor() -> None: + """microcosm#1063 c9: where the gate declares a donor floor, the receipt + must show it applied, no negative donor row left and no clip range below it.""" + + def evidence(**floor_changes): + return { + "stage": "lcfs_consumption", + "support_clip": { + "columns": { + "housing_water_and_electricity_consumption": { + "donor_min": 0.0, + "donor_max": 9_000.0, + "clipped_low_rows": 0, + "clipped_high_rows": 0, + "rows_considered": 2, + } + } + }, + "donor_floor": { + "floor": 0.0, + "rows_raised": 250, + "remaining_negative_rows": 0, + "columns": { + "housing_water_and_electricity_consumption": { + "rows_raised": 250, + "negative_mass": -1.0e6, + "minimum_before": -14_165.0, + } + }, + **floor_changes, + }, + } + + parameters = { + "stage": "lcfs_consumption", + "check": "support_clip", + "columns": ["housing_water_and_electricity_consumption"], + "max_clipped_low_rows_by_column": { + "housing_water_and_electricity_consumption": 0 + }, + "max_clipped_high_rows_by_column": { + "housing_water_and_electricity_consumption": 0 + }, + "donor_floor": 0.0, + } + + def gate(evidence_payload, **parameter_changes): + return uk_stage_health_gate( + evidence=evidence_payload, + stage="lcfs_consumption", + check="support_clip", + parameters={**parameters, **parameter_changes}, + ) + + passed = gate(evidence()) + assert _passed(passed) + assert passed.details["donor_floor_rows_raised"] == 250 + assert any( + "donor_floor receipt" in f + for f in gate({**evidence(), "donor_floor": None}).failures + ) + assert any( + "negative consumption row" in f + for f in gate(evidence(remaining_negative_rows=3)).failures + ) + assert any("not the declared" in f for f in gate(evidence(floor=-1.0)).failures) + below = evidence() + below["support_clip"]["columns"]["housing_water_and_electricity_consumption"][ + "donor_min" + ] = -14_165.0 + assert any("below the donor floor" in f for f in gate(below).failures) + # Without a declared floor (the WAS clip gate) the receipt is not required. + undeclared = {**parameters} + del undeclared["donor_floor"] + assert _passed( + uk_stage_health_gate( + evidence={**evidence(), "donor_floor": None}, + stage="lcfs_consumption", + check="support_clip", + parameters=undeclared, + ) + ) + + def test_support_clip_gate_fails_closed_on_a_missing_allowance() -> None: """An undeclared allowance skipped the comparison entirely, so a stage clipping every row passed a release-blocking gate — the green-by-absence @@ -735,6 +819,100 @@ class the #787 review named. A non-exempt column now needs both bounds assert any("no clipped_high_rows allowance" in f for f in result.failures) +def _wealth_coherence_evidence() -> dict: + return { + "stage": "was_wealth", + "tenure_coherence": { + "main_residence_mortgage_off_mortgaged_tenure_rows": 0, + "main_residence_value_off_owner_tenure_rows": 0, + "owner_share_without_main_residence_value": 0.001, + "donor_owner_share_without_main_residence_value": 0.0, + }, + "identities": { + "property_wealth_violation_rows": 0, + "corporate_wealth_violation_rows": 0, + "gross_financial_wealth_violation_rows": 0, + "net_financial_wealth_violation_rows": 0, + "mortgage_debt_violation_rows": 0, + "property_wealth_capped_rows": 3, + }, + } + + +_WEALTH_COHERENCE_PARAMETERS = { + "stage": "was_wealth", + "check": "wealth_coherence", + "maximum_owner_share_without_main_residence_excess": 0.005, +} + + +def _wealth_coherence(evidence: dict): + return uk_stage_health_gate( + evidence=evidence, + stage="was_wealth", + check="wealth_coherence", + parameters=_WEALTH_COHERENCE_PARAMETERS, + ) + + +def test_wealth_coherence_gate_passes_a_coherent_receipt() -> None: + result = _wealth_coherence(_wealth_coherence_evidence()) + + assert _passed(result) + assert result.details["main_residence_mortgage_off_mortgaged_tenure_rows"] == 0 + # The donor-range cap is recorded, never a failure. + assert result.details["capped_rows"] == {"property_wealth_capped_rows": 3} + + +@pytest.mark.parametrize( + ("block", "key"), + [ + ("tenure_coherence", "main_residence_mortgage_off_mortgaged_tenure_rows"), + ("tenure_coherence", "main_residence_value_off_owner_tenure_rows"), + ("identities", "property_wealth_violation_rows"), + ("identities", "corporate_wealth_violation_rows"), + ("identities", "gross_financial_wealth_violation_rows"), + ("identities", "net_financial_wealth_violation_rows"), + ("identities", "mortgage_debt_violation_rows"), + ], +) +def test_wealth_coherence_gate_requires_every_count_and_requires_it_zero( + block: str, key: str +) -> None: + evidence = _wealth_coherence_evidence() + evidence[block][key] = 2 + failed = _wealth_coherence(evidence) + assert not _passed(failed) + assert f"{key} is 2, expected 0" in failed.failures[0] + + del evidence[block][key] + missing = _wealth_coherence(evidence) + assert not _passed(missing) + assert f"receipt is missing {key}" in missing.failures[0] + + +def test_wealth_coherence_gate_bounds_owners_without_a_main_residence() -> None: + evidence = _wealth_coherence_evidence() + evidence["tenure_coherence"]["owner_share_without_main_residence_value"] = 0.084 + failed = _wealth_coherence(evidence) + assert not _passed(failed) + assert "no main-residence value" in failed.failures[0] + + # The bound is on the excess over the donor's own share. + evidence["tenure_coherence"]["donor_owner_share_without_main_residence_value"] = ( + 0.08 + ) + assert _passed(_wealth_coherence(evidence)) + + +def test_wealth_coherence_gate_fails_closed_without_its_receipt_blocks() -> None: + result = _wealth_coherence({"stage": "was_wealth", "support_clip": {}}) + + assert not _passed(result) + assert any("missing tenure_coherence" in f for f in result.failures) + assert any("missing identities" in f for f in result.failures) + + def _latent_receipt() -> dict: row = {"target": 0.5, "realized": 0.51, "tolerance": 0.05, "rows": 1000} return { @@ -808,20 +986,43 @@ def test_latent_attribute_realization_fails_on_empty_blocks_and_zero_rows() -> N assert not _latent_gate(zero_rows).passed +def _residential_bands() -> list[dict[str, object]]: + """Two gain bands whose achieved masses sum to the residential receipt's.""" + + return [ + { + "gain_lower_bound": 0.0, + "gain_upper_bound": 250_000.0, + "liable_rows": 2_900, + "achieved_rows": 2_600, + "achieved_count": 190_000.0, + "achieved_gains": 8.0e9, + }, + { + "gain_lower_bound": 250_000.0, + "gain_upper_bound": None, + "liable_rows": 900, + "achieved_rows": 777, + "achieved_count": 12_630.0, + "achieved_gains": 4.24e9, + }, + ] + + def _asset_type_evidence(**overrides: object) -> dict[str, object]: evidence: dict[str, object] = { "stage": "hmrc_cgt_asset_type_spine", "residential": { + "source_stage": "cgt_residential_split", "count_target_individuals_basis": 202_630.0, "gains_target_individuals_basis": 12.24e9, - "expected_count": 202_630.0, - "expected_gains": 12.24e9, - "achieved_count": 202_620.0, - "achieved_gains": 11.9e9, + "achieved_count": 202_630.0, + "achieved_gains": 12.24e9, "achieved_rows": 3_377, + "count_relative_error": 0.0, + "gains_relative_error": 0.0, "max_liable_weight": 60.0, - "count_bernoulli_sigma": 2_000.0, - "gains_bernoulli_sigma": 0.15e9, + "bands": _residential_bands(), }, "asset_type": { "achieved_gains_share": { @@ -898,80 +1099,58 @@ def _badr_receipt(**overrides: object) -> dict[str, object]: return receipt -def test_cgt_asset_type_summary_holds_the_residential_realisation() -> None: +def test_cgt_asset_type_summary_holds_the_residential_identities() -> None: parameters = { "stage": "hmrc_cgt_asset_type_spine", "check": "cgt_asset_type_summary", - "maximum_relative_deviation": 0.05, - "maximum_gains_sigma": 3.0, "maximum_solve_relative_error": 1e-6, } - passed = uk_stage_health_gate( - stage="hmrc_cgt_asset_type_spine", - check="cgt_asset_type_summary", - evidence=_asset_type_evidence(), - parameters=parameters, - ) + def gate(evidence: dict[str, object]): + return uk_stage_health_gate( + stage="hmrc_cgt_asset_type_spine", + check="cgt_asset_type_summary", + evidence=evidence, + parameters=parameters, + ) + + def with_residential(**changes: object) -> dict[str, object]: + return _asset_type_evidence( + residential={**_asset_type_evidence()["residential"], **changes} + ) + + passed = gate(_asset_type_evidence()) assert passed.passed - assert passed.details["residential_count_gap"] == 10.0 - assert passed.details["residential_gains_bound"] == 0.05 * 12.24e9 - - # A small frame: the noise floor is wider than the band and governs. - noisy = _asset_type_evidence( - residential={ - **_asset_type_evidence()["residential"], - "achieved_gains": 9.0e9, - "gains_bernoulli_sigma": 1.5e9, - } - ) - assert uk_stage_health_gate( - stage="hmrc_cgt_asset_type_spine", - check="cgt_asset_type_summary", - evidence=noisy, - parameters=parameters, - ).passed + assert passed.details["residential_count_relative_error"] == 0.0 + assert passed.details["residential_gains_relative_error"] == 0.0 + assert passed.details["residential_gain_bands"] == 2 - # A count more than one person off the expectation is a broken draw. - off_count = _asset_type_evidence( - residential={ - **_asset_type_evidence()["residential"], - "achieved_count": 202_500.0, - } - ) - failed = uk_stage_health_gate( - stage="hmrc_cgt_asset_type_spine", - check="cgt_asset_type_summary", - evidence=off_count, - parameters=parameters, - ) + # The arms carry Table 8a at design weights: a departure beyond the solve + # tolerance is not a realisation gap but a broken split. + failed = gate(with_residential(achieved_count=202_500.0)) assert not failed.passed - assert any("count gap" in failure for failure in failed.failures) + assert any("on the arms" in failure for failure in failed.failures) + failed = gate(with_residential(achieved_gains=11.9e9)) + assert not failed.passed + assert any("gains on the arms" in failure for failure in failed.failures) - drifted = _asset_type_evidence( - residential={ - **_asset_type_evidence()["residential"], - "achieved_gains": 10.0e9, - } - ) - failed = uk_stage_health_gate( - stage="hmrc_cgt_asset_type_spine", - check="cgt_asset_type_summary", - evidence=drifted, - parameters=parameters, - ) + # The receipt must come from the split stage and restate it by band. + failed = gate(with_residential(source_stage="hmrc_cgt_asset_type_spine")) assert not failed.passed - assert any("gains gap" in failure for failure in failed.failures) + assert any("cgt_residential_split" in failure for failure in failed.failures) + bands = _residential_bands() + bands[0]["achieved_gains"] = 7.0e9 + failed = gate(with_residential(bands=bands)) + assert not failed.passed + assert any("sums to" in failure for failure in failed.failures) + failed = gate(with_residential(bands=[])) + assert not failed.passed + assert any("no gain bands" in failure for failure in failed.failures) bad_share = _asset_type_evidence( asset_type={"achieved_gains_share": {"listed_shares": 1.5}} ) - failed = uk_stage_health_gate( - stage="hmrc_cgt_asset_type_spine", - check="cgt_asset_type_summary", - evidence=bad_share, - parameters=parameters, - ) + failed = gate(bad_share) assert not failed.passed with pytest.raises(ValueError, match="residential"): @@ -983,12 +1162,127 @@ def test_cgt_asset_type_summary_holds_the_residential_realisation() -> None: ) +def _residential_split_evidence(**overrides: object) -> dict[str, object]: + evidence: dict[str, object] = { + "stage": "cgt_residential_split", + "mass": {"old_total": 28_600_000.0, "new_total": 28_600_000.0}, + "households_before": 58_288, + "households_after": 60_600, + "arms_created": 2_312, + "households_by_liable_gainers": {"1": 2_300, "2": 4}, + "arm_weights_exact": True, + "identities": { + "count_target_individuals_basis": 202_630.0, + "gains_target_individuals_basis": 12.24e9, + "expected_count": 202_630.0, + "expected_gains": 12.24e9, + "achieved_count": 202_630.0, + "achieved_gains": 12.24e9, + "count_solve_relative_error": 1e-15, + "gains_solve_relative_error": 7e-10, + "count_identity_relative_error": 0.0, + "gains_identity_relative_error": 0.0, + }, + "bands": [ + { + "gain_lower_bound": 0.0, + "gain_upper_bound": 250_000.0, + "expected_count": 190_000.0, + "expected_gains": 8.0e9, + "achieved_count": 190_000.0, + "achieved_gains": 8.0e9, + }, + { + "gain_lower_bound": 250_000.0, + "gain_upper_bound": None, + "expected_count": 12_630.0, + "expected_gains": 4.24e9, + "achieved_count": 12_630.0, + "achieved_gains": 4.24e9, + }, + ], + "arm_weights": { + "residential_arms": 2_312, + "minimum": 0.04, + "maximum": 700.0, + "below_one_household": 300, + "below_one_tenth": 12, + }, + "concentration": { + "top_arms_share_of_achieved_gains": 0.12, + "largest_liable_stake_share_of_gains_target": 0.89, + }, + } + evidence.update(overrides) + return evidence + + +def test_cgt_residential_split_holds_the_identities_and_the_arm_structure() -> None: + parameters = { + "stage": "cgt_residential_split", + "check": "cgt_residential_split", + "maximum_relative_mass_deviation": 1e-9, + "maximum_solve_relative_error": 1e-6, + "maximum_identity_relative_error": 1e-9, + "maximum_liable_gainers_per_household": 3, + } + + def gate(evidence: dict[str, object]): + return uk_stage_health_gate( + stage="cgt_residential_split", + check="cgt_residential_split", + evidence=evidence, + parameters=parameters, + ) + + def with_identities(**changes: object) -> dict[str, object]: + return _residential_split_evidence( + identities={**_residential_split_evidence()["identities"], **changes} + ) + + passed = gate(_residential_split_evidence()) + assert _passed(passed) + assert passed.details["arms_created"] == 2_312 + assert passed.details["arm_below_one_household"] == 300 + assert passed.details["top_arms_share_of_achieved_gains"] == 0.12 + + # Mass created or lost, arms not the declared products, or an arm count + # that does not follow from the households by liable gainers. + failed = gate( + _residential_split_evidence( + mass={"old_total": 28_600_000.0, "new_total": 28_600_100.0} + ) + ) + assert not failed.passed and "mass deviation" in " ".join(failed.failures) + failed = gate(_residential_split_evidence(arm_weights_exact=False)) + assert not failed.passed and "declared products" in " ".join(failed.failures) + failed = gate(_residential_split_evidence(arms_created=2_311)) + assert not failed.passed and "imply 2312" in " ".join(failed.failures) + failed = gate( + _residential_split_evidence(households_by_liable_gainers={"1": 2_300, "4": 1}) + ) + assert not failed.passed and "above the declared maximum" in " ".join( + failed.failures + ) + # The solve must reach the targets and the arms must realise it exactly. + failed = gate(with_identities(count_solve_relative_error=1e-3)) + assert not failed.passed and "solve error" in " ".join(failed.failures) + failed = gate(with_identities(gains_identity_relative_error=1e-6)) + assert not failed.passed and "departs from its expectation" in " ".join( + failed.failures + ) + bands = _residential_split_evidence()["bands"] + bands[1] = {**bands[1], "achieved_gains": 4.0e9} + failed = gate(_residential_split_evidence(bands=bands)) + assert not failed.passed and "gain band from 250000.0" in " ".join(failed.failures) + failed = gate(_residential_split_evidence(bands=[])) + assert not failed.passed and "no gain bands" in " ".join(failed.failures) + + def test_cgt_asset_type_summary_holds_every_badr_band_to_the_walk_bound() -> None: parameters = { "stage": "hmrc_cgt_asset_type_spine", "check": "cgt_asset_type_summary", - "maximum_relative_deviation": 0.05, - "maximum_gains_sigma": 3.0, "maximum_solve_relative_error": 1e-6, } @@ -1347,84 +1641,187 @@ def run(ev): run(None) -def test_spi_income_band_donor_support_gate_checks_every_reserved_band() -> None: - bands = [200_000, 500_000, 1_000_000, 2_000_000] - taxpayers = { - 200_000: 359_000.0, - 500_000: 61_000.0, - 1_000_000: 20_000.0, - 2_000_000: 10_000.0, +def _spi_income_band_donor_evidence() -> dict[str, object]: + plan = { + 200_000: (1_436, 359_000.0), + 500_000: (244, 61_000.0), + 1_000_000: (120, 20_000.0), + 2_000_000: (120, 10_000.0), } rows = [ { "lower_bound": lower, - "donor_households": 120, - "carriers": 120, - "donor_weight": taxpayers[lower] / 120, - "weighted_taxpayers": taxpayers[lower], - "published_taxpayers": taxpayers[lower], + "expected_donor_households": count, + "donor_households": count, + "carriers": count, + "planned_donor_weight": taxpayers / count, + "donor_weight": taxpayers / count, + "weighted_taxpayers": taxpayers, + "published_taxpayers": taxpayers, } - for lower in bands + for lower, (count, taxpayers) in plan.items() ] - evidence = { + return { "stage": "spi_income_band_donors", - "donors_per_band": 120, + "minimum_donors_per_band": 120, + "maximum_donor_weight": 250.0, + "seating_scale": 1.0, + "mass_scale": 1.0, + "sample_fraction": 1.0, + "donor_count": 1_920, + "reallocated_mass": 450_000.0, + "mass": {"old_total": 28_600_000.0, "new_total": 28_600_000.0}, "bands": rows, + "funding": [ + { + "stratum": "LONDON", + "donor_households": 1_200, + "incumbent_mass": 3_700_000.0, + "donor_mass": 300_000.0, + "factor": 1.0 - 300_000.0 / 3_700_000.0, + }, + { + "stratum": "SOUTH_EAST", + "donor_households": 720, + "incumbent_mass": 3_900_000.0, + "donor_mass": 150_000.0, + "factor": 1.0 - 150_000.0 / 3_900_000.0, + }, + { + "stratum": "WALES", + "donor_households": 0, + "incumbent_mass": 1_400_000.0, + "donor_mass": 0.0, + "factor": 1.0, + }, + ], } - parameters = { - "stage": "spi_income_band_donors", - "check": "spi_income_band_donor_support", - "donors_per_band": 120, - "band_lower_bounds": bands, - } - assert _passed( - uk_stage_health_gate( - evidence=evidence, - stage="spi_income_band_donors", - check="spi_income_band_donor_support", - parameters=parameters, - ) + + +_SPI_INCOME_BAND_DONOR_GATE_PARAMETERS = { + "stage": "spi_income_band_donors", + "check": "spi_income_band_donor_support", + "minimum_donors_per_band": 120, + "maximum_donor_weight": 250.0, + "minimum_funding_factor": 0.5, + "maximum_band_taxpayer_deviation": 0.5, + "maximum_relative_mass_deviation": 1e-9, + "band_lower_bounds": [200_000, 500_000, 1_000_000, 2_000_000], +} + + +def _spi_income_band_donor_gate(evidence: dict[str, object]): + return uk_stage_health_gate( + evidence=evidence, + stage="spi_income_band_donors", + check="spi_income_band_donor_support", + parameters=_SPI_INCOME_BAND_DONOR_GATE_PARAMETERS, ) + + +def test_spi_income_band_donor_support_gate_checks_every_reserved_band() -> None: + evidence = _spi_income_band_donor_evidence() + result = _spi_income_band_donor_gate(evidence) + assert _passed(result) + assert result.details["donor_count"] == 1_920 + assert result.details["heaviest_donor_weight"] == 250.0 + rows = evidence["bands"] # A band whose copy lost its carrier fails. broken = [dict(row) for row in rows] broken[-1]["carriers"] = 119 - result = uk_stage_health_gate( - evidence={**evidence, "bands": broken}, - stage="spi_income_band_donors", - check="spi_income_band_donor_support", - parameters=parameters, - ) + result = _spi_income_band_donor_gate({**evidence, "bands": broken}) assert not result.passed and "119 carriers" in " ".join(result.failures) # A missing band fails against the declared four. - result = uk_stage_health_gate( - evidence={**evidence, "bands": rows[:-1]}, - stage="spi_income_band_donors", - check="spi_income_band_donor_support", - parameters=parameters, - ) + result = _spi_income_band_donor_gate({**evidence, "bands": rows[:-1]}) assert not result.passed and "differ from the declared" in " ".join(result.failures) - # A scaled rung (fewer donors than declared) is not held to the published - # mass, but a full stack whose weights do not sum to it is. - scaled = [{**row, "donor_households": 2, "carriers": 2} for row in rows] - assert _passed( - uk_stage_health_gate( - evidence={**evidence, "donors_per_band": 2, "bands": scaled}, - stage="spi_income_band_donors", - check="spi_income_band_donor_support", - parameters=parameters, - ) - ) + # A full stack whose weights do not sum to the published mass fails. off = [dict(row) for row in rows] off[0]["donor_weight"] = 1.0 - result = uk_stage_health_gate( - evidence={**evidence, "bands": off}, - stage="spi_income_band_donors", - check="spi_income_band_donor_support", - parameters=parameters, - ) + result = _spi_income_band_donor_gate({**evidence, "bands": off}) assert not result.passed and "differ from the published" in " ".join( result.failures ) + # A donor above the maximum weight fails, as does a band seated below the + # minimum at full scale. + heavy = [dict(row) for row in rows] + heavy[0].update(donor_households=718, carriers=718, donor_weight=500.0) + failures = " ".join( + _spi_income_band_donor_gate({**evidence, "bands": heavy}).failures + ) + assert "exceeds the maximum 250.0" in failures + assert "the plan seats 1436" in failures + + +def test_spi_income_band_donor_support_gate_holds_the_mass_conserved() -> None: + evidence = _spi_income_band_donor_evidence() + # Given donors added on top of the incoming mass (the pre-#1063 stage), + # the gate refuses. + added = {**evidence, "mass": {"old_total": 28_600_000.0, "new_total": 29_050_000.0}} + result = _spi_income_band_donor_gate(added) + assert not result.passed and "funded from the incumbent households" in " ".join( + result.failures + ) + # A stratum scaled below the funding floor fails. + funding = [dict(row) for row in evidence["funding"]] + funding[0]["factor"] = 0.49 + result = _spi_income_band_donor_gate({**evidence, "funding": funding}) + assert not result.passed and "outside [0.5, 1]" in " ".join(result.failures) + # Funding rows that do not account for the reallocated mass fail. + funding = [dict(row) for row in evidence["funding"]] + funding[1]["donor_mass"] = 100_000.0 + result = _spi_income_band_donor_gate({**evidence, "funding": funding}) + assert not result.passed and "funding strata carry donor mass" in " ".join( + result.failures + ) + # The receipt must be the reviewed rule's. + result = _spi_income_band_donor_gate({**evidence, "maximum_donor_weight": 500.0}) + assert not result.passed and "differs from the gate's 250.0" in " ".join( + result.failures + ) + with pytest.raises(ValueError, match="funding must be a non-empty list"): + _spi_income_band_donor_gate({**evidence, "funding": []}) + + +def test_spi_income_band_donor_support_gate_scales_only_off_the_full_sample() -> None: + evidence = _spi_income_band_donor_evidence() + # Given a small frame: fewer, lighter donors. The bands fall short of the + # published mass by construction and the gate passes on the receipt. + scale = 0.5 + rows = [ + { + **row, + "donor_households": 2, + "carriers": 2, + "donor_weight": row["planned_donor_weight"] * scale, + } + for row in evidence["bands"] + ] + donor_mass = sum(row["donor_weight"] * 2 for row in rows) + scaled = { + **evidence, + "seating_scale": 8 / 1_920, + "mass_scale": scale, + "sample_fraction": 0.01, + "donor_count": 8, + "reallocated_mass": donor_mass, + "bands": rows, + "funding": [ + { + "stratum": "LONDON", + "donor_households": 8, + "incumbent_mass": 2.0 * donor_mass, + "donor_mass": donor_mass, + "factor": 0.5, + } + ], + } + assert _passed(_spi_income_band_donor_gate(scaled)) + # The same receipt on a full-sample build fails: a full build seats and + # funds the donors at full scale. + result = _spi_income_band_donor_gate({**scaled, "sample_fraction": 1.0}) + assert not result.passed and "at full scale" in " ".join(result.failures) + result = _spi_income_band_donor_gate({**scaled, "mass_scale": 0.0}) + assert not result.passed and "outside (0, 1]" in " ".join(result.failures) def test_spi_support_channel_checks_the_declared_pension_age_share() -> None: @@ -1508,3 +1905,68 @@ def gate(*bands): # the failure without blocking, every other posture blocks on it. committed = _gate_parameters("uk_stage_pension_credit_take_up") assert committed["check"] == "pension_credit_take_up" + + +def test_child_benefit_take_up_holds_claims_by_age_and_the_opt_out_share() -> None: + parameters = { + "stage": "child_benefit_take_up", + "check": "child_benefit_take_up", + "maximum_claim_rate_deviation": 0.02, + "maximum_age_claim_rate_deviation": 0.05, + "minimum_age_child_rows": 200, + "maximum_opt_out_share_deviation": 0.01, + "minimum_eligible_family_units": 1, + } + + def age(value, published, realized, *, rows=1_000, clipped=False): + return { + "age": value, + "published_rate": published, + "realized_rate": realized, + "eligible_child_rows": rows, + "clipped": clipped, + } + + def gate(*, realized=0.868, ages=None, share=0.0905, exhausted=False, units=100): + evidence = { + "stage": "child_benefit_take_up", + "claims": { + "eligible_family_units": units, + "target_rate": 0.871, + "realized_rate": realized, + "ages": ages + if ages is not None + else [age(0, 0.688, 0.69), age(1, 0.768, 0.75)], + }, + "opt_outs": { + "target_share": 0.0907, + "realized_share": share, + "pool_exhausted": exhausted, + }, + } + return uk_stage_health_gate( + evidence=evidence, + stage="child_benefit_take_up", + check="child_benefit_take_up", + parameters=parameters, + ) + + result = gate() + assert _passed(result) + assert result.details["ages_measured"] == 2 + assert result.details["largest_age_deviation"]["age"] == 1 + # The overall claimed share is held to the published rates at the frame's + # age mix. + assert "against 0.8710" in " ".join(gate(realized=0.90).failures) + # An age is held where it has the rows to measure and was not clipped. + assert "at age 1" in " ".join(gate(ages=[age(1, 0.768, 0.70)]).failures) + assert _passed(gate(ages=[age(1, 0.768, 0.70, rows=50)])) + clipped = gate(ages=[age(19, 0.497, 0.60, clipped=True)]) + assert _passed(clipped) and clipped.details["clipped_ages"] == [19] + # The opted-out share may fall short only where the charged families ran + # out, and never exceeds the published share. + assert "opted out against" in " ".join(gate(share=0.05).failures) + assert _passed(gate(share=0.05, exhausted=True)) + assert not gate(share=0.12, exhausted=True).passed + assert not gate(units=0).passed + assert not gate(ages=[]).passed diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_student_loans.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_student_loans.py index 194ddd532..a261848c5 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_student_loans.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_student_loans.py @@ -285,6 +285,13 @@ def test_top_up_receipt_records_the_taken_rows_lineage() -> None: "household_support_channel": ["frs", "spi", "frs", "spi", "frs"], "household_is_cgt_support_copy": [False, False, True, True, False], "household_is_capital_gains_clone": [False, True, False, False, True], + "household_is_cgt_residential_clone": [ + False, + False, + False, + True, + False, + ], }, ), stocks=_stocks(plan_2=4, plan_5=0), @@ -297,12 +304,14 @@ def test_top_up_receipt_records_the_taken_rows_lineage() -> None: channels = np.asarray(["frs", "spi", "frs", "spi", "frs"])[taken] copies = np.asarray([False, False, True, True, False])[taken] clones = np.asarray([False, True, False, False, True])[taken] + arms = np.asarray([False, False, False, True, False])[taken] weights = np.asarray([5.0, 3.0, 2.0, 2.0, 1.0])[taken] assert lineage["columns"] == [ "household_support_channel", "household_is_cgt_support_copy", "household_is_capital_gains_clone", + "household_is_cgt_residential_clone", ] assert lineage["rows_by_channel"] == { channel: int((channels == channel).sum()) for channel in sorted(set(channels)) @@ -315,6 +324,8 @@ def test_top_up_receipt_records_the_taken_rows_lineage() -> None: assert lineage["support_copy_mass"] == float(weights[copies].sum()) assert lineage["clone_rows"] == int(clones.sum()) assert lineage["clone_mass"] == float(weights[clones].sum()) + assert lineage["residential_arm_rows"] == int(arms.sum()) + assert lineage["residential_arm_mass"] == float(weights[arms].sum()) assert sum(lineage["rows_by_channel"].values()) == receipt.topped_up_rows assert receipt.evidence()["topped_up_lineage"] == lineage diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_take_up_gate.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_take_up_gate.py index 4a4085f26..6e3339327 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_take_up_gate.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_take_up_gate.py @@ -89,7 +89,11 @@ def test_take_up_gate_seeded_fixture_passes() -> None: ) assert result.passed is True - assert "benunit.would_claim_child_benefit" in result.details + assert "benunit.would_claim_uc" in result.details + # The Child Benefit flags are measured by their own stage gate + # (microcosm#1063), like would_claim_pc (microcosm#1069). + assert "benunit.would_claim_child_benefit" not in result.details + assert "benunit.child_benefit_opts_out" not in result.details def test_take_up_gate_constant_column_fails() -> None: diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_tenure_constants.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_tenure_constants.py new file mode 100644 index 000000000..f2880c63a --- /dev/null +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_tenure_constants.py @@ -0,0 +1,35 @@ +from __future__ import annotations + +from microcosm.build.uk_runtime import local_targets +from microcosm.build.uk_runtime.frs_spine import TENURE_MAP +from microcosm.build.uk_runtime.tenure_constants import ( + UK_OWNER_TENURE_CATEGORIES, + UK_TENURE_CATEGORIES, + UK_TENURE_TYPE_TO_CATEGORY, +) +from microcosm.build.uk_runtime.was_wealth import ( + UK_WAS_STRATIFIED_TARGETS, + UK_WAS_TENURE_PREDICTORS, + WAS_OWNER_TENURE_CODES, +) + + +def test_every_spine_tenure_type_has_a_category() -> None: + """The FRS spine's tenure values are exactly the mapped engine names.""" + + assert set(TENURE_MAP.values()) == set(UK_TENURE_TYPE_TO_CATEGORY) + assert set(UK_TENURE_TYPE_TO_CATEGORY.values()) == set(UK_TENURE_CATEGORIES) + assert set(UK_OWNER_TENURE_CATEGORIES) < set(UK_TENURE_CATEGORIES) + + +def test_consumers_derive_their_views_from_the_shared_categories() -> None: + assert set(local_targets._TENURE_METRIC_CATEGORIES.values()) == set( + UK_TENURE_CATEGORIES + ) + # One WAS flag per category but the base level, in category order. + assert UK_WAS_TENURE_PREDICTORS == tuple( + f"tenure_{category}" for category in UK_TENURE_CATEGORIES[1:] + ) + assert set(WAS_OWNER_TENURE_CODES.values()) == set(UK_OWNER_TENURE_CATEGORIES) + for categories in UK_WAS_STRATIFIED_TARGETS.values(): + assert set(categories) <= set(UK_TENURE_CATEGORIES) diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_terminal_gates.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_terminal_gates.py index fcaf03d39..da447d30e 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_terminal_gates.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_terminal_gates.py @@ -282,6 +282,75 @@ def test_export_surface_allows_claimant_roles_but_not_unreviewed_columns() -> No assert any("unreviewed_extra" in failure for failure in unrelated.failures) +def test_export_candidate_columns_strip_ids_and_carry_the_weight() -> None: + """microcosm#1063 c9: the certifier rehearsal listed every id column as an + unreviewed extra and the frame's weights as a missing reference column.""" + import numpy as np + import pandas as pd + + from microcosm.build.uk_runtime.national_frame import ( + UK_RELEASE_EXPORT_DROPPED_COLUMNS, + uk_national_frame, + uk_release_export_frame, + ) + from microcosm.build.uk_runtime.terminal_gates import ( + UK_REVIEWED_EXPORT_EXCLUSIONS, + uk_export_candidate_columns, + ) + + frame = uk_national_frame( + person=pd.DataFrame( + { + "person_id": [1, 2], + "person_benunit_id": [1, 2], + "person_household_id": [1, 2], + "age": [30, 40], + "incapacity_benefit_reported": [0.0, 0.0], + } + ), + benunit=pd.DataFrame({"benunit_id": [1, 2]}), + household=pd.DataFrame({"household_id": [1, 2], "region": ["LONDON"] * 2}), + time_period="2024", + household_weights=np.asarray([1.0, 2.0]), + ) + assert uk_export_candidate_columns(frame) == { + "person.age", + "person.incapacity_benefit_reported", + "household.region", + "household.household_weight", + } + exported = uk_release_export_frame(frame) + assert uk_export_candidate_columns(exported) == { + "person.age", + "household.region", + "household.household_weight", + } + assert exported.weights_for("household").values.tolist() == [1.0, 2.0] + assert exported.mass_log == frame.mass_log + # The boundary drops the reviewed export exclusions, the internal + # disability carriers and the salary-sacrifice pre-conversion pay + # carrier, and nothing else. + from microcosm.build.uk_runtime.frs_disability import ( + UK_INTERNAL_DISABILITY_REPORTED_COLUMNS, + ) + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) + + dropped = { + f"{entity}.{column}" + for entity, columns in UK_RELEASE_EXPORT_DROPPED_COLUMNS.items() + for column in columns + } + assert dropped == ( + set(UK_REVIEWED_EXPORT_EXCLUSIONS) + | {f"person.{column}" for column in UK_INTERNAL_DISABILITY_REPORTED_COLUMNS} + | {f"person.{SALSAC_PRE_CONVERSION_PAY_COLUMN}"} + ) + # A frame without the column passes through untouched. + assert uk_release_export_frame(exported) is exported + + def _target_fit_exclusion( *, approved_on: str = "2026-08-30", @@ -397,9 +466,19 @@ def test_committed_target_fit_register_retains_only_live_deferrals() -> None: # fits at +24.5 %, back inside the bound. The state-pension 50-70k # deferral (ruling D6) is retired in turn (microcosm#1069): with the SPI # channel's State Pension at 66 repaired the row fits at +23.0 % to +24.0 % - # across the c5-fix and c6 arms, back inside the bound. The register is - # empty. - assert set(register) == set() + # across the c5-fix and c6 arms, back inside the bound. The SE + # 12,570-15,000 cell is deferred again on the microcosm#1063 stack + # (2026-10-02, four weeks): its design-weight value sits on the target and + # the solver's pull, the same as main's, crosses the bound at +32.0 %. + assert set(register) == { + "hmrc.spi_region.income_tax_by_region_12570_15000@E12000008@2025" + } + deferral = register[ + "hmrc.spi_region.income_tax_by_region_12570_15000@E12000008@2025" + ] + assert deferral.approved_on == "2026-10-02" + assert deferral.expires_on == "2026-11-02" + assert "microcosm#1063" in deferral.adjudication # Aggregate errors from the fresh UC #882 development run: 1,500 epochs with @@ -424,7 +503,7 @@ def test_restored_fit_checks_leave_empty_payment_tail_cells_blocked() -> None: **empty_tail, }, reviewed_exclusions=uk_default_target_fit_reviewed_exclusions(), - now=date(2026, 9, 23), + now=date(2026, 10, 2), ) assert not fit.passed @@ -446,7 +525,7 @@ def test_restored_fit_checks_apply_if_a_later_run_breaches_again( fit = uk_target_fit_gate( {name: relative_error}, reviewed_exclusions=uk_default_target_fit_reviewed_exclusions(), - now=date(2026, 9, 23), + now=date(2026, 10, 2), ) assert fit.passed is passes @@ -461,7 +540,7 @@ def test_observed_liability_has_no_retired_cash_exemption() -> None: fit = uk_target_fit_gate( {"hmrc.cgt.liability_total@2025": 0.30}, reviewed_exclusions=register, - now=date(2026, 9, 23), + now=date(2026, 10, 2), ) assert not fit.passed assert fit.details["failing_targets"] == {"hmrc.cgt.liability_total@2025": 0.30} diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_capital_coherence.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_capital_coherence.py index 30e375c3e..dba1371d9 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_capital_coherence.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_capital_coherence.py @@ -55,12 +55,14 @@ def test_stage_orders_after_every_universal_credit_report_writer() -> None: assert reporter_writers assert all(index < coherence_index for index, _ in reporter_writers) - # The Pension Credit take-up redraw sits between the coherence stage and the - # deduction attributes (microcosm#1069 c7); it writes no UC report. + # The Pension Credit take-up redraw (microcosm#1069 c7) and the Child + # Benefit redraw (microcosm#1063 c8) sit between the coherence stage and + # the deduction attributes; neither writes a UC report. assert [ - stage.stage for stage in stages[coherence_index + 1 : coherence_index + 3] + stage.stage for stage in stages[coherence_index + 1 : coherence_index + 4] ] == [ "pension_credit_take_up", + "child_benefit_take_up", "uc_deduction_attributes", ] diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_family.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_family.py index 77d621d6b..4bfe15d9d 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_family.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_uc_family.py @@ -349,12 +349,16 @@ def test_full_uc_payment_registry_matches_independent_relationship_and_band_case ] full_registry = TargetRegistry(specs, country="uk") active = [spec for spec in specs if spec.name not in excluded] - # 83 active / 17 excluded since the 2026-09-21 A16 retirements: the - # LONE_PARENT 10,800-12,000 payment-band row measured inside the fence on - # the v20 national build and left the exclusion register. + # 81 active / 19 excluded: the 2026-09-21 A16 retirements returned the + # LONE_PARENT 10,800-12,000 payment-band row (measured inside the fence + # on the v20 national build), and microcosm#1063 excluded the two thin + # cells whose one or two supporting households left the band when the + # larger donor block reordered the rows the later positional draws read + # (SINGLE 28,800-30,000 and COUPLE_NO_CHILDREN 22,800-24,000, the + # uk-data#452 class). assert len(specs) == 100 - assert len(active) == 83 - assert len({s.name for s in specs} & excluded) == 17 + assert len(active) == 81 + assert len({s.name for s in specs} & excluded) == 19 registry = TargetRegistry(active, country="uk") result = materialize_uk_ledger_targets( adapter, registry, period=2025, band_edge_registry=full_registry @@ -364,7 +368,7 @@ def test_full_uc_payment_registry_matches_independent_relationship_and_band_case adapter.to_frame(), registry.to_target_set(), weight_entity="household" ) assert not problem.skipped - assert len(problem.names) == 83 + assert len(problem.names) == 81 assert not {f"{name}@2025" for name in excluded}.intersection(problem.names) expected_family = np.asarray(expected_families)[bu.benunit_id.to_numpy()] uc = bu.universal_credit.to_numpy() diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_was_wealth.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_was_wealth.py index 20312abb8..849582d80 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_was_wealth.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_was_wealth.py @@ -8,17 +8,55 @@ from microcosm.build.source_manifest import SourceStageSpec from microcosm.build.uk_runtime.national_frame import uk_national_frame +from microcosm.build.uk_runtime.tenure_constants import UK_TENURE_CATEGORIES from microcosm.build.uk_runtime.was_wealth import ( REGIONS, + UK_WAS_DERIVED_TOTALS, + UK_WAS_DRAWN_ONLY_COLUMNS, UK_WAS_ENGINE_PREDICTOR_ENTITIES, + UK_WAS_NET_FINANCIAL_LIABILITIES, + UK_WAS_STRATIFIED_TARGETS, + UK_WAS_TENURE_CATEGORY_COLUMN, + UK_WAS_TENURE_PREDICTORS, UK_WAS_WEALTH_OUTPUT_COLUMNS, + UK_WAS_WEALTH_PREDICTORS, UKWASWealthStageTransform, allocate_student_loan_balance_to_people, + cap_derived_totals_to_donor_range, clean_was_household_table, + impute_was_wealth, recipient_predictors, + recipient_tenure_category, support_clip_to_donor, + tenure_coherence_receipt, + was_tenure_category, + wealth_identity_violations, ) +_DRAW_COLUMNS = [*UK_WAS_WEALTH_OUTPUT_COLUMNS, *UK_WAS_DRAWN_ONLY_COLUMNS] + + +def _chain_operation(n_estimators: int = 2) -> dict[str, object]: + return { + "kind": "fit_weighted_qrf_chain", + "seed": 0, + "n_estimators": n_estimators, + "stratified_targets": { + target: list(categories) + for target, categories in UK_WAS_STRATIFIED_TARGETS.items() + }, + "derived_totals": { + **{ + total: list(components) + for total, components in UK_WAS_DERIVED_TOTALS.items() + }, + "net_financial_wealth": [ + "gross_financial_wealth", + *(f"-{column}" for column in UK_WAS_NET_FINANCIAL_LIABILITIES), + ], + }, + } + class _FakeEngine: """Returns engine variables at their native entity, like the real adapter.""" @@ -49,14 +87,7 @@ def _stage() -> SourceStageSpec: "source": "test", "grain": "household", "artifacts": [], - "operations": [ - { - "kind": "fit_weighted_qrf_chain", - "seed": 0, - "n_estimators": 2, - "debt_segment_predictors": ["has_mortgage_tenure"], - } - ], + "operations": [_chain_operation()], "outputs": list(UK_WAS_WEALTH_OUTPUT_COLUMNS), "nonnegative_outputs": [ name @@ -68,11 +99,15 @@ def _stage() -> SourceStageSpec: def _raw_was() -> pd.DataFrame: + """Two donors that satisfy the round-8 identities: a mortgaged owner and a + social renter. Property is the sum of the property values plus 90 / 400 of + other property; gross financial wealth is the listed assets plus 5 / 14.""" + return pd.DataFrame( { "R8xshhwgt": [2.0, 3.0], "DVLUKValR8_sum": [10.0, 100.0], - "DVPropertyR8": [20.0, 200.0], + "DVPropertyR8": [104_100.0, 6_500.0], "DVFESHARESR8_aggr": [1.0, 2.0], "DVFShUKVR8_aggr": [3.0, 4.0], "DVIISAVR8_aggR": [5.0, 6.0], @@ -88,14 +123,16 @@ def _raw_was() -> pd.DataFrame: "DVGIEMPR8_AGGR": [17.0, 18.0], "HBedRmR8": [3, 4], "GORR8": [8, 12], - "DVPriRntR8": [1, 2], + "DVPriRntR8": [-9, 2], "CTAmtR8": [1000.0, 1200.0], - "HFINWNTR8_Sum": [-5.0, 50.0], - "HFINWNTR8_exSLC_Sum": [20.0, 40.0], - "HFINWR8_SUM": [30.0, 40.0], + "HFINWNTR8_Sum": [-4_480.0, -1_356.0], + "HFINWNTR8_exSLC_Sum": [520.0, 644.0], + "HFINWR8_SUM": [530.0, 644.0], "HMortGR8": [1000.0, 0.0], + "TotMortR8": [1000.0, 0.0], + "OthMortR8_sum": [0.0, 0.0], "Ten1R8": [2, 4], - "DVhvalueR8": [100000.0, 200000.0], + "DVhvalueR8": [100_000.0, 0.0], "DVHseValR8_sum": [1000.0, 2000.0], "DVBlDValR8_sum": [3000.0, 4000.0], "DVTotinc_bhcR8": [50000.0, 60000.0], @@ -144,7 +181,6 @@ def _frame() -> object: "num_bedrooms": [3, 4], "council_tax": [1000.0, 1200.0], "household_net_income": [50000.0, 60000.0], - "is_renting": [False, True], "tenure_type": ["OWNED_WITH_MORTGAGE", "OWNED_OUTRIGHT"], } ) @@ -169,24 +205,98 @@ def test_was_donor_cleaning_arithmetic_and_exact_case_insensitive_columns() -> N assert donor["student_loan_balance"].tolist() == [5000.0, 2000.0] assert donor["mortgage_debt"].tolist() == [1000.0, 0.0] assert donor["consumer_debt"].tolist() == [10.0, 0.0] - assert donor["has_mortgage_tenure"].tolist() == [True, False] + assert donor[UK_WAS_TENURE_CATEGORY_COLUMN].tolist() == [ + "owned_with_mortgage", + "social_rent", + ] + assert donor["tenure_owned_with_mortgage"].tolist() == [True, False] + assert donor["tenure_social_rent"].tolist() == [False, True] + assert donor["tenure_private_rent"].tolist() == [False, False] + # The remainders that make each total the sum of its drawn components. + assert donor["other_property_value"].tolist() == [90.0, 400.0] + assert donor["other_financial_assets"].tolist() == [5.0, 14.0] + assert donor["main_residence_mortgage"].tolist() == [1000.0, 0.0] + assert donor["other_mortgage"].tolist() == [0.0, 0.0] assert donor["region"].tolist() == ["LONDON", "SCOTLAND"] - assert donor["is_renting"].tolist() == [True, False] + # The private-renter flag the Lifetime ISA stage reads on both sides. + assert donor["is_renting"].tolist() == [False, False] assert 3 not in REGIONS +def test_was_tenure_category_reads_the_round_8_codes() -> None: + codes = pd.Series([1, 2, 3, 4, 4, 5, 5, 6, -8]) + private = pd.Series([-9, -9, -9, 1, 2, 1, 2, -9, -9]) + + assert was_tenure_category(codes, private).tolist() == [ + "owned_outright", + "owned_with_mortgage", + "owned_with_mortgage", + "private_rent", + "social_rent", + "private_rent", + "social_rent", + # Codes outside 1-5 take the FRS spine's own fallback and are counted. + "private_rent", + "private_rent", + ] + raw = pd.concat([_raw_was()] * 2, ignore_index=True) + raw.loc[3, "Ten1R8"] = 6 + donor = clean_was_household_table(raw) + assert donor["tenure_code_unclassified"].tolist() == [False, False, False, True] + + +def test_recipient_tenure_category_groups_council_and_housing_association() -> None: + tenure = pd.Series( + [ + "OWNED_OUTRIGHT", + "OWNED_WITH_MORTGAGE", + "RENT_PRIVATELY", + "RENT_FROM_COUNCIL", + "RENT_FROM_HA", + ] + ) + + assert recipient_tenure_category(tenure).tolist() == [ + "owned_outright", + "owned_with_mortgage", + "private_rent", + "social_rent", + "social_rent", + ] + with pytest.raises(ValueError, match="outside the engine's enum"): + recipient_tenure_category(pd.Series(["RENT_FREE"])) + + +@pytest.mark.parametrize( + ("column", "value", "match"), + [ + ("DVPropertyR8", 1.0, "property_wealth is below the sum"), + ("HFINWR8_SUM", 1.0, "gross_financial_wealth is below the sum"), + ("HFINWNTR8_Sum", 0.0, "net financial wealth is not gross"), + ], +) +def test_was_donor_breaking_an_identity_is_refused(column, value, match) -> None: + raw = _raw_was() + raw.loc[0, column] = value + + with pytest.raises(ValueError, match=match): + clean_was_household_table(raw) + + def test_was_donor_sentinel_codes_recode_to_zero_for_nonnegative_domains() -> None: raw = _raw_was() raw.loc[0, "vcarnr8"] = -8 raw.loc[1, "HBedRmR8"] = -8 + # -8 is a legal net financial wealth value; the liabilities move with it + # so the row still satisfies the identity. raw.loc[0, "HFINWNTR8_Sum"] = -8.0 + raw.loc[0, "Tot_LosR8_aggr"] = raw.loc[0, "Tot_los_exc_SLCR8_aggr"] + 528.0 donor = clean_was_household_table(raw) assert donor["num_vehicles"].tolist()[0] == 0 assert donor["num_bedrooms"].tolist()[1] == 0 - # Genuinely negative domains are never recoded: -8 is a legal net - # financial wealth value. + # Genuinely negative domains are never recoded. assert donor["net_financial_wealth"].tolist()[0] == -8.0 @@ -214,8 +324,16 @@ def test_recipient_predictors_sum_person_and_benunit_variables_to_household() -> assert predictors["num_adults"].tolist() == [2.0, 2.0] assert predictors["num_children"].tolist() == [0.0, 0.0] assert predictors["household_net_income"].tolist() == [50000.0, 60000.0] - assert predictors["is_renting"].tolist() == [False, True] - assert predictors["has_mortgage_tenure"].tolist() == [True, False] + assert predictors[UK_WAS_TENURE_CATEGORY_COLUMN].tolist() == [ + "owned_with_mortgage", + "owned_outright", + ] + assert predictors["tenure_owned_with_mortgage"].tolist() == [True, False] + assert predictors["tenure_private_rent"].tolist() == [False, False] + assert predictors["tenure_social_rent"].tolist() == [False, False] + # The stage reads no engine tenure formula. + assert "is_renting" not in predictors + assert "is_renting" not in UK_WAS_ENGINE_PREDICTOR_ENTITIES def test_recipient_predictors_fail_loud_on_entity_mismatch() -> None: @@ -268,14 +386,14 @@ def test_support_clip_and_integer_vehicle_output( ) -> None: donor = clean_was_household_table(_raw_was()) draws = pd.DataFrame( - {column: [999999.0, -999999.0] for column in UK_WAS_WEALTH_OUTPUT_COLUMNS} + {column: [9_999_999.0, -9_999_999.0] for column in UK_WAS_WEALTH_OUTPUT_COLUMNS} ) clip_result = support_clip_to_donor(draws, donor) clipped = clip_result.clipped assert clipped["owned_land"].tolist() == [100.0, 10.0] - assert clipped["net_financial_wealth"].tolist() == [50.0, -5.0] + assert clipped["net_financial_wealth"].tolist() == [-1_356.0, -4_480.0] receipt = clip_result.receipt.evidence()["columns"] assert receipt["owned_land"] == { "donor_min": 10.0, @@ -285,8 +403,8 @@ def test_support_clip_and_integer_vehicle_output( "rows_considered": 2, } assert receipt["net_financial_wealth"] == { - "donor_min": -5.0, - "donor_max": 50.0, + "donor_min": -4_480.0, + "donor_max": -1_356.0, "clipped_low_rows": 1, "clipped_high_rows": 1, "rows_considered": 2, @@ -295,7 +413,7 @@ def test_support_clip_and_integer_vehicle_output( import microcosm.build.uk_runtime.was_wealth as module def fake_impute(*args, **kwargs): - result = donor.loc[:, UK_WAS_WEALTH_OUTPUT_COLUMNS].reset_index(drop=True) + result = donor.loc[:, _DRAW_COLUMNS].reset_index(drop=True) result = result.copy() result["num_vehicles"] = [1.2, 2.8] return module.UKWASWealthImputationResult( @@ -335,6 +453,35 @@ def fake_impute(*args, **kwargs): "clipped_high_rows": 0, "rows_considered": 2, } + evidence = transform.checkpoint_metadata()["evidence"] + # The drawn-only components never reach the frame. + assert not set(UK_WAS_DRAWN_ONLY_COLUMNS) & set( + transformed.table("household").columns + ) + assert evidence["identities"] == { + "property_wealth_violation_rows": 0, + "corporate_wealth_violation_rows": 0, + "gross_financial_wealth_violation_rows": 0, + "mortgage_debt_violation_rows": 0, + "net_financial_wealth_violation_rows": 0, + "property_wealth_capped_rows": 0, + "corporate_wealth_capped_rows": 0, + "gross_financial_wealth_capped_rows": 0, + "mortgage_debt_capped_rows": 0, + "net_financial_wealth_capped_high_rows": 0, + "net_financial_wealth_capped_low_rows": 0, + } + tenure = evidence["tenure_coherence"] + assert tenure["main_residence_mortgage_off_mortgaged_tenure_rows"] == 0 + assert tenure["main_residence_value_off_owner_tenure_rows"] == 0 + # The fake imputer hands the outright owner the renter donor's row. + assert tenure["owner_rows_without_main_residence_value"] == 1 + assert tenure["donor_rows_by_tenure"] == { + "owned_outright": 0, + "owned_with_mortgage": 1, + "private_rent": 0, + "social_rent": 1, + } assert "student_loan_balance" not in transformed.table("household").columns assert transformed.table("person")["student_loan_balance"].sum() == pytest.approx( 7000.0 @@ -351,7 +498,7 @@ def test_stage_transform_is_deterministic_with_fast_synthetic_imputer( module, "impute_was_wealth", lambda *args, **kwargs: module.UKWASWealthImputationResult( - draws=donor.loc[:, UK_WAS_WEALTH_OUTPUT_COLUMNS].reset_index(drop=True), + draws=donor.loc[:, _DRAW_COLUMNS].reset_index(drop=True), fit_weight_records=(), ), ) @@ -394,14 +541,7 @@ def test_stage_transform_refuses_sha_mismatched_tab(tmp_path) -> None: "runtime_sha256_required": True, } ], - "operations": [ - { - "kind": "fit_weighted_qrf_chain", - "seed": 0, - "n_estimators": 2, - "debt_segment_predictors": ["has_mortgage_tenure"], - } - ], + "operations": [_chain_operation()], "outputs": list(UK_WAS_WEALTH_OUTPUT_COLUMNS), } ) @@ -422,6 +562,7 @@ def test_was_imputer_uses_checkpointed_chain_segments( import microcosm.fit calls = [] + donors = [] seeds = [] class FakeQRF: @@ -432,7 +573,10 @@ def __init__(self, *, n_estimators, seed): def start_chain(self, donor, predictors, targets, *, weights): assert weights == "weight" calls.append((tuple(predictors), tuple(targets))) - return SimpleNamespace(targets=tuple(targets), position=0) + donors.append(len(donor)) + return SimpleNamespace( + targets=tuple(targets), position=0, weight_kind="explicit" + ) def fit_draw_next( self, @@ -448,6 +592,7 @@ def fit_draw_next( if state.position: previous = state.targets[state.position - 1] assert previous in raw_prior_draws + assert raw_prior_draws.index.equals(recipient_predictors.index) return SimpleNamespace( target=target, raw_draw=np.full(len(recipient_predictors), float(state.position + 1)), @@ -455,6 +600,7 @@ def fit_draw_next( state=SimpleNamespace( targets=state.targets, position=state.position + 1, + weight_kind="explicit", ), ) @@ -464,23 +610,62 @@ def fit_draw_next( result = module.impute_was_wealth(donor, recipient, seed=0, n_estimators=7) - assert result.draws.columns.tolist() == list(UK_WAS_WEALTH_OUTPUT_COLUMNS) - assert calls[0][1] == ("owned_land", "property_wealth") - assert calls[1][1] == ( - "private_pension_wealth", - "corporate_wealth_excl_isa", - "stocks_and_shares_isa", - ) - assert "private_pension_wealth" in calls[2][0] - assert "corporate_wealth" in calls[2][0] - assert "private_pension_wealth" not in calls[1][0] - assert calls[2][1][-1] == "cash_isa" - assert calls[3][1] == ("mortgage_debt", "consumer_debt") - assert calls[3][0] == ( - *calls[0][0], - "has_mortgage_tenure", - *UK_WAS_WEALTH_OUTPUT_COLUMNS[:-2], + assert result.draws.columns.tolist() == _DRAW_COLUMNS + assert [targets for _, targets in calls] == [ + ("owned_land",), + ("main_residence_value",), + ( + "other_residential_property_value", + "non_residential_property_value", + "other_property_value", + ), + ( + "private_pension_wealth", + "corporate_wealth_excl_isa", + "stocks_and_shares_isa", + ), + ( + "savings", + "cash_isa", + "other_financial_assets", + "num_vehicles", + "student_loan_balance", + ), + ("main_residence_mortgage",), + ("other_mortgage",), + ("consumer_debt",), + ] + base = calls[0][0] + assert set(UK_WAS_TENURE_PREDICTORS) <= set(base) + assert "is_renting" not in base + assert calls[1][0] == (*base, "owned_land") + assert calls[2][0] == (*base, "owned_land", "main_residence_value") + assert calls[3][0] == (*base, "owned_land", "property_wealth") + assert "private_pension_wealth" in calls[4][0] + assert "corporate_wealth" in calls[4][0] + prior = tuple( + column + for column in UK_WAS_WEALTH_OUTPUT_COLUMNS + if column not in ("mortgage_debt", "consumer_debt", "net_financial_wealth") ) + assert calls[5][0] == (*base, *prior) + assert calls[6][0] == (*base, *prior, "main_residence_mortgage") + assert calls[7][0] == (*base, *prior, "mortgage_debt") + # The stratified targets are fitted on their stratum's donors only: the one + # mortgaged owner of the two donors; the other mortgages on both. + assert donors == [2, 1, 2, 2, 2, 1, 2, 2] + # Both recipients are owners, one of them mortgaged: the outright owner + # holds no main-residence mortgage by rule but may hold other mortgages, + # and mortgage_debt is the sum of the two. + assert result.draws["main_residence_value"].tolist() == [1.0, 1.0] + assert result.draws["main_residence_mortgage"].tolist() == [1.0, 0.0] + assert result.draws["other_mortgage"].tolist() == [1.0, 1.0] + assert result.draws["mortgage_debt"].tolist() == [2.0, 1.0] + # Totals are sums of the drawn components, never draws of their own. + assert result.draws["property_wealth"].tolist() == [8.0, 8.0] + assert result.draws["corporate_wealth"].tolist() == [5.0, 5.0] + assert result.draws["gross_financial_wealth"].tolist() == [11.0, 11.0] + assert result.draws["net_financial_wealth"].tolist() == [5.0, 5.0] fitted_targets = [name for _, targets in calls for name in targets] assert [record.fit_name for record in result.fit_weight_records] == [ f"uk_was_2018_20_wealth:{target}" for target in fitted_targets @@ -488,10 +673,56 @@ def fit_draw_next( assert {record.weight_kind for record in result.fit_weight_records} == {"explicit"} # One independent RNG root per segment, derived from the declared seed. assert seeds == list(module.was_wealth_segment_seeds(0)) - assert len(set(seeds)) == 4 + assert len(set(seeds)) == module.UK_WAS_CHAIN_SEGMENTS == 8 assert result.segment_seeds == tuple(seeds) +def test_stratified_target_refuses_a_donor_without_the_stratum() -> None: + raw = _raw_was() + raw.loc[0, ["Ten1R8", "HMortGR8", "TotMortR8"]] = [1, 0.0, 0.0] + donor = clean_was_household_table(raw) + recipient = recipient_predictors(_frame(), _FakeEngine()) + + with pytest.raises(ValueError, match="no household in the tenure stratum"): + impute_was_wealth(donor, recipient, seed=0, n_estimators=2) + + +def test_chain_declaration_drift_is_refused() -> None: + import microcosm.build.uk_runtime.was_wealth as module + + operation = _chain_operation() + operation["stratified_targets"] = {"mortgage_debt": ["owned_with_mortgage"]} + stage = SourceStageSpec.from_mapping( + { + "stage": "was_wealth", + "survey": "test", + "source": "test", + "grain": "household", + "artifacts": [], + "operations": [operation], + "outputs": list(UK_WAS_WEALTH_OUTPUT_COLUMNS), + } + ) + + with pytest.raises(ValueError, match="stratified_targets drifted"): + module._assert_chain_declaration(stage) + operation = _chain_operation() + operation["derived_totals"].pop("net_financial_wealth") + stage = SourceStageSpec.from_mapping( + { + "stage": "was_wealth", + "survey": "test", + "source": "test", + "grain": "household", + "artifacts": [], + "operations": [operation], + "outputs": list(UK_WAS_WEALTH_OUTPUT_COLUMNS), + } + ) + with pytest.raises(ValueError, match="derived_totals drifted"): + module._assert_chain_declaration(stage) + + def test_private_pension_wealth_split_preserves_the_old_corporate_wealth_identity() -> ( None ): @@ -528,40 +759,29 @@ def test_segment_seeds_are_distinct_and_deterministic() -> None: seeds = module.was_wealth_segment_seeds(0) # Golden pin: the production roots for the declared stage seed 0. Moving - # them is a spec-visible RNG change, never an accident. - assert seeds == (3757552657, 673228719, 3241444873, 3685993406) - assert len(seeds) == 4 - assert len(set(seeds)) == 4 + # them is a spec-visible RNG change, never an accident. The first four are + # the roots the four-segment chain used before microcosm#1063. + assert seeds == ( + 3757552657, + 673228719, + 3241444873, + 3685993406, + 1216546553, + 2078861726, + 2471122328, + 23012616, + ) + assert len(seeds) == 8 + assert len(set(seeds)) == 8 assert seeds[:3] == module.was_wealth_segment_seeds(0, segments=3) assert seeds == module.was_wealth_segment_seeds(0) assert seeds != module.was_wealth_segment_seeds(1) assert all(isinstance(seed, int) for seed in seeds) -def test_debt_segment_leaves_the_fourteen_e5_columns_byte_equal() -> None: - """The fourth chain segment cannot move the first three segments' draws. - - Runs the real chain on the hermetic H2 fixture donor twice, stopping after - the third segment and after the fourth, and asserts the fourteen E5 - columns are identical: the debt segment draws from the fourth child seed - and only appends columns. - """ - - from microcosm.build.uk_runtime.was_wealth import ( - UK_WAS_DEBT_OUTPUT_COLUMNS, - UK_WAS_WEALTH_OUTPUT_COLUMNS, - UK_WAS_WEALTH_PREDICTORS, - clean_was_household_table, - impute_was_wealth, - ) - from tools.graph_uk_spine_fixture import _was_donor - - donor = clean_was_household_table(_was_donor()) - rng = np.random.default_rng(685) - n = 48 - regions = np.resize( - np.asarray(["LONDON", "SCOTLAND", "WALES", "NORTH_EAST", "SOUTH_WEST"]), n - ) +def _synthetic_recipients(n: int = 80, seed: int = 1063) -> pd.DataFrame: + rng = np.random.default_rng(seed) + category = np.resize(np.asarray(UK_TENURE_CATEGORIES), n) recipient = pd.DataFrame( { "household_net_income": rng.uniform(5_000, 90_000, n), @@ -573,24 +793,135 @@ def test_debt_segment_leaves_the_fourteen_e5_columns_byte_equal() -> None: "capital_income": rng.uniform(0, 5_000, n), "num_bedrooms": rng.integers(1, 6, n), "council_tax": rng.uniform(800, 3_000, n), - "is_renting": rng.random(n) < 0.35, - "region": regions, - "has_mortgage_tenure": rng.random(n) < 0.3, } ) - assert list(recipient.columns[:11]) == list(UK_WAS_WEALTH_PREDICTORS) + for predictor in UK_WAS_TENURE_PREDICTORS: + recipient[predictor] = category == predictor.removeprefix("tenure_") + recipient["region"] = np.resize( + np.asarray(["LONDON", "SCOTLAND", "WALES", "NORTH_EAST", "SOUTH_WEST"]), n + ) + assert list(recipient.columns) == list(UK_WAS_WEALTH_PREDICTORS) + recipient[UK_WAS_TENURE_CATEGORY_COLUMN] = category + return recipient - three = impute_was_wealth(donor, recipient, seed=0, n_estimators=2, segments=3) - four = impute_was_wealth(donor, recipient, seed=0, n_estimators=2, segments=4) - e5_columns = [ - column - for column in UK_WAS_WEALTH_OUTPUT_COLUMNS - if column not in UK_WAS_DEBT_OUTPUT_COLUMNS - ] - assert list(three.draws.columns) == e5_columns - assert list(four.draws.columns) == list(UK_WAS_WEALTH_OUTPUT_COLUMNS) - pd.testing.assert_frame_equal(three.draws, four.draws.loc[:, e5_columns]) - assert three.segment_seeds == four.segment_seeds - assert len(three.fit_weight_records) == len(e5_columns) - assert len(four.fit_weight_records) == len(UK_WAS_WEALTH_OUTPUT_COLUMNS) +def _fixture_donor() -> pd.DataFrame: + from tools.graph_uk_spine_fixture import _was_donor + + return clean_was_household_table(_was_donor()) + + +def test_later_segments_leave_the_earlier_draws_byte_equal() -> None: + """A later chain segment cannot move an earlier segment's draws. + + Runs the real chain on the hermetic H2 fixture donor, stopping after each + segment in turn: every segment draws from its own child seed and only + appends columns. + """ + + donor = _fixture_donor() + recipient = _synthetic_recipients(48, seed=685) + + full = impute_was_wealth(donor, recipient, seed=0, n_estimators=2) + assert list(full.draws.columns) == _DRAW_COLUMNS + for segments in range(1, 8): + partial = impute_was_wealth( + donor, recipient, seed=0, n_estimators=2, segments=segments + ) + pd.testing.assert_frame_equal( + partial.draws, full.draws.loc[:, list(partial.draws.columns)] + ) + assert partial.segment_seeds == full.segment_seeds + # One fit record per drawn target; the derived totals are never fitted. + derived = {*UK_WAS_DERIVED_TOTALS, "net_financial_wealth"} + assert len(full.fit_weight_records) == len(_DRAW_COLUMNS) - len(derived) + + +def test_real_chain_holds_the_tenure_rules_and_the_identities() -> None: + donor = _fixture_donor() + recipient = _synthetic_recipients() + category = recipient[UK_WAS_TENURE_CATEGORY_COLUMN].to_numpy() + + draws = impute_was_wealth(donor, recipient, seed=0, n_estimators=3).draws + + owner = np.isin(category, ["owned_outright", "owned_with_mortgage"]) + mortgaged = category == "owned_with_mortgage" + # The two structural columns live in their tenure stratum and nowhere else; + # every fixture donor of the stratum holds a positive value. + assert (draws.loc[~owner, "main_residence_value"] == 0.0).all() + assert (draws.loc[owner, "main_residence_value"] > 0.0).all() + assert (draws.loc[~mortgaged, "main_residence_mortgage"] == 0.0).all() + assert (draws.loc[mortgaged, "main_residence_mortgage"] > 0.0).all() + # The other mortgages are carried on every tenure and mortgage_debt is + # the sum of the two, so a renter's buy-to-let keeps its mortgage. + assert (draws["other_mortgage"] >= 0.0).all() + assert ( + draws["mortgage_debt"] + == draws["main_residence_mortgage"] + draws["other_mortgage"] + ).all() + assert wealth_identity_violations(draws) == { + "property_wealth_violation_rows": 0, + "corporate_wealth_violation_rows": 0, + "gross_financial_wealth_violation_rows": 0, + "mortgage_debt_violation_rows": 0, + "net_financial_wealth_violation_rows": 0, + } + + capped, fired = cap_derived_totals_to_donor_range(draws, donor) + clipped = support_clip_to_donor(capped, donor) + # The cap keeps every total inside the donor's range, so the support clip + # moves nothing and the identities survive it. + assert all( + column["clipped_low_rows"] == 0 and column["clipped_high_rows"] == 0 + for column in clipped.receipt.evidence()["columns"].values() + ) + assert set(wealth_identity_violations(clipped.clipped).values()) == {0} + assert set(fired) == { + "property_wealth_capped_rows", + "corporate_wealth_capped_rows", + "gross_financial_wealth_capped_rows", + "mortgage_debt_capped_rows", + "net_financial_wealth_capped_high_rows", + "net_financial_wealth_capped_low_rows", + } + + receipt = tenure_coherence_receipt( + donor=donor, + recipient_category=recipient[UK_WAS_TENURE_CATEGORY_COLUMN], + recipient_draws=clipped.clipped, + recipient_weights=np.ones(len(recipient)), + ) + assert receipt["main_residence_mortgage_off_mortgaged_tenure_rows"] == 0 + assert receipt["main_residence_value_off_owner_tenure_rows"] == 0 + assert receipt["owner_rows_without_main_residence_value"] == 0 + shares = receipt["positive_share_by_tenure"]["main_residence_value"] + assert shares["social_rent"] == {"donor": 0.0, "recipient": 0.0} + assert shares["owned_outright"] == {"donor": 1.0, "recipient": 1.0} + # The fixture donor carries mortgages on other property off a mortgaged + # tenure; the chain carries them too, and the receipt says how much on + # each side. + assert receipt["donor_mortgage_debt_mass_off_mortgaged_tenure"] > 0.0 + assert receipt["mortgage_debt_rows_off_mortgaged_tenure"] >= 0 + assert receipt["donor_other_property_mortgage_mass_off_mortgaged_tenure"] == ( + pytest.approx(receipt["donor_mortgage_debt_mass_off_mortgaged_tenure"]) + ) + assert receipt["donor_main_residence_mortgage_mass_off_mortgaged_tenure"] == 0.0 + + +def test_total_cap_takes_the_excess_from_the_remainder_first() -> None: + donor = clean_was_household_table(_raw_was()) + draws = donor.loc[[0], _DRAW_COLUMNS].reset_index(drop=True) + # Push the property total 500 past the donor's largest (104,100) with + # donor-plausible components: the remainder (90) goes first, then the + # buildings, and the main residence is never touched. + draws.loc[0, "non_residential_property_value"] += 500.0 + draws.loc[0, "property_wealth"] += 500.0 + + capped, fired = cap_derived_totals_to_donor_range(draws, donor) + + assert fired["property_wealth_capped_rows"] == 1 + assert capped.loc[0, "property_wealth"] == 104_100.0 + assert capped.loc[0, "other_property_value"] == 0.0 + assert capped.loc[0, "non_residential_property_value"] == 3_090.0 + assert capped.loc[0, "main_residence_value"] == 100_000.0 + assert set(wealth_identity_violations(capped).values()) == {0} diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_wealth_resources.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_wealth_resources.py index c21728107..5f27278bc 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_wealth_resources.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_wealth_resources.py @@ -109,7 +109,7 @@ def test_support_bounds_tool_rounds_synthetic_donor_outward(tmp_path: Path) -> N rows = { "R8xshhwgt": [1, 1], "DVLUKValR8_sum": [10, 100], - "DVPropertyR8": [20, 200], + "DVPropertyR8": [104100, 6500], "DVFESHARESR8_aggr": [1, 2], "DVFShUKVR8_aggr": [3, 4], "DVIISAVR8_aggr": [5, 6], @@ -125,14 +125,16 @@ def test_support_bounds_tool_rounds_synthetic_donor_outward(tmp_path: Path) -> N "DVGIEMPR8_AGGR": [17, 18], "HBedRmR8": [3, 4], "GORR8": [8, 12], - "DVPriRntR8": [1, 2], + "DVPriRntR8": [-9, 2], "CTAmtR8": [1000, 1200], - "HFINWNTR8_Sum": [-5, 50], - "HFINWNTR8_exSLC_Sum": [20, 40], - "HFINWR8_SUM": [30, 40], + "HFINWNTR8_Sum": [-4480, -1356], + "HFINWNTR8_exSLC_Sum": [520, 644], + "HFINWR8_SUM": [530, 644], "HMortGR8": [1000, 0], + "TotMortR8": [1000, 0], + "OthMortR8_sum": [0, 0], "Ten1R8": [2, 4], - "DVhvalueR8": [100000, 200000], + "DVhvalueR8": [100000, 0], "DVHseValR8_sum": [1000, 2000], "DVBlDValR8_sum": [3000, 4000], "DVTotinc_bhcR8": [50000, 60000], @@ -148,6 +150,6 @@ def test_support_bounds_tool_rounds_synthetic_donor_outward(tmp_path: Path) -> N payload = _support_tool().build_support_bounds(tab) assert payload["bounds"]["owned_land"] == [10.0, 100.0] - assert payload["bounds"]["net_financial_wealth"] == [-5.0, 50.0] + assert payload["bounds"]["net_financial_wealth"] == [-5000.0, -1000.0] assert payload["bounds"]["mortgage_debt"] == [0.0, 1000.0] assert payload["bounds"]["consumer_debt"] == [0.0, 10.0] diff --git a/packages/microcosm-build/tests/engine_free/uk/test_uk_weighted_integrity.py b/packages/microcosm-build/tests/engine_free/uk/test_uk_weighted_integrity.py index e7eb15ea3..49b860dce 100644 --- a/packages/microcosm-build/tests/engine_free/uk/test_uk_weighted_integrity.py +++ b/packages/microcosm-build/tests/engine_free/uk/test_uk_weighted_integrity.py @@ -659,11 +659,30 @@ def test_committed_exclusion_registers_load() -> None: ) assert set(input_mass) == {"efrs-post-calibration"} - # Both former entries retired 2026-09-21: the first national candidate - # measured charitable_investment_gifts and owned_land inside the fence, - # and the gate fails a stale exclusion by design. - assert input_mass["efrs-post-calibration"] == {} - assert qrf_tail == {} + # The 2026-09-21 retirements (charitable_investment_gifts and owned_land + # measured inside the fence on the first national candidate) stand; the + # certifier rehearsal on the 2026-09-30 build earned three entries + # (microcosm#1063 c9): the two FRS education inputs the enhanced-FRS + # reference barely carries, and the sparse charity-gifts leaf back at + # the concentration threshold. The gate fails any of them as stale the + # day a build brings the column inside the fence, which the 2 October + # final build did for adult_ema (+74% against the reference): its entry + # is retired. The 3 October final build's certifier earned three more, + # signed 2026-10-04: the FRS access-fund outlier at the weight bound + # (expiring with the owed period rule) and the SPI channel's two + # legacy-benefit leaves. + assert set(input_mass["efrs-post-calibration"]) == { + "access_fund", + "dfe_education_spending", + "jsa_income_reported", + "working_tax_credit_reported", + } + assert set(qrf_tail) == {"charitable_investment_gifts"} + for record in (*input_mass["efrs-post-calibration"].values(), *qrf_tail.values()): + assert record.approved_by == "juaristi22" + assert record.approved_on in {"2026-10-02", "2026-10-04"} + assert record.expires_on in {"2027-04-02", "2027-01-04", "2027-04-04"} + assert "microcosm#1063" in record.adjudication assert uk_default_input_mass_reviewed_exclusions() is ( uk_default_input_mass_reviewed_exclusions() ) diff --git a/packages/microcosm-data/src/microcosm/data/contract.py b/packages/microcosm-data/src/microcosm/data/contract.py index 2fdccd6e2..2f9a3775b 100644 --- a/packages/microcosm-data/src/microcosm/data/contract.py +++ b/packages/microcosm-data/src/microcosm/data/contract.py @@ -416,13 +416,13 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: # fingerprint derives from the manifest digest. Editing the spec moves all # three here in the same reviewed change. _UK_GATE_BATTERY_POLICY_SHA256 = ( - "e6ad5f71e55a9c639026eb3794753ad9da0f2d1633949166956fc2f13be4d5c2" + "82a92dfa7ba4e4ce81cfd44d7b89737cf3cce50d69fd2d6bbeeae0ca15a0f8d5" ) _UK_GATE_BATTERY_GATES_MANIFEST_SHA256 = ( - "9ade6a80be484b410c47bf93191bd17cce01542d3d9032ba672d38bd23cdb98c" + "c117864fac6df5bb7f37f466c068b8dd9dd032b694cf3d521c51b5dca71b5865" ) _UK_GATE_BATTERY_SPEC_FINGERPRINT = ( - "7830c47cab40144f503e2e377391fe2a0f008b0dd20b88a584f750acdb8127ae" + "7ff71ed911b840c01a1b6395de37a04fdf95498444bd692b32d89f3b1767a128" ) #: Spec entry id -> the legacy gate name whose observable detail checks #: apply unchanged (the battery re-keys the report by entry id; the gate @@ -468,6 +468,7 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: ), "uk_stage_was_wealth_support": ("stage_health", "transferred"), "uk_stage_was_lisa_support": ("stage_health", "transferred"), + "uk_stage_was_wealth_coherence": ("stage_health", "transferred"), "uk_stage_nts_bus_travel_support": ("stage_health", "transferred"), "uk_stage_nts_bus_travel_facts": ("stage_health", "transferred"), "uk_stage_uc_deduction_attributes": ("stage_health", "transferred"), @@ -487,8 +488,10 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: "transferred", ), "uk_stage_pension_credit_take_up": ("stage_health", "transferred"), + "uk_stage_child_benefit_take_up": ("stage_health", "transferred"), "uk_stage_cgt_incidence_clone_mass": ("stage_health", "transferred"), "uk_stage_cgt_support_split_mass": ("stage_health", "transferred"), + "uk_stage_cgt_residential_split_mass": ("stage_health", "transferred"), "uk_stage_spi_income_band_donors_support": ("stage_health", "transferred"), "uk_stage_hmrc_cgt_gains_spine_summary": ( "stage_health", @@ -569,6 +572,7 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: "uk_input_mass_parity", "uk_stage_was_wealth_support", "uk_stage_was_lisa_support", + "uk_stage_was_wealth_coherence", "uk_stage_nts_bus_travel_support", "uk_stage_nts_bus_travel_facts", "uk_stage_uc_deduction_attributes", @@ -582,8 +586,10 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: "uk_stage_spi_support_channel_mass", "uk_stage_hmrc_spi_income_spine_identity", "uk_stage_pension_credit_take_up", + "uk_stage_child_benefit_take_up", "uk_stage_cgt_incidence_clone_mass", "uk_stage_cgt_support_split_mass", + "uk_stage_cgt_residential_split_mass", "uk_stage_spi_income_band_donors_support", "uk_stage_hmrc_cgt_gains_spine_summary", "uk_stage_hmrc_cgt_asset_type_spine_summary", @@ -600,7 +606,7 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: # canonical hash; this pins the wrapped digest so the entry's evidence line # still binds the enhanced-FRS incumbent totals. _UK_GATE_BATTERY_INPUT_MASS_EVIDENCE_SHA256 = ( - "17545916b6926c77e9f8fc90876266cc3f8e4a381079bafc8d1c63fa8df43c04" + "8829fb83db6a16482be80cbe1b1da125608448b5a34b9cbb0561aac32a870447" ) # The degenerate binding's evidence payload digests the resolved exclusion # records; for a release that must be the committed register, so its digest @@ -775,7 +781,9 @@ def _uk_line_cut_tag_re(release_id: str) -> re.Pattern[str] | None: "uk_ons_household_type_enum_domain", "uk_stage_age_tail_targets", "uk_stage_cgt_support_split_mass", + "uk_stage_cgt_residential_split_mass", "uk_stage_cgt_incidence_anchor_composition", + "uk_stage_child_benefit_take_up", "uk_stage_spi_income_band_donors_support", "uk_stage_cgt_incidence_clone_mass", "uk_stage_etb_services_support", @@ -797,6 +805,7 @@ def _uk_line_cut_tag_re(release_id: str) -> re.Pattern[str] | None: "uk_stage_student_loans_realization", "uk_stage_uc_deduction_attributes", "uk_stage_was_lisa_support", + "uk_stage_was_wealth_coherence", "uk_stage_was_wealth_support", } ), @@ -839,10 +848,10 @@ def _uk_line_cut_tag_re(release_id: str) -> re.Pattern[str] | None: _UK_CERTIFICATION_PART_DIGESTS: Mapping[str, Mapping[str, str]] = { "spine": { "gates_manifest_sha256": ( - "c6fce886d43fc981a1d65fbe24decca84bca0988d98f0bbb054758e7d8bbe0e7" + "9708ee040b34e8dbecdb88691079f00682ccf9ca20599f07342b4c33c2e56f40" ), "policy_sha256": ( - "68399e4cb348b7cb7f1ac07e9b03cb0027cbc94f056b412bd1db7fdf49e7bff8" + "b22f79584c16ab81fbfa131c19c16f5bc5c1e484003c73a695abf3446ff6175b" ), }, "calibration_seam": { @@ -855,10 +864,10 @@ def _uk_line_cut_tag_re(release_id: str) -> re.Pattern[str] | None: }, "release_cut": { "gates_manifest_sha256": ( - "60e37d4ac2ed3725e1a6d89eeb939513486efddd6ac7f6d689857485bd9a1ef1" + "4eed0ed4dc6785f7f4611e9af96a84594e73806eccc6eaf6b76f85630b378f35" ), "policy_sha256": ( - "264084717c05c80509795a69299fe60f4513e821f443e0c3f2126911c8ce3605" + "a673ec3efe1be53567ddc12b344e44ab872e05a6ebcea2862b1762952901ce78" ), }, } diff --git a/packages/microcosm-data/tests/engine_free/shared/test_contract.py b/packages/microcosm-data/tests/engine_free/shared/test_contract.py index a6660b969..7412ea5e5 100644 --- a/packages/microcosm-data/tests/engine_free/shared/test_contract.py +++ b/packages/microcosm-data/tests/engine_free/shared/test_contract.py @@ -89,7 +89,38 @@ "SPI-channel-exclusive columns are comparable only through per-reference " "reviewed exclusions." ) -UK_INPUT_MASS_REVIEWED_EXCLUSIONS: dict[str, dict[str, str]] = {} +# The committed per-reference register (microcosm#1063 c9): the policy payloads +# sealed into the input-mass evidence digest. +UK_INPUT_MASS_REVIEWED_EXCLUSIONS: dict[str, dict[str, str]] = { + "access_fund": { + "reason": "One FRS household carries 98% of the candidate's access-fund mass (GBP 1.01bn against the incumbent's GBP 160m, +535%): sernum 14443's 19-year-old reports an access fund of GBP 4,500 with FRS period code 5 (calendar month), which the FRS weeklyises to GBP 1,035.62 (ACCSSAMT) and the spine annualises to GBP 54,037, eleven times the next largest FRS value; the calibration takes the household and its two capital-gains copies to the 10x weight bound. The value is almost certainly an annual or termly award under the wrong period code; the input stays on the release surface until the period rule is settled in the spine, and the exclusion expires with it.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (PR #1089 release-cut certifier on the 2026-10-03 final build; ruling 2026-10-04 in session: the deferrals needed to build and publish the national dataset are signed, each with its diagnosis)", + "approved_on": "2026-10-04", + "expires_on": "2027-01-04", + }, + "dfe_education_spending": { + "reason": "The candidate carries DfE education spending through the build (GBP 98.8bn against the incumbent's GBP 52.8m, +187,061%): the enhanced-FRS reference carries almost none of this input, so the comparison measures the reference's omission, not the candidate's. The input stays on the release surface and in the coverage manifest.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 c9 (certifier rehearsal R5 on the 2026-09-30 build, 2026-10-01; ruling 2026-10-02 in session)", + "approved_on": "2026-10-02", + "expires_on": "2027-04-02", + }, + "jsa_income_reported": { + "reason": "The candidate carries GBP 272m of reported income-based JSA against the incumbent's GBP 23m (+1,058%): the enhanced-FRS reference holds only the FRS respondents' amounts, while the spine's SPI synthetic channel receives this legacy-benefit leaf from the stage-2 QRF at FRS rates (76 of the 196 nonzero rows and most of the mass) and a few rows sit at the calibration's weight bound. Neither is new to this build: the 2026-09-30 main candidate stood at +428% on the same column. The comparison measures the SPI channel's imputation of a legacy benefit, which is a design question for the channel, not a mass the candidate lost or invented.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (PR #1089 release-cut certifier on the 2026-10-03 final build; ruling 2026-10-04 in session: the deferrals needed to build and publish the national dataset are signed, each with its diagnosis)", + "approved_on": "2026-10-04", + "expires_on": "2027-04-04", + }, + "working_tax_credit_reported": { + "reason": "The candidate carries GBP 625m of reported Working Tax Credit against the incumbent's GBP 74m (+747%): the enhanced-FRS reference holds only the FRS respondents' amounts, while the spine's SPI synthetic channel receives this legacy-benefit leaf from the stage-2 QRF at FRS rates (274 of the 372 nonzero rows) and a few rows sit at the calibration's weight bound. The 2026-09-30 main candidate stood at +421% on the same column. Tax credits closed in April 2025, so the SPI channel's treatment of the leaf is a design question for the channel, not a mass the candidate lost or invented.", + "approved_by": "juaristi22", + "adjudication": "microcosm#1063 (PR #1089 release-cut certifier on the 2026-10-03 final build; ruling 2026-10-04 in session: the deferrals needed to build and publish the national dataset are signed, each with its diagnosis)", + "approved_on": "2026-10-04", + "expires_on": "2027-04-04", + }, +} GIT_COMMIT = "5fa48f07436a806ad75ff76fd22cfb8613bddbe0" DATASET_SHA = "d" * 64 CALIBRATION_SHA = "a" * 64 @@ -117,19 +148,19 @@ def _trusted_terminal_gate_signing_key(monkeypatch) -> None: UK_GATE_BATTERY_PRODUCER = "microcosm.build.gate_battery" UK_GATE_BATTERY_SIGNING_KEY_ENV = "MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY" UK_GATE_BATTERY_POLICY_SHA256 = ( - "e6ad5f71e55a9c639026eb3794753ad9da0f2d1633949166956fc2f13be4d5c2" + "82a92dfa7ba4e4ce81cfd44d7b89737cf3cce50d69fd2d6bbeeae0ca15a0f8d5" ) UK_GATE_BATTERY_GATES_MANIFEST_SHA256 = ( - "9ade6a80be484b410c47bf93191bd17cce01542d3d9032ba672d38bd23cdb98c" + "c117864fac6df5bb7f37f466c068b8dd9dd032b694cf3d521c51b5dca71b5865" ) UK_GATE_BATTERY_SPEC_FINGERPRINT = ( - "7830c47cab40144f503e2e377391fe2a0f008b0dd20b88a584f750acdb8127ae" + "7ff71ed911b840c01a1b6395de37a04fdf95498444bd692b32d89f3b1767a128" ) UK_GATE_BATTERY_DEGENERATE_EVIDENCE_SHA256 = ( "6f0243bcda09dad26945376230c44ec3cf55d4e417c3a25e29bae8c59bc1a69d" ) UK_GATE_BATTERY_INPUT_MASS_EVIDENCE_SHA256 = ( - "17545916b6926c77e9f8fc90876266cc3f8e4a381079bafc8d1c63fa8df43c04" + "8829fb83db6a16482be80cbe1b1da125608448b5a34b9cbb0561aac32a870447" ) #: Spec entry id -> (neutral gate name, phase, legacy detail-schema name). UK_GATE_BATTERY_ENTRIES = { @@ -151,6 +182,7 @@ def _trusted_terminal_gate_signing_key(monkeypatch) -> None: ), "uk_stage_was_wealth_support": ("stage_health", "transferred", None), "uk_stage_was_lisa_support": ("stage_health", "transferred", None), + "uk_stage_was_wealth_coherence": ("stage_health", "transferred", None), "uk_stage_uc_deduction_attributes": ( "stage_health", "transferred", @@ -180,11 +212,17 @@ def _trusted_terminal_gate_signing_key(monkeypatch) -> None: None, ), "uk_stage_pension_credit_take_up": ("stage_health", "transferred", None), + "uk_stage_child_benefit_take_up": ("stage_health", "transferred", None), "uk_stage_cgt_support_split_mass": ( "stage_health", "transferred", None, ), + "uk_stage_cgt_residential_split_mass": ( + "stage_health", + "transferred", + None, + ), "uk_stage_cgt_incidence_clone_mass": ( "stage_health", "transferred", @@ -1232,10 +1270,12 @@ def _gate_battery_payload( "hmrc_spi_income_spine", "uc_capital_coherence", "pension_credit_take_up", + "child_benefit_take_up", "uc_deduction_attributes", "cgt_support_split", "cgt_incidence_clone", "hmrc_cgt_gains_spine", + "cgt_residential_split", "salary_sacrifice", "student_loans", "age_tail", @@ -1243,6 +1283,7 @@ def _gate_battery_payload( stage_health_stages = { "uk_stage_was_wealth_support": "was_wealth", "uk_stage_was_lisa_support": "was_lisa", + "uk_stage_was_wealth_coherence": "was_wealth", "uk_stage_uc_deduction_attributes": "uc_deduction_attributes", "uk_stage_lcfs_consumption_support": "lcfs_consumption", "uk_stage_lcfs_consumption_energy_rake": "lcfs_consumption", @@ -1256,7 +1297,9 @@ def _gate_battery_payload( "uk_stage_spi_support_channel_mass": "spi_support_channel", "uk_stage_hmrc_spi_income_spine_identity": "hmrc_spi_income_spine", "uk_stage_pension_credit_take_up": "pension_credit_take_up", + "uk_stage_child_benefit_take_up": "child_benefit_take_up", "uk_stage_cgt_support_split_mass": "cgt_support_split", + "uk_stage_cgt_residential_split_mass": "cgt_residential_split", "uk_stage_cgt_incidence_clone_mass": "cgt_incidence_clone", "uk_stage_spi_income_band_donors_support": "spi_income_band_donors", "uk_stage_hmrc_cgt_gains_spine_summary": "hmrc_cgt_gains_spine", diff --git a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/cgt_asset_type_facts.json b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/cgt_asset_type_facts.json index a2cf4dbb7..8655dd6e6 100644 --- a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/cgt_asset_type_facts.json +++ b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/cgt_asset_type_facts.json @@ -2,70 +2,70 @@ "resource": "synthetic-cgt-asset-type.json", "resource_sha256": "synthetic", "source_commit": "synthetic", - "table4_all_gains": 86140764.44464564, + "table4_all_gains": 6803877.8232513005, "table4_all_tax": 0.0, - "table4_all_taxpayers": 194.81827357660933, + "table4_all_taxpayers": 16.094983343942488, "table4_bands": [ { - "gains": 109881.50549018131, + "gains": 8469.81629743428, "lower_bound": 0, "tax": null, - "taxpayers": 24.254564087265, + "taxpayers": 2.0315589349981042, "upper_bound": 10000 }, { - "gains": 292228.7986384038, + "gains": 21793.660633989297, "lower_bound": 10000, "tax": null, - "taxpayers": 16.58439246485428, + "taxpayers": 1.4192719822325444, "upper_bound": 25000 }, { - "gains": 669730.4631223604, + "gains": 41822.32276068652, "lower_bound": 25000, "tax": null, - "taxpayers": 16.663812046135067, + "taxpayers": 1.1984100692769308, "upper_bound": 50000 }, { - "gains": 1661458.7134747333, + "gains": 98900.8658246953, "lower_bound": 50000, "tax": null, - "taxpayers": 23.49154791320917, + "taxpayers": 1.3284701286752632, "upper_bound": 100000 }, { - "gains": 3395372.364664346, + "gains": 315094.2653986191, "lower_bound": 100000, "tax": null, - "taxpayers": 16.459524279304198, + "taxpayers": 2.0148156202095766, "upper_bound": 250000 }, { - "gains": 6803922.927752389, + "gains": 853577.3892905796, "lower_bound": 250000, "tax": null, - "taxpayers": 16.407377522325543, + "taxpayers": 2.300508382822492, "upper_bound": 500000 }, { - "gains": 16060242.575711947, + "gains": 985119.1545275399, "lower_bound": 500000, "tax": null, - "taxpayers": 23.809128167724772, + "taxpayers": 1.3228478772098182, "upper_bound": 1000000 }, { - "gains": 57147927.09579129, + "gains": 4479100.348517757, "lower_bound": 1000000, "tax": null, - "taxpayers": 57.14792709579129, + "taxpayers": 4.479100348517757, "upper_bound": null } ], - "table4_individuals_gains": 86140764.44464564, + "table4_individuals_gains": 6803877.8232513005, "table4_individuals_tax": 0.0, - "table4_individuals_taxpayers": 194.81827357660933, + "table4_individuals_taxpayers": 16.094983343942488, "table4_trusts_gains": 0.0, "table4_trusts_tax": 0.0, "table7_total_disposals": 2743095.2380952383, @@ -114,12 +114,12 @@ "proceeds": 9000000000.0 } ], - "table8a_disposals_total": 366.328632, - "table8a_gains_total": 572565131.7814949, - "table8a_tax_total": 114513026.35629898, - "table8a_taxpayers_total": 333.0260290909091, - "table8b_all_gains": 572565131.7814949, - "table8b_all_taxpayers": 333.0260290909091, - "table8b_individuals_gains": 572565131.7814949, - "table8b_individuals_taxpayers": 333.0260290909091 + "table8a_disposals_total": 30.35054002000583, + "table8a_gains_total": 36521680.19444137, + "table8a_tax_total": 7304336.038888275, + "table8a_taxpayers_total": 27.591400018187116, + "table8b_all_gains": 36521680.19444137, + "table8b_all_taxpayers": 27.591400018187116, + "table8b_individuals_gains": 36521680.19444137, + "table8b_individuals_taxpayers": 27.591400018187116 } diff --git a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/fixture.json b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/fixture.json index ef551bb80..7e28c5a62 100644 --- a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/fixture.json +++ b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/fixture.json @@ -249,6 +249,87 @@ "stage": "cgt_incidence_clone", "survey": "Family Resources Survey 2024-25, SPI synthetic support, and Advani-Summers capital-gains incidence" }, + "cgt_residential_split": { + "artifacts": [ + { + "dependency": "policyengine-uk>=2.100.0 via microcosm-build[uk]", + "dependency_discipline": "deferred inside uk_cgt_policy_parameters; the base package never imports policyengine-uk at import time", + "instant_rule": "raw dated parameter files evaluated at 1 June of the build period's tax year", + "kind": "versioned_parameter_tree", + "parameters": [ + "gov.hmrc.cgt.annual_exempt_amount" + ], + "role": "policy_parameters", + "runtime_sha256_required": false + }, + { + "format": "json", + "kind": "public_aggregate_reference", + "resource": "hmrc_cgt_asset_type_facts.json", + "role": "cgt_asset_type_facts", + "runtime_sha256_required": true + } + ], + "grain": "household", + "nonnegative_outputs": [ + "cgt_residential_probability", + "capital_gains_residential_property" + ], + "notes": "Structural stage after the incidence anchor and before the asset-type stage (microcosm#1063). The residential-property flag used to be realised by a weighted systematic walk, one whole flag per row, whose gains error is bounded by one row's stake: at the top of the gains distribution one stake is several times a band's expected residential gains, so the realised gains sat short of expectation by the lottery expectation of the giant stakes or over it by one of them. This stage solves the same logistic (intercept to the Table 8a 2024-25 taxpayers on the individuals basis, slope to the gains, with the stock shift) and carries the probability as weight instead: every household with liable gainers becomes 2^k arms at product weights, one per subset of residential gainers, the arm with no residential gainer keeping the household's ids and arm j taking every entity id plus j times the id multiplier, the lineage rule the executor, the geography identity kernel and the identity tool read for the support split and the clone. At design weights the residential count and gains equal the solved expectations in every gain band, with no draw, no offset and no seed; the calibration, which binds hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains, decides between the arms afterwards as it does between the incidence clone's halves. Every household's mass and composition and the total household mass are conserved; the receipt is gated at stage time.", + "operations": [ + { + "artifact_role": "cgt_asset_type_facts", + "fail_on_mismatch": true, + "feed_pin": "chronicle_feed.json", + "kind": "verify_vendored_fact_resource", + "mapped_build_period": 2024, + "period_mapping": "published_tax_year_equals_build_period", + "record_sets": [ + "hmrc.cgt_table8_2026.table8a.ty2024", + "hmrc.cgt_table8_2026.table8b.ty2024" + ], + "require_before_source_read": true, + "resource": "hmrc_cgt_asset_type_facts.json", + "runtime_sha256_required": true, + "source_vintage": "2024-25" + }, + { + "basis_assumption": "trusts hold the same share of the Self Assessment component as of the UK Property service, a residential gainer's whole net gain is attributed to residential property, and the gainers of one household are residential independently of each other", + "conservation": "exact_total", + "count_target": "Table 8a 2024-25 total taxpayers reporting residential property disposals times the Table 8b 2024-25 individuals/all taxpayer share on the UK Property service", + "declared_factor": 1, + "flag_column": "household_is_cgt_residential_clone", + "gainer_order": "person_id ascending within the household", + "gains_target": "Table 8a 2024-25 total residential property gains times the Table 8b 2024-25 individuals/all gains share on the UK Property service", + "id_remapping": "id_multiplier_for_values; arm j takes every entity id + j x multiplier, bit m of j marking the household's m-th liable gainer residential", + "index_column": "cgt_residential_clone_index", + "kind": "split_liable_gainers_by_residential_probability", + "maximum_liable_gainers_per_household": 3, + "model": "logistic probability in centred log gains with a stock shift, p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean log gain of the population, s the stock log-odds and h one where the stock signal holds, zero otherwise", + "output_column": "capital_gains_residential_property", + "output_semantics": "the person's capital_gains on an arm where the person is residential, 0 otherwise", + "parameter_solver": "nested bisection; the intercept matches the expected weighted count at each trial slope, the slope matches the expected weighted gains", + "population": "persons with net capital gains above the annual exempt amount (the national taxpayer proxy)", + "probability_column": "cgt_residential_probability", + "realization": "probability carried as weight, no draw: a household carrying k liable gainers becomes 2^k arms, one per subset of residential gainers, at the household's weight times the product of p over the residential gainers and 1 - p over the others; the arm with no residential gainer keeps the household's ids; at design weights the residential count and gains equal the solved expectations sum(p w) and sum(p w g) in every gain band", + "reason": "Capital-gains residential split divides every household carrying liable gainers into arms weighted by the solved residential probabilities, one arm per subset of residential gainers; every household's mass and the total household mass are conserved.", + "stock_log_odds": 1.6, + "stock_signal": "household other residential property value above zero, or the person's property income above zero", + "values": "every other column copied unchanged; no draw, no seed, no salt", + "weight_kind_out": "importance" + } + ], + "outputs": [ + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", + "cgt_residential_probability", + "capital_gains_residential_property" + ], + "rewrites": [], + "source": "https://www.gov.uk/government/statistics/capital-gains-tax-statistics", + "stage": "cgt_residential_split", + "survey": "HMRC Capital Gains Tax statistics 2026 release, Table 8 (UK residential property disposals, 2024-25, administrative) on the amounts the Table 3 redraw placed" + }, "cgt_support_split": { "artifacts": [ { @@ -342,6 +423,79 @@ "stage": "cgt_support_split", "survey": "Family Resources Survey 2024-25, SPI synthetic support and HMRC Capital Gains Tax statistics Table 3 (2024-25 individuals by size of gain and taxable income)" }, + "child_benefit_take_up": { + "artifacts": [ + { + "format": "json", + "kind": "public_aggregate_reference", + "resource": "hmrc_child_benefit_statistics.json", + "role": "child_benefit_statistics", + "runtime_sha256_required": true + } + ], + "grain": "benunit", + "nonnegative_outputs": [], + "notes": "Runs after the SPI income chain, UC capital coherence and the Pension Credit redraw, before CGT cloning (microcosm#1063). frs_take_up draws would_claim_child_benefit early at one flat rate over every benefit unit with the reporters on top, and child_benefit_opts_out at a flat rate whatever the unit's income; this stage redraws both. One temporary engine materialization gives each person's eligibility as a child or qualifying young person and each person's adjusted net income (consumed only). A family claims for all its children or none, so the draw is by family, keyed on the age of its eldest eligible child: a reporter claims, and the other families claim at rates solved from the eldest age down so that the claimed share of the eligible children of each single year of age is HMRC's published one (Table 15, May 2025, opted-out families included), weighted at the stage's household weights. The published share of registered families that opted out of payment (August 2025) of the claiming families then opts out, drawn among claiming families that do not report receipt and whose highest adjusted net income is at or above the income at which the High Income Child Benefit Charge takes the whole benefit, then inside the taper for any remainder. The engine pays on would_claim_child_benefit alone, so the stage stores claims-and-not-opted-out there and the opt-out in child_benefit_opts_out; a benefit unit with no eligible child keeps its early draw and is never opted out.", + "operations": [ + { + "consumed_only": true, + "kind": "materialize_rules_engine_predictors", + "predictors": [ + "is_child_or_qualifying_young_person_for_child_benefit", + "adjusted_net_income" + ] + }, + { + "aggregates": { + "child_benefit_reported_anchor": "child_benefit_reported" + }, + "consumed_only": true, + "kind": "aggregate_person_to_benunit", + "method": "any_positive" + }, + { + "anchor": "child_benefit_reported_anchor", + "draw": "would_claim_child_benefit", + "eligibility": "is_child_or_qualifying_young_person_for_child_benefit", + "family_key": "age of the benefit unit's eldest eligible child", + "kind": "assign_family_claims_by_child_age", + "output": "would_claim_child_benefit", + "rate_measure": "claim_rate", + "rate_period": "2025-05", + "resource": "hmrc_child_benefit_statistics.json", + "rule": "a reporter claims; a non-reporting family whose eldest eligible child is aged g claims at r_g = (t_g * C_g - R_g - sum over k > g of N_gk * r_k) / N_gg, weighted, solved from the eldest age down and clipped to [0, 1]", + "seed": 0, + "weight_mapping": "household_to_benunit" + }, + { + "caseload_period": "2025-08", + "charge_income": "adjusted_net_income", + "charge_parameters": "gov.hmrc.income_tax.charges.CB_HITC", + "denominator_measure": "families_registered", + "draw": "child_benefit_opts_out", + "kind": "assign_opt_out_by_charge_income", + "numerator_measure": "families_opted_out_claimant_all", + "output": "child_benefit_opts_out", + "payment_rule": "would_claim_child_benefit = claims and not opted out on families with an eligible child; a benefit unit without one keeps its early draw", + "pools": [ + "fully_charged", + "taper" + ], + "resource": "hmrc_child_benefit_statistics.json", + "rule": "published opted-out share of registered families, of the claiming families; drawn among claiming non-reporting families with the highest adjusted net income at or above the full-charge income, then, for any remainder, among those inside the taper", + "seed": 0, + "weight_mapping": "household_to_benunit" + } + ], + "outputs": [], + "rewrites": [ + "would_claim_child_benefit", + "child_benefit_opts_out" + ], + "source": "HMRC Child Benefit statistics, annual release at August 2025 (percentage of eligible children for whom Child Benefit is claimed by single year of age; families registered and families opted out of payment), applied to PolicyEngine-UK Child Benefit eligibility and adjusted net income after the SPI income chain.", + "stage": "child_benefit_take_up", + "survey": "Family Resources Survey 2024-25" + }, "etb_services": { "artifacts": [ { @@ -2127,10 +2281,9 @@ ], "grain": "person", "nonnegative_outputs": [ - "capital_gains_residential_property", "capital_gains_badr" ], - "notes": "Spine stage after the amounts redraw: every liable gainer carries one main asset type; where flagged residential, its net gain is restated in capital_gains_residential_property; where drawn as a BADR or Investors' Relief claimant, its qualifying gain is written to capital_gains_badr and its main type is one the reliefs apply to. The residential flag is solved to the Table 8a 2024-25 totals on the individuals basis, which the hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains targets bind; the BADR claims are solved band by band to Table 4.1 2024-25, whose individuals band rows the hmrc.cgt.badr_ir_taxpayers_by_band and hmrc.cgt.badr_ir_qualifying_gains_by_band targets bind (microcosm#1014); the non-residential composition follows the fenced Table 7 2023-24 prior and is diagnostic only. Carried interest (Table 9) is not modelled (microcosm#725).", + "notes": "Spine stage after the amounts redraw, the incidence anchor and the residential split: every liable gainer carries one main asset type; a residential arm (capital_gains_residential_property above zero, written by cgt_residential_split since microcosm#1063) is typed residential and its net gain is already restated there; where drawn as a BADR or Investors' Relief claimant, its qualifying gain is written to capital_gains_badr and its main type is one the reliefs apply to. The residential arms carry the Table 8a 2024-25 totals on the individuals basis at design weights, which the hmrc.cgt.residential_property_taxpayers and hmrc.cgt.residential_property_gains targets bind; the BADR claims are solved band by band to Table 4.1 2024-25, whose individuals band rows the hmrc.cgt.badr_ir_taxpayers_by_band and hmrc.cgt.badr_ir_qualifying_gains_by_band targets bind (microcosm#1014); the non-residential composition follows the fenced Table 7 2023-24 prior and is diagnostic only: since microcosm#1063 the type is fitted once per claimant status (claimants to Table 7's composition among the types the reliefs apply to, non-claimants to the composition that restores Table 7 overall) and realised by a deterministic walk that keeps the realised gains by type on the fit. Carried interest (Table 9) is not modelled (microcosm#725).", "operations": [ { "artifact_role": "cgt_asset_type_facts", @@ -2155,22 +2308,6 @@ "runtime_sha256_required": true, "source_vintage": "2024-25" }, - { - "basis_assumption": "trusts hold the same share of the Self Assessment component as of the UK Property service, and a flagged person's whole net gain is attributed to residential property", - "count_target": "Table 8a 2024-25 total taxpayers reporting residential property disposals times the Table 8b 2024-25 individuals/all taxpayer share on the UK Property service", - "gains_target": "Table 8a 2024-25 total residential property gains times the Table 8b 2024-25 individuals/all gains share on the UK Property service", - "kind": "assign_residential_property_flag", - "model": "logistic probability in centred log gains with a stock shift, p = 1 / (1 + exp(-(a + b (log g - c) + s h))) with c the weighted mean log gain of the population, s the stock log-odds and h one where the stock signal holds, zero otherwise", - "output_column": "capital_gains_residential_property", - "output_semantics": "the person's capital_gains where flagged residential, 0 otherwise", - "parameter_solver": "nested bisection; the intercept matches the expected weighted count at each trial slope, the slope matches the expected weighted gains", - "population": "persons with net capital gains above the annual exempt amount (the national taxpayer proxy)", - "realization": "weighted systematic sampling in ascending gain order with one seeded offset: a person is flagged when the expected weight owed so far reaches their own weight, so the flagged weight tracks the expected weight within one person's weight along the gains axis and the realised count and gains sit close to their expectations; the Bernoulli sigma of each is reported as the envelope", - "seed": 553, - "seed_mixing": "seed combined with the build period", - "stock_log_odds": 1.6, - "stock_signal": "household other residential property value above zero, or the person's property income above zero" - }, { "band_pool": "the population with net gains inside the band's range; the open top band takes net gains at or above the lifetime limit", "bands": "HMRC Table 4.1 2024-25 individuals claiming Business Asset Disposal Relief or Investors' Relief by band of qualifying gain (lower limits 0, 10,000, 25,000, 50,000, 100,000, 250,000, 500,000 and 1,000,000)", @@ -2183,7 +2320,7 @@ "gov.hmrc.cgt.badr.lifetime_limit", "gov.hmrc.cgt.badr.rate" ], - "population": "liable gainers not flagged residential", + "population": "liable gainers whose arm is not residential (capital_gains_residential_property == 0, cgt_residential_split)", "qualifying_amount": "below the lifetime limit a claimant's whole net gain qualifies; in the open top band the qualifying gain is the lifetime limit", "realization": "weighted systematic sampling within each band in ascending (gain, person_id) order with one seeded offset per band, drawn in band order whether or not the band carries a target", "reported_not_fitted": "tax at the relief rate, with the annual exempt amount set against the main-rate gains first as the engine allocates it, against Table 4.1's tax column", @@ -2195,6 +2332,7 @@ "type_coupling": "a claimant's main asset type is drawn only from unlisted_shares, agricultural_commercial_industrial_land_buildings, other_non_financial_assets" }, { + "basis_assumption": "Table 7 does not split gains by relief, so claimants are given its composition among the claimant categories", "categories": [ "listed_shares", "unlisted_shares", @@ -2208,16 +2346,16 @@ "other_non_financial_assets" ], "claimant_restriction": "a BADR or Investors' Relief claimant draws only from the claimant categories, so the type follows the claim", + "claimant_share_targets": "the share targets among the claimant categories, each over their sum", "diagnostic_only": true, "kernel": "log-normal density in log gains centred on each type's Table 7 2023-24 mean gain per disposal, one common log-scale width, multiplied by the stock floor where the type declares a stock signal the gainer does not show; types without a declared signal are not scaled", "kind": "assign_main_asset_type", "log_sigma": 1.5, + "non_claimant_share_targets": "(share target - k x claimant share target) / (1 - k) with k the claimants' share of the non-residential gains, the composition that restores the share targets over claimants and non-claimants together", "none_semantics": "no positive net gains", "output_column": "capital_gains_asset_type", "population": "liable gainers not flagged residential", - "realization": "one seeded uniform per person against the cumulative type probabilities, consumed in person_id order", - "seed": 554, - "seed_mixing": "seed combined with the build period", + "realization": "error diffusion per claimant status in descending (gain, person_id) order: each person adds weight x gain x type probability to a running balance per type and takes the type it may hold with the largest balance, which is drawn down by the whole weight x gain, so the realised gains by type track the fitted composition instead of following the heaviest rows; deterministic, no seed", "share_targets": "Table 7 2023-24 gains by asset type excluding residential land and buildings, as shares", "stock_signals": { "listed_shares": "household stocks-and-shares ISA above zero, or the person's dividend income above zero", @@ -2236,7 +2374,7 @@ "agricultural_commercial_industrial_land_buildings", "other_non_financial_assets" ], - "weight_fitting": "multiplicative updates of the type weights, with claimants restricted to the claimant categories, until the household-weighted expected gains shares by type match the share targets; the fit refuses if it has not converged within the iteration limit, and refuses up front when claimants hold more of the gains than the claimant categories' share", + "weight_fitting": "one fit per claimant status: multiplicative updates of the type weights until the household-weighted expected gains shares by type match that status's share targets; a fit refuses if it has not converged within the iteration limit, and the stage refuses up front when claimants hold more of the gains than the claimant categories' share", "weight_fitting_iterations": 200, "weight_fitting_tolerance": 1e-06 }, @@ -2260,7 +2398,6 @@ ], "outputs": [ "capital_gains_asset_type", - "capital_gains_residential_property", "capital_gains_badr" ], "rewrites": [], @@ -2784,10 +2921,13 @@ 2000000 ], "carrier_column": "person_is_spi_income_band_carrier", + "draw": "identity_keyed_inverse_cdf: stable_identity_uniforms(draw_key, seed, salt) over the pool's cumulative FACT in tape order", + "draw_key": "person_source_id", "kind": "resample_band_donor_leaves", "outputs": "stage-1 outputs, uprated as the stage-1 draws", "pool": "full prepared donor tape by published total income band (TEI + TII), narrowed to the carrier's SPI age band where that pool holds the age minimum (composites never age-match), then to the carrier's region where that pool holds the regional minimum", "regional_pool_minimum": 20, + "salt": "spi_income_band_donor_leaf_draw", "seed": 44, "weighting": "FACT", "with_replacement": true @@ -3333,6 +3473,7 @@ ] }, "donor_weight": "weighta * 1000", + "floor": 0, "kind": "derive", "logged_dropna_row_count": true, "lossy_mappings": [ @@ -4277,7 +4418,7 @@ ], "grain": "household", "nonnegative_outputs": [], - "notes": "Deterministically rescales owner rows so regional unweighted owner means match the public house-price reference. Northern Ireland has no reference row and is never scaled; empty and nonpositive regions are skipped. The unweighted mean follows the incumbent behavior.", + "notes": "Deterministically rescales owner rows so regional unweighted owner means match the public house-price reference. Northern Ireland has no reference row and is never scaled; empty and nonpositive regions are skipped. The unweighted mean follows the incumbent behavior. The factor is read off the main residence and has always scaled the property total; since microcosm#1063 it scales the total's other components in the frame as well (land, other houses, buildings), so an owner's property_wealth stays the sum of its components after the uprating. Mortgage debt is never uprated.", "operations": [ { "factor": "avg_house_price / unweighted_mean(main_residence_value | region, >0)", @@ -4297,7 +4438,10 @@ "outputs": [], "rewrites": [ "main_residence_value", - "property_wealth" + "property_wealth", + "owned_land", + "other_residential_property_value", + "non_residential_property_value" ], "source": "MHCLG dwellings and ONS UK House Price Index December 2025 regional average prices.", "stage": "regional_property_uprating", @@ -4317,7 +4461,8 @@ "nonnegative_outputs": [ "pension_contributions_via_salary_sacrifice", "employee_pension_contributions", - "employment_income" + "employment_income", + "salary_sacrifice_pre_conversion_pay" ], "notes": "The QRF trains only on the 2024-25 FRS asked subset. The second arm creates support toward the reviewed 5.4m staging target by moving contributors' full pension amounts into salary sacrifice and lowering their pay by the same amount, since the engine reads employment_income after sacrifice; before microcosm#1069 c9 the pay stayed whole and the converted record lost its net-pay relief.", "operations": [ @@ -4332,7 +4477,7 @@ ], "preserve_asked_rows": true, "seed": 42, - "target_population": "salary_sacrifice_asked != 1 frame-wide", + "target_population": "salary_sacrifice_asked != 1 and employment_income > 0, frame-wide; people not asked who have no employment income are set to zero", "targets": [ "pension_contributions_via_salary_sacrifice" ], @@ -4343,7 +4488,7 @@ { "donor_pool": "employee_pension_contributions > 0 and pension_contributions_via_salary_sacrifice == 0 and employment_income > 0", "kind": "convert_donors_to_target_stock", - "move": "full employee_pension_contributions to pension_contributions_via_salary_sacrifice; source zeroed; employment_income lowered by the amount sacrificed", + "move": "full employee_pension_contributions to pension_contributions_via_salary_sacrifice; source zeroed; employment_income lowered by the amount sacrificed; the converted record's pay before the conversion kept on salary_sacrifice_pre_conversion_pay (zero elsewhere)", "rate_cap": 0.5, "reason": "Salary-sacrifice support stage rewrites pension columns and converted records' pay; household rows and typed household weights pass through and total household mass is conserved.", "receipt": "weighted_headcount", @@ -4356,7 +4501,8 @@ "outputs": [ "pension_contributions_via_salary_sacrifice", "employee_pension_contributions", - "employment_income" + "employment_income", + "salary_sacrifice_pre_conversion_pay" ], "rewrites": [ "pension_contributions_via_salary_sacrifice", @@ -4490,7 +4636,7 @@ "nonnegative_outputs": [ "spi_income_band_donor_lower_bound" ], - "notes": "Reserves 120 SPI-channel households per Table 2.5 band from GBP 200,000 (480 in all) so the top of the income distribution has support before calibration. Carriers are FRS adults drawn by the tape's own band propensity given region, sex and age band, their households are copied whole at clone index 2 with the synthetic flag, and each starts at the build tax year's published taxpayers over 120. The income stage gives every carrier a band-conditional draw from the tape (resample_band_donor_leaves).", + "notes": "Reserves SPI-channel households for every Table 2.5 band from GBP 200,000 so the top of the income distribution has support before calibration. A band seats enough donors that none starts above the maximum weight and never fewer than the minimum, at equal weights of the build tax year's published taxpayers over the band's count. Carriers are FRS adults seated by identity-keyed draws on household weight times the tape's own band propensity given region, sex and age band; their households are copied whole at clone index 2 with the synthetic flag. The mass the donors carry is taken in proportion from the households of each donor's region, so the national total and every region's mass are unchanged (microcosm#1063). The income stage gives every carrier a band-conditional draw from the tape (resample_band_donor_leaves).", "operations": [ { "band_column": "spi_income_band_donor_lower_bound", @@ -4500,25 +4646,39 @@ 1000000, 2000000 ], - "candidate_order": "person_id ascending", + "candidate_order": "person_id ascending breaks equal draw keys", "candidate_population": "raw FRS base channel adults aged 16 and over", "carrier_column": "person_is_spi_income_band_carrier", "clone_index": 2, - "donors_per_band": 120, - "draw": "weighted_without_replacement", + "conservation": "exact_total", + "count_rule": "max(minimum_donors_per_band, ceil(published band taxpayers / maximum_donor_weight))", + "declared_factor": 1, + "draw": "weighted_without_replacement_identity_keyed: Efraimidis-Spirakis keys log(u) / (household weight x propensity) on stable_identity_uniforms(person_id, seed, salt:band); the largest keys are seated, one carrier per household, the highest band first", "expected_band_count": 4, - "expected_donor_count": 480, "flag_column": "household_is_spi_income_band_donor", - "initial_weight": "published band taxpayers / donors_per_band", + "funding": "stratum reallocation: every incumbent household of the stratum a donor's source household belongs to is scaled by 1 - donor mass / stratum mass", + "funding_strata": [ + "region" + ], "kind": "stack_income_band_donor_households", + "mass_scale": "min(1, min over funding strata of maximum_reallocation_share x stratum mass / seated donor mass); every band's weight scaled alike", + "maximum_donor_household_share": 0.15, + "maximum_donor_weight": 250, + "maximum_reallocation_share": 0.5, + "minimum_donors_per_band": 120, "never_zero_weight": true, "propensity": "SPI 2022-23 FACT-weighted band share by region, sex and age band", - "reason": "SPI income band donors stack positive-weight support households per HMRC Table 2.5 total-income band from GBP 200,000, one carrier adult each; the published band taxpayer mass is added explicitly and the base and synthetic channels keep their weights.", + "reason": "Reallocate the published HMRC Table 2.5 taxpayer mass of each total-income band from GBP 200,000 from the households of each donor's region to its income band donor support rows; every region's household mass and the total national mass are conserved.", + "salt": "spi_income_band_donor_draw", + "seating_scale": "min(1, maximum_donor_household_share x eligible base households / expected donor count); band counts floored, at least one per band", "seed": 3, + "tape_preparation": "as hmrc_spi_income_spine prepares the tape: its stage-1 seed and draw_spi_donor_ages_by_population", + "tape_preparation_seed": 42, "taxpayer_count_measure": "total_taxpayer_count", "taxpayer_count_period": "build_period_tax_year", "taxpayer_count_resource": "hmrc_itl_taxpayer_counts.json", - "weight_kind_out": "importance" + "weight_kind_out": "importance", + "weight_rule": "published band taxpayers / band donor count, equal within a band" } ], "outputs": [ @@ -5005,8 +5165,7 @@ "private_pension_income", "employment_income", "self_employment_income", - "capital_income", - "is_renting" + "capital_income" ] }, { @@ -5136,30 +5295,35 @@ "mortgage_debt", "consumer_debt" ], - "notes": "Ports incumbent WAS round-8 wealth imputation with signed E5 differences: exact lower-case column matching replaces the fuzzy r/w fallback; cash ISA uses DVCISAVR8_aggr and stocks-and-shares ISA uses DVIISAVR8_aggr; private pension wealth (WAS total private pension wealth less the current defined-benefit component; DB-type components at the SCAPE rate, money-purchase components at reported fund values) is drawn as its own output private_pension_wealth and no longer folded into corporate_wealth, which keeps only the share-like holdings - pension rights are disregarded capital in every means test (UC Regs 2013 Sch 10 para 10 and the HB/JSA/ESA/IS/SPC parallels; uk-data#452) - and the downstream chain segment conditions on both components; each chain segment starts its own fit and draw RNG streams from a child seed derived deterministically from the declared seed (start_chain respawns from the model seed on every call, so one shared model coupled the k-th target of every segment); corporate_wealth folds stocks-and-shares ISA after drawing corporate_wealth_excl_isa; recipient Northern Ireland regions are mapped to Wales for prediction only; student_loan_balance is allocated by household id rather than the incumbent positional off-by-one. The debt segment (fourth chain segment, fourth child seed) reads WAS round-8 derived variables HMortGR8 (total outstanding mortgage debt), HFINWNTR8_exSLC_Sum (net financial wealth excluding student loans; gross less it is the non-student-loan financial liabilities) and Ten1R8 (tenure; codes 2 and 3 are buying with a mortgage and part own/part rent, confirmed by the I1 audit of the pinned tab); UKDS SN 7215 documentation is the codebook. Engine predictors materialize at their native entity and person/benunit values are summed to household, reproducing the incumbent map_to=household semantics; region is one-hot encoded jointly across donor and recipient (the incumbent's dummy encoding), with unmapped donor GOR codes becoming all-zero dummy rows. The WAS and FRS predictor definitions are not fully like-for-like and are ported as-is; raw WAS missing values are blanket-filled with zero; UKDS negative sentinel codes (-9/-8/-7/-6) are recoded to zero for the nonnegative-domain columns the licensed audit found carrying them (vcarnr8: 2 rows; HBedRmR8: 95.8 percent - the bedrooms question is effectively unasked in the WAS household file, predictor-quality revisit registered on microcosm#145) - a signed difference vs the incumbent, which trains on raw sentinels; DVPriRntR8's -9 is structural not-applicable so the is_renting mapping is unchanged; genuinely negative domains are never recoded.", + "notes": "Ports incumbent WAS round-8 wealth imputation with signed E5 differences: exact lower-case column matching replaces the fuzzy r/w fallback; cash ISA uses DVCISAVR8_aggr and stocks-and-shares ISA uses DVIISAVR8_aggr; private pension wealth (WAS total private pension wealth less the current defined-benefit component; DB-type components at the SCAPE rate, money-purchase components at reported fund values) is drawn as its own output private_pension_wealth and no longer folded into corporate_wealth, which keeps only the share-like holdings - pension rights are disregarded capital in every means test (UC Regs 2013 Sch 10 para 10 and the HB/JSA/ESA/IS/SPC parallels; uk-data#452) - and the downstream chain segment conditions on both components; each chain segment starts its own fit and draw RNG streams from a child seed derived deterministically from the declared seed (start_chain respawns from the model seed on every call, so one shared model coupled the k-th target of every segment); corporate_wealth folds stocks-and-shares ISA after drawing corporate_wealth_excl_isa; recipient Northern Ireland regions are mapped to Wales for prediction only; student_loan_balance is allocated by household id rather than the incumbent positional off-by-one. Tenure enters every chain segment as one four-way category built the same way on both sides (microcosm#1063): the donor's from Ten1R8 and DVPriRntR8 (owned outright; buying with a mortgage or part rent, part mortgage; rented or rent-free from a private landlord; rented from a council or housing association), the recipient's from the frame's tenure_type (council and housing association together), so the stage reads no engine tenure formula. Two columns are structural in tenure and are fitted and drawn inside their stratum only, zero by rule elsewhere: main_residence_value on the owner tenures (every WAS owner has a positive value and no renter has one) and main_residence_mortgage (TotMortR8) on the mortgaged tenure. mortgage_debt is HMortGR8, all of the household's mortgages, and is derived as the main-residence mortgage plus the mortgages on other property (other_mortgage, HMortGR8 less TotMortR8 on the donor: buy-to-let, second homes and the rest), which are held on every tenure and drawn without a stratum, so an outright owner's or a renter's buy-to-let keeps its mortgage beside the property it is secured on (OthMortR8_sum is read for the receipt only); the donor's conditioning copy carries the same rule, so consumer_debt is fitted on values recipients can hold. Totals are not drawn: property_wealth is the sum of land, the main residence, other houses, buildings and a drawn remainder (buy-to-let, overseas and other property), gross_financial_wealth is savings, the cash ISA, the share-like holdings and a drawn remainder, and net_financial_wealth is gross less consumer debt and the student loan balance - identities the pinned tab satisfies on every row and the cleaning step refuses a donor that breaks. A sum of donor-valued components that leaves the donor's range for the total gives up the excess from the remainder first, and the receipt counts the rows. Eight chain segments, each with its own child seed: land; the main residence; the other property components; pension and share-like wealth; the financial components, vehicles and the student loan balance; the main-residence mortgage; the other mortgages; consumer debt. HFINWNTR8_exSLC_Sum is net financial wealth excluding student loans, so gross less it is the non-student-loan financial liabilities; UKDS SN 7215 documentation is the codebook. Engine predictors materialize at their native entity and person/benunit values are summed to household, reproducing the incumbent map_to=household semantics; region is one-hot encoded jointly across donor and recipient (the incumbent's dummy encoding), with unmapped donor GOR codes becoming all-zero dummy rows. The WAS and FRS predictor definitions are not fully like-for-like and are ported as-is; raw WAS missing values are blanket-filled with zero; UKDS negative sentinel codes (-9/-8/-7/-6) are recoded to zero for the nonnegative-domain columns the licensed audit found carrying them (vcarnr8: 2 rows; HBedRmR8: 95.8 percent - the bedrooms question is effectively unasked in the WAS household file, predictor-quality revisit registered on microcosm#145) - a signed difference vs the incumbent, which trains on raw sentinels; DVPriRntR8's -9 is structural not-applicable so the private-renter reading is unchanged; genuinely negative domains are never recoded.", "operations": [ { "derived": { "cash_isa": "DVCISAVR8_aggr", "consumer_debt": "max(HFINWR8_SUM - HFINWNTR8_exSLC_Sum, 0)", "corporate_wealth_excl_isa": "DVFESHARESR8_aggr + DVFShUKVR8_aggr + DVFCollVR8_aggr", - "has_mortgage_tenure": "Ten1R8 in {2, 3}", "is_renting": "DVPriRntR8 == 1", + "other_financial_assets": "HFINWR8_SUM - (DVSaValR8_aggr + DVCISAVR8_aggr + corporate_wealth), refused when negative", + "other_property_value": "DVPropertyR8 - (DVhvalueR8 + DVHseValR8_sum + DVBlDValR8_sum + DVLUKValR8_sum), refused when negative", "private_pension_wealth": "totalpenr8_aggr - dvvaldbt_scaper8_aggr", "region": "GORR8 via incumbent REGIONS map", "stocks_and_shares_isa": "DVIISAVR8_aggr", - "student_loan_balance": "Tot_LosR8_aggr - Tot_los_exc_SLCR8_aggr" + "student_loan_balance": "Tot_LosR8_aggr - Tot_los_exc_SLCR8_aggr", + "tenure_category": "Ten1R8 1 owned_outright; 2 or 3 owned_with_mortgage; 4 or 5 private_rent when DVPriRntR8 == 1, otherwise social_rent; any other code private_rent, counted in the receipt" }, "direct_maps": { "gross_financial_wealth": "HFINWR8_SUM", + "main_residence_mortgage": "TotMortR8", "main_residence_value": "DVhvalueR8", "mortgage_debt": "HMortGR8", "net_financial_wealth": "HFINWNTR8_Sum", "net_financial_wealth_exsl": "HFINWNTR8_exSLC_Sum", "non_residential_property_value": "DVBlDValR8_sum", "num_vehicles": "vcarnr8", + "other_property_mortgage": "OthMortR8_sum", "other_residential_property_value": "DVHseValR8_sum", "owned_land": "DVLUKValR8_sum", + "private_rent_code": "DVPriRntR8", "property_wealth": "DVPropertyR8", "savings": "DVSaValR8_aggr", "tenure_code": "Ten1R8", @@ -5179,40 +5343,75 @@ "private_pension_income", "employment_income", "self_employment_income", - "capital_income", - "is_renting" + "capital_income" ] }, { "categorical_predictors": [ "region", - "is_renting" + "tenure_owned_with_mortgage", + "tenure_private_rent", + "tenure_social_rent" ], "chain_order": [ "owned_land", + "main_residence_value", + "other_residential_property_value", + "non_residential_property_value", + "other_property_value", "property_wealth", "private_pension_wealth", "corporate_wealth_excl_isa", "stocks_and_shares_isa", "corporate_wealth", - "gross_financial_wealth", - "net_financial_wealth", - "main_residence_value", - "other_residential_property_value", - "non_residential_property_value", "savings", + "cash_isa", + "other_financial_assets", "num_vehicles", "student_loan_balance", - "cash_isa", + "gross_financial_wealth", + "main_residence_mortgage", + "other_mortgage", "mortgage_debt", - "consumer_debt" - ], - "debt_segment_predictors": [ - "has_mortgage_tenure" + "consumer_debt", + "net_financial_wealth" ], + "derived_totals": { + "corporate_wealth": [ + "corporate_wealth_excl_isa", + "stocks_and_shares_isa" + ], + "gross_financial_wealth": [ + "savings", + "cash_isa", + "corporate_wealth", + "other_financial_assets" + ], + "mortgage_debt": [ + "main_residence_mortgage", + "other_mortgage" + ], + "net_financial_wealth": [ + "gross_financial_wealth", + "-consumer_debt", + "-student_loan_balance" + ], + "property_wealth": [ + "owned_land", + "main_residence_value", + "other_residential_property_value", + "non_residential_property_value", + "other_property_value" + ] + }, "integer_outputs": [ "num_vehicles" ], + "internal_components": [ + "other_property_value", + "other_financial_assets", + "other_mortgage" + ], "kind": "fit_weighted_qrf_chain", "n_estimators": 4, "predictors": [ @@ -5225,13 +5424,24 @@ "capital_income", "num_bedrooms", "council_tax", - "is_renting", + "tenure_owned_with_mortgage", + "tenure_private_rent", + "tenure_social_rent", "region" ], "region_remap": { "NORTHERN_IRELAND": "WALES" }, "seed": 0, + "stratified_targets": { + "main_residence_mortgage": [ + "owned_with_mortgage" + ], + "main_residence_value": [ + "owned_outright", + "owned_with_mortgage" + ] + }, "weights": "explicit" }, { diff --git a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/was.csv b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/was.csv index 9dac330a8..96bfd2181 100644 --- a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/was.csv +++ b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/sources/was.csv @@ -1,65 +1,65 @@ -CASER8,DVFLISAVR8_aggr,R8xshhwgt,DVLUKValR8_sum,DVPropertyR8,DVFESHARESR8_aggr,DVFShUKVR8_aggr,DVIISAVR8_aggR,DVCISAVR8_aggr,DVFCollVR8_aggr,totalpenr8_aggr,dvvaldbt_scaper8_aggr,NumAdultR8,NumCh18R8,DVGIPPENR8_AGGR,DVGISER8_AGGR,DVGIINVR8_aggr,DVGIEMPR8_AGGR,HBedRmR8,GORR8,DVPriRntR8,CTAmtR8,HFINWNTR8_Sum,HFINWNTR8_exSLC_Sum,HMortGR8,Ten1R8,HFINWR8_SUM,DVhvalueR8,DVHseValR8_sum,DVBlDValR8_sum,DVTotinc_bhcR8,DVSaValR8_aggr,vcarnr8,Tot_LosR8_aggr,Tot_los_exc_SLCR8_aggr -1,5,1,10,20000,1,3,5,7,9,100,40,1,0,11,13,15,17,1,8,1,900,-50,20,0,1,30,100000,1000,3000,20000,500,0,9000,4000 -2,0,1.1000000000000001,11,20500,2,4,6,8,10,110,41,2,1,12,14,16,18,2,11,2,910,-46,20,10100,2,31,102000,1050,3060,21000,520,1,9010,4005 -3,0,1.2,12,21000,3,5,7,9,11,120,42,3,2,13,15,17,19,3,12,1,920,-42,20,10200,3,32,104000,1100,3120,22000,540,2,9020,4010 -4,0,1.3,13,21500,4,6,8,10,12,130,43,1,0,14,16,18,20,4,1,2,930,-38,20,0,4,33,106000,1150,3180,23000,560,3,9030,4015 -5,0,1.3999999999999999,14,22000,5,7,9,11,13,140,44,2,1,15,17,19,21,5,8,1,940,-34,20,0,1,34,108000,1200,3240,24000,580,0,9040,4020 -6,10,1.5,15,22500,6,8,10,12,14,150,45,3,2,16,18,20,22,1,11,2,950,-30,25,10500,2,35,110000,1250,3300,25000,600,1,9050,4025 -7,11,1.6000000000000001,16,23000,7,9,11,13,15,160,46,1,0,17,19,21,23,2,12,1,960,-26,25,10600,3,36,112000,1300,3360,26000,620,2,9060,4030 -8,0,1,17,23500,8,10,12,14,16,170,47,2,1,18,20,22,24,3,1,2,970,-22,25,0,4,37,114000,1350,3420,27000,640,3,9070,4035 -9,0,1.1000000000000001,18,24000,9,11,13,15,17,180,48,3,2,19,21,23,25,4,8,1,980,-18,25,0,1,38,116000,1400,3480,28000,660,0,9080,4040 -10,0,1.2,19,24500,10,12,14,16,18,190,49,1,0,20,22,24,26,5,11,2,990,-14,25,10900,2,39,118000,1450,3540,29000,680,1,9090,4045 -11,15,1.3,20,25000,11,13,15,17,19,200,50,2,1,21,23,25,27,1,12,1,1000,-10,30,11000,3,40,120000,1500,3600,30000,700,2,9100,4050 -12,0,1.3999999999999999,21,25500,12,14,16,18,20,210,51,3,2,22,24,26,28,2,1,2,1010,-6,30,0,4,41,122000,1550,3660,31000,720,3,9110,4055 -13,0,1.5,22,26000,13,15,17,19,21,220,52,1,0,23,25,27,29,3,8,1,1020,-2,30,0,1,42,124000,1600,3720,32000,740,0,9120,4060 -14,0,1.6000000000000001,23,26500,14,16,18,20,22,230,53,2,1,24,26,28,30,4,11,2,1030,2,30,11300,2,43,126000,1650,3780,33000,760,1,9130,4065 -15,19,1,24,27000,15,17,19,21,23,240,54,3,2,25,27,29,31,5,12,1,1040,6,30,11400,3,44,128000,1700,3840,34000,780,2,9140,4070 -16,0,1.1000000000000001,25,27500,16,18,20,22,24,250,55,1,0,26,28,30,32,1,1,2,1050,10,35,0,4,45,130000,1750,3900,35000,800,3,9150,4075 -17,0,1.2,26,28000,17,19,21,23,25,260,56,2,1,27,29,31,33,2,8,1,1060,14,35,0,1,46,132000,1800,3960,36000,820,0,9160,4080 -18,0,1.3,27,28500,18,20,22,24,26,270,57,3,2,28,30,32,34,3,11,2,1070,18,35,11700,2,47,134000,1850,4020,37000,840,1,9170,4085 -19,0,1.3999999999999999,28,29000,19,21,23,25,27,280,58,1,0,29,31,33,35,4,12,1,1080,22,35,11800,3,48,136000,1900,4080,38000,860,2,9180,4090 -20,0,1.5,29,29500,20,22,24,26,28,290,59,2,1,30,32,34,36,5,1,2,1090,26,35,0,4,49,138000,1950,4140,39000,880,3,9190,4095 -21,5,1.6000000000000001,30,30000,21,23,25,27,29,300,60,3,2,31,33,35,37,1,8,1,1100,30,40,0,1,50,140000,2000,4200,40000,900,0,9200,4100 -22,0,1,31,30500,22,24,26,28,30,310,61,1,0,32,34,36,38,2,11,2,1110,34,40,12100,2,51,142000,2050,4260,41000,920,1,9210,4105 -23,0,1.1000000000000001,32,31000,23,25,27,29,31,320,62,2,1,33,35,37,39,3,12,1,1120,38,40,12200,3,52,144000,2100,4320,42000,940,2,9220,4110 -24,0,1.2,33,31500,24,26,28,30,32,330,63,3,2,34,36,38,40,4,1,2,1130,42,40,0,4,53,146000,2150,4380,43000,960,3,9230,4115 -25,9,1.3,34,32000,25,27,29,31,33,340,64,1,0,35,37,39,41,5,8,1,1140,46,40,0,1,54,148000,2200,4440,44000,980,0,9240,4120 -26,0,1.3999999999999999,35,32500,26,28,30,32,34,350,65,2,1,36,38,40,42,1,11,2,1150,50,45,12500,2,55,150000,2250,4500,45000,1000,1,9250,4125 -27,0,1.5,36,33000,27,29,31,33,35,360,66,3,2,37,39,41,43,2,12,1,1160,54,45,12600,3,56,152000,2300,4560,46000,1020,2,9260,4130 -28,0,1.6000000000000001,37,33500,28,30,32,34,36,370,67,1,0,38,40,42,44,3,1,2,1170,58,45,0,4,57,154000,2350,4620,47000,1040,3,9270,4135 -29,13,1,38,34000,29,31,33,35,37,380,68,2,1,39,41,43,45,4,8,1,1180,62,45,0,1,58,156000,2400,4680,48000,1060,0,9280,4140 -30,0,1.1000000000000001,39,34500,30,32,34,36,38,390,69,3,2,40,42,44,46,5,11,2,1190,66,45,12900,2,59,158000,2450,4740,49000,1080,1,9290,4145 -31,0,1.2,40,35000,31,33,35,37,39,400,70,1,0,41,43,45,47,1,12,1,1200,70,50,13000,3,60,160000,2500,4800,50000,1100,2,9300,4150 -32,0,1.3,41,35500,32,34,36,38,40,410,71,2,1,42,44,46,48,2,1,2,1210,74,50,0,4,61,162000,2550,4860,51000,1120,3,9310,4155 -33,0,1.3999999999999999,42,36000,33,35,37,39,41,420,72,3,2,43,45,47,49,3,8,1,1220,78,50,0,1,62,164000,2600,4920,52000,1140,0,9320,4160 -34,0,1.5,43,36500,34,36,38,40,42,430,73,1,0,44,46,48,50,4,11,2,1230,82,50,13300,2,63,166000,2650,4980,53000,1160,1,9330,4165 -35,19,1.6000000000000001,44,37000,35,37,39,41,43,440,74,2,1,45,47,49,51,5,12,1,1240,86,50,13400,3,64,168000,2700,5040,54000,1180,2,9340,4170 -36,0,1,45,37500,36,38,40,42,44,450,75,3,2,46,48,50,52,1,1,2,1250,90,55,0,4,65,170000,2750,5100,55000,1200,3,9350,4175 -37,0,1.1000000000000001,46,38000,37,39,41,43,45,460,76,1,0,47,49,51,53,2,8,1,1260,94,55,0,1,66,172000,2800,5160,56000,1220,0,9360,4180 -38,0,1.2,47,38500,38,40,42,44,46,470,77,2,1,48,50,52,54,3,11,2,1270,98,55,13700,2,67,174000,2850,5220,57000,1240,1,9370,4185 -39,23,1.3,48,39000,39,41,43,45,47,480,78,3,2,49,51,53,55,4,12,1,1280,102,55,13800,3,68,176000,2900,5280,58000,1260,2,9380,4190 -40,0,1.3999999999999999,49,39500,40,42,44,46,48,490,79,1,0,50,52,54,56,5,1,2,1290,106,55,0,4,69,178000,2950,5340,59000,1280,3,9390,4195 -41,0,1.5,50,40000,41,43,45,47,49,500,80,2,1,51,53,55,57,1,8,1,1300,110,60,0,1,70,180000,3000,5400,60000,1300,0,9400,4200 -42,0,1.6000000000000001,51,40500,42,44,46,48,50,510,81,3,2,52,54,56,58,2,11,2,1310,114,60,14100,2,71,182000,3050,5460,61000,1320,1,9410,4205 -43,7,1,52,41000,43,45,47,49,51,520,82,1,0,53,55,57,59,3,12,1,1320,118,60,14200,3,72,184000,3100,5520,62000,1340,2,9420,4210 -44,0,1.1000000000000001,53,41500,44,46,48,50,52,530,83,2,1,54,56,58,60,4,1,2,1330,122,60,0,4,73,186000,3150,5580,63000,1360,3,9430,4215 -45,0,1.2,54,42000,45,47,49,51,53,540,84,3,2,55,57,59,61,5,8,1,1340,126,60,0,1,74,188000,3200,5640,64000,1380,0,9440,4220 -46,0,1.3,55,42500,46,48,50,52,54,550,85,1,0,56,58,60,62,1,11,2,1350,130,65,14500,2,75,190000,3250,5700,65000,1400,1,9450,4225 -47,0,1.3999999999999999,56,43000,47,49,51,53,55,560,86,2,1,57,59,61,63,2,12,1,1360,134,65,14600,3,76,192000,3300,5760,66000,1420,2,9460,4230 -48,0,1.5,57,43500,48,50,52,54,56,570,87,3,2,58,60,62,64,3,1,2,1370,138,65,0,4,77,194000,3350,5820,67000,1440,3,9470,4235 -49,13,1.6000000000000001,58,44000,49,51,53,55,57,580,88,1,0,59,61,63,65,4,8,1,1380,142,65,0,1,78,196000,3400,5880,68000,1460,0,9480,4240 -50,0,1,59,44500,50,52,54,56,58,590,89,2,1,60,62,64,66,5,11,2,1390,146,65,14900,2,79,198000,3450,5940,69000,1480,1,9490,4245 -51,0,1.1000000000000001,60,45000,51,53,55,57,59,600,90,3,2,61,63,65,67,1,12,1,1400,150,70,15000,3,80,200000,3500,6000,70000,1500,2,9500,4250 -52,0,1.2,61,45500,52,54,56,58,60,610,91,1,0,62,64,66,68,2,1,2,1410,154,70,0,4,81,202000,3550,6060,71000,1520,3,9510,4255 -53,17,1.3,62,46000,53,55,57,59,61,620,92,2,1,63,65,67,69,3,8,1,1420,158,70,0,1,82,204000,3600,6120,72000,1540,0,9520,4260 -54,0,1.3999999999999999,63,46500,54,56,58,60,62,630,93,3,2,64,66,68,70,4,11,2,1430,162,70,15300,2,83,206000,3650,6180,73000,1560,1,9530,4265 -55,0,1.5,64,47000,55,57,59,61,63,640,94,1,0,65,67,69,71,5,12,1,1440,166,70,15400,3,84,208000,3700,6240,74000,1580,2,9540,4270 -56,0,1.6000000000000001,65,47500,56,58,60,62,64,650,95,2,1,66,68,70,72,1,1,2,1450,170,75,0,4,85,210000,3750,6300,75000,1600,3,9550,4275 -57,21,1,66,48000,57,59,61,63,65,660,96,3,2,67,69,71,73,2,8,1,1460,174,75,0,1,86,212000,3800,6360,76000,1620,0,9560,4280 -58,0,1.1000000000000001,67,48500,58,60,62,64,66,670,97,1,0,68,70,72,74,3,11,2,1470,178,75,15700,2,87,214000,3850,6420,77000,1640,1,9570,4285 -59,0,1.2,68,49000,59,61,63,65,67,680,98,2,1,69,71,73,75,4,12,1,1480,182,75,15800,3,88,216000,3900,6480,78000,1660,2,9580,4290 -60,0,1.3,69,49500,60,62,64,66,68,690,99,3,2,70,72,74,76,5,1,2,1490,186,75,0,4,89,218000,3950,6540,79000,1680,3,9590,4295 -61,0,1.3999999999999999,70,50000,61,63,65,67,69,700,100,1,0,71,73,75,77,1,8,1,1500,190,80,0,1,90,220000,4000,6600,80000,1700,0,9600,4300 -62,0,1.5,71,50500,62,64,66,68,70,710,101,2,1,72,74,76,78,2,11,2,1510,194,80,16100,2,91,222000,4050,6660,81000,1720,1,9610,4305 -63,7,1.6000000000000001,72,51000,63,65,67,69,71,720,102,3,2,73,75,77,79,3,12,1,1520,198,80,16200,3,92,224000,4100,6720,82000,1740,2,9620,4310 -64,0,1,73,51500,64,66,68,70,72,730,103,1,0,74,76,78,80,4,1,2,1530,202,80,0,4,93,226000,4150,6780,83000,1760,3,9630,4315 +CASER8,DVFLISAVR8_aggr,R8xshhwgt,DVLUKValR8_sum,DVPropertyR8,DVFESHARESR8_aggr,DVFShUKVR8_aggr,DVIISAVR8_aggR,DVCISAVR8_aggr,DVFCollVR8_aggr,totalpenr8_aggr,dvvaldbt_scaper8_aggr,NumAdultR8,NumCh18R8,DVGIPPENR8_AGGR,DVGISER8_AGGR,DVGIINVR8_aggr,DVGIEMPR8_AGGR,HBedRmR8,GORR8,DVPriRntR8,CTAmtR8,HFINWNTR8_Sum,HFINWNTR8_exSLC_Sum,HMortGR8,TotMortR8,OthMortR8_sum,Ten1R8,HFINWR8_SUM,DVhvalueR8,DVHseValR8_sum,DVBlDValR8_sum,DVTotinc_bhcR8,DVSaValR8_aggr,vcarnr8,Tot_LosR8_aggr,Tot_los_exc_SLCR8_aggr +1,5,1,10,104510,1,3,5,7,9,100,40,1,0,11,13,15,17,1,8,-9,900,555,555,2000,0,2000,1,555,100000,1000,3000,20000,500,0,9000,9000 +2,0,1.1000000000000001,0,102000,2,4,6,8,10,110,41,2,1,12,14,16,18,2,11,-9,910,-4464,541,10100,10100,0,2,581,102000,0,0,21000,520,1,14015,9010 +3,0,1.2,0,104000,3,5,7,9,11,120,42,3,2,13,15,17,19,3,12,-9,920,527,527,10200,10200,0,3,607,104000,0,0,22000,540,2,9020,9020 +4,0,1.3,0,1150,4,6,8,10,12,130,43,1,0,14,16,18,20,4,1,1,930,513,513,0,0,0,4,633,0,1150,0,23000,560,3,9030,9030 +5,0,1.3999999999999999,0,3240,5,7,9,11,13,140,44,2,1,15,17,19,21,5,8,2,940,-4521,499,0,0,0,4,659,0,0,3240,24000,580,0,14060,9040 +6,10,1.5,15,15,6,8,10,12,14,150,45,3,2,16,18,20,22,1,11,1,950,685,685,0,0,0,5,685,0,0,0,25000,600,1,9050,9050 +7,11,1.6000000000000001,0,113300,7,9,11,13,15,160,46,1,0,17,19,21,23,2,12,-9,960,671,671,2006,0,2006,1,711,112000,1300,0,26000,620,2,9060,9060 +8,0,1,0,114535,8,10,12,14,16,170,47,2,1,18,20,22,24,3,1,-9,970,-4378,657,10700,10700,0,2,737,114000,0,0,27000,640,3,14105,9070 +9,0,1.1000000000000001,0,119480,9,11,13,15,17,180,48,3,2,19,21,23,25,4,8,-9,980,643,643,10800,10800,0,3,763,116000,0,3480,28000,660,0,9080,9080 +10,0,1.2,0,1450,10,12,14,16,18,190,49,1,0,20,22,24,26,5,11,1,990,629,629,0,0,0,4,789,0,1450,0,29000,680,1,9090,9090 +11,15,1.3,20,20,11,13,15,17,19,200,50,2,1,21,23,25,27,1,12,2,1000,-4235,815,0,0,0,4,815,0,0,0,30000,700,2,14150,9100 +12,0,1.3999999999999999,0,0,12,14,16,18,20,210,51,3,2,22,24,26,28,2,1,1,1010,801,801,0,0,0,5,841,0,0,0,31000,720,3,9110,9110 +13,0,1.5,0,129320,13,15,17,19,21,220,52,1,0,23,25,27,29,3,8,-9,1020,787,787,2012,0,2012,1,867,124000,1600,3720,32000,740,0,9120,9120 +14,0,1.6000000000000001,0,126000,14,16,18,20,22,230,53,2,1,24,26,28,30,4,11,-9,1030,-4292,773,11300,11300,0,2,893,126000,0,0,33000,760,1,14195,9130 +15,19,1,0,128570,15,17,19,21,23,240,54,3,2,25,27,29,31,5,12,-9,1040,759,759,11400,11400,0,3,919,128000,0,0,34000,780,2,9140,9140 +16,0,1.1000000000000001,25,1775,16,18,20,22,24,250,55,1,0,26,28,30,32,1,1,1,1050,945,945,0,0,0,4,945,0,1750,0,35000,800,3,9150,9150 +17,0,1.2,0,3960,17,19,21,23,25,260,56,2,1,27,29,31,33,2,8,2,1060,-4149,931,0,0,0,4,971,0,0,3960,36000,820,0,14240,9160 +18,0,1.3,0,0,18,20,22,24,26,270,57,3,2,28,30,32,34,3,11,1,1070,917,917,0,0,0,5,997,0,0,0,37000,840,1,9170,9170 +19,0,1.3999999999999999,0,137900,19,21,23,25,27,280,58,1,0,29,31,33,35,4,12,-9,1080,903,903,2018,0,2018,1,1023,136000,1900,0,38000,860,2,9180,9180 +20,0,1.5,0,138000,20,22,24,26,28,290,59,2,1,30,32,34,36,5,1,-9,1090,-4206,889,11900,11900,0,2,1049,138000,0,0,39000,880,3,14285,9190 +21,5,1.6000000000000001,30,144230,21,23,25,27,29,300,60,3,2,31,33,35,37,1,8,-9,1100,1075,1075,12000,12000,0,3,1075,140000,0,4200,40000,900,0,9200,9200 +22,0,1,0,2655,22,24,26,28,30,310,61,1,0,32,34,36,38,2,11,1,1110,1061,1061,0,0,0,4,1101,0,2050,0,41000,920,1,9210,9210 +23,0,1.1000000000000001,0,0,23,25,27,29,31,320,62,2,1,33,35,37,39,3,12,2,1120,-4063,1047,0,0,0,4,1127,0,0,0,42000,940,2,14330,9220 +24,0,1.2,0,0,24,26,28,30,32,330,63,3,2,34,36,38,40,4,1,1,1130,1033,1033,0,0,0,5,1153,0,0,0,43000,960,3,9230,9230 +25,9,1.3,0,154640,25,27,29,31,33,340,64,1,0,35,37,39,41,5,8,-9,1140,1019,1019,2024,0,2024,1,1179,148000,2200,4440,44000,980,0,9240,9240 +26,0,1.3999999999999999,35,150035,26,28,30,32,34,350,65,2,1,36,38,40,42,1,11,-9,1150,-3920,1205,12500,12500,0,2,1205,150000,0,0,45000,1000,1,14375,9250 +27,0,1.5,0,152000,27,29,31,33,35,360,66,3,2,37,39,41,43,2,12,-9,1160,1191,1191,12600,12600,0,3,1231,152000,0,0,46000,1020,2,9260,9260 +28,0,1.6000000000000001,0,2350,28,30,32,34,36,370,67,1,0,38,40,42,44,3,1,1,1170,1177,1177,0,0,0,4,1257,0,2350,0,47000,1040,3,9270,9270 +29,13,1,0,5320,29,31,33,35,37,380,68,2,1,39,41,43,45,4,8,2,1180,-3977,1163,0,0,0,4,1283,0,0,4680,48000,1060,0,14420,9280 +30,0,1.1000000000000001,0,0,30,32,34,36,38,390,69,3,2,40,42,44,46,5,11,1,1190,1149,1149,0,0,0,5,1309,0,0,0,49000,1080,1,9290,9290 +31,0,1.2,40,162540,31,33,35,37,39,400,70,1,0,41,43,45,47,1,12,-9,1200,1335,1335,2030,0,2030,1,1335,160000,2500,0,50000,1100,2,9300,9300 +32,0,1.3,0,162000,32,34,36,38,40,410,71,2,1,42,44,46,48,2,1,-9,1210,-3834,1321,13100,13100,0,2,1361,162000,0,0,51000,1120,3,14465,9310 +33,0,1.3999999999999999,0,168920,33,35,37,39,41,420,72,3,2,43,45,47,49,3,8,-9,1220,1307,1307,13200,13200,0,3,1387,164000,0,4920,52000,1140,0,9320,9320 +34,0,1.5,0,2650,34,36,38,40,42,430,73,1,0,44,46,48,50,4,11,1,1230,1293,1293,0,0,0,4,1413,0,2650,0,53000,1160,1,9330,9330 +35,19,1.6000000000000001,0,0,35,37,39,41,43,440,74,2,1,45,47,49,51,5,12,2,1240,-3891,1279,0,0,0,4,1439,0,0,0,54000,1180,2,14510,9340 +36,0,1,45,720,36,38,40,42,44,450,75,3,2,46,48,50,52,1,1,1,1250,1465,1465,0,0,0,5,1465,0,0,0,55000,1200,3,9350,9350 +37,0,1.1000000000000001,0,179960,37,39,41,43,45,460,76,1,0,47,49,51,53,2,8,-9,1260,1451,1451,2036,0,2036,1,1491,172000,2800,5160,56000,1220,0,9360,9360 +38,0,1.2,0,174000,38,40,42,44,46,470,77,2,1,48,50,52,54,3,11,-9,1270,-3748,1437,13700,13700,0,2,1517,174000,0,0,57000,1240,1,14555,9370 +39,23,1.3,0,176000,39,41,43,45,47,480,78,3,2,49,51,53,55,4,12,-9,1280,1423,1423,13800,13800,0,3,1543,176000,0,0,58000,1260,2,9380,9380 +40,0,1.3999999999999999,0,2950,40,42,44,46,48,490,79,1,0,50,52,54,56,5,1,1,1290,1409,1409,0,0,0,4,1569,0,2950,0,59000,1280,3,9390,9390 +41,0,1.5,50,5450,41,43,45,47,49,500,80,2,1,51,53,55,57,1,8,2,1300,-3605,1595,0,0,0,4,1595,0,0,5400,60000,1300,0,14600,9400 +42,0,1.6000000000000001,0,0,42,44,46,48,50,510,81,3,2,52,54,56,58,2,11,1,1310,1581,1581,0,0,0,5,1621,0,0,0,61000,1320,1,9410,9410 +43,7,1,0,187810,43,45,47,49,51,520,82,1,0,53,55,57,59,3,12,-9,1320,1567,1567,2042,0,2042,1,1647,184000,3100,0,62000,1340,2,9420,9420 +44,0,1.1000000000000001,0,186000,44,46,48,50,52,530,83,2,1,54,56,58,60,4,1,-9,1330,-3662,1553,14300,14300,0,2,1673,186000,0,0,63000,1360,3,14645,9430 +45,0,1.2,0,193640,45,47,49,51,53,540,84,3,2,55,57,59,61,5,8,-9,1340,1539,1539,14400,14400,0,3,1699,188000,0,5640,64000,1380,0,9440,9440 +46,0,1.3,55,3305,46,48,50,52,54,550,85,1,0,56,58,60,62,1,11,1,1350,1725,1725,0,0,0,4,1725,0,3250,0,65000,1400,1,9450,9450 +47,0,1.3999999999999999,0,0,47,49,51,53,55,560,86,2,1,57,59,61,63,2,12,2,1360,-3519,1711,0,0,0,4,1751,0,0,0,66000,1420,2,14690,9460 +48,0,1.5,0,0,48,50,52,54,56,570,87,3,2,58,60,62,64,3,1,1,1370,1697,1697,0,0,0,5,1777,0,0,0,67000,1440,3,9470,9470 +49,13,1.6000000000000001,0,205280,49,51,53,55,57,580,88,1,0,59,61,63,65,4,8,-9,1380,1683,1683,2048,0,2048,1,1803,196000,3400,5880,68000,1460,0,9480,9480 +50,0,1,0,198745,50,52,54,56,58,590,89,2,1,60,62,64,66,5,11,-9,1390,-3576,1669,14900,14900,0,2,1829,198000,0,0,69000,1480,1,14735,9490 +51,0,1.1000000000000001,60,200060,51,53,55,57,59,600,90,3,2,61,63,65,67,1,12,-9,1400,1855,1855,15000,15000,0,3,1855,200000,0,0,70000,1500,2,9500,9500 +52,0,1.2,0,3550,52,54,56,58,60,610,91,1,0,62,64,66,68,2,1,1,1410,1841,1841,0,0,0,4,1881,0,3550,0,71000,1520,3,9510,9510 +53,17,1.3,0,6120,53,55,57,59,61,620,92,2,1,63,65,67,69,3,8,2,1420,-3433,1827,0,0,0,4,1907,0,0,6120,72000,1540,0,14780,9520 +54,0,1.3999999999999999,0,0,54,56,58,60,62,630,93,3,2,64,66,68,70,4,11,1,1430,1813,1813,0,0,0,5,1933,0,0,0,73000,1560,1,9530,9530 +55,0,1.5,0,211700,55,57,59,61,63,640,94,1,0,65,67,69,71,5,12,-9,1440,1799,1799,2054,0,2054,1,1959,208000,3700,0,74000,1580,2,9540,9540 +56,0,1.6000000000000001,65,210065,56,58,60,62,64,650,95,2,1,66,68,70,72,1,1,-9,1450,-3290,1985,15500,15500,0,2,1985,210000,0,0,75000,1600,3,14825,9550 +57,21,1,0,219140,57,59,61,63,65,660,96,3,2,67,69,71,73,2,8,-9,1460,1971,1971,15600,15600,0,3,2011,212000,0,6360,76000,1620,0,9560,9560 +58,0,1.1000000000000001,0,3850,58,60,62,64,66,670,97,1,0,68,70,72,74,3,11,1,1470,1957,1957,0,0,0,4,2037,0,3850,0,77000,1640,1,9570,9570 +59,0,1.2,0,0,59,61,63,65,67,680,98,2,1,69,71,73,75,4,12,2,1480,-3347,1943,0,0,0,4,2063,0,0,0,78000,1660,2,14870,9580 +60,0,1.3,0,0,60,62,64,66,68,690,99,3,2,70,72,74,76,5,1,1,1490,1929,1929,0,0,0,5,2089,0,0,0,79000,1680,3,9590,9590 +61,0,1.3999999999999999,70,230670,61,63,65,67,69,700,100,1,0,71,73,75,77,1,8,-9,1500,2115,2115,2060,0,2060,1,2115,220000,4000,6600,80000,1700,0,9600,9600 +62,0,1.5,0,222000,62,64,66,68,70,710,101,2,1,72,74,76,78,2,11,-9,1510,-3204,2101,16100,16100,0,2,2141,222000,0,0,81000,1720,1,14915,9610 +63,7,1.6000000000000001,0,224000,63,65,67,69,71,720,102,3,2,73,75,77,79,3,12,-9,1520,2087,2087,16200,16200,0,3,2167,224000,0,0,82000,1740,2,9620,9620 +64,0,1,0,4965,64,66,68,70,72,730,103,1,0,74,76,78,80,4,1,1,1530,2073,2073,0,0,0,4,2193,0,4150,0,83000,1760,3,9630,9630 diff --git a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/uk_spine.json b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/uk_spine.json index 03603c243..933b17ae0 100644 --- a/packages/microcosm-graph/tests/fixtures/parity/uk_spine/uk_spine.json +++ b/packages/microcosm-graph/tests/fixtures/parity/uk_spine/uk_spine.json @@ -1 +1 @@ -{"country":"uk","nodes":[{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Load the source-bound UK FRS root population.","id":"create_uk_frs","inputs":[],"kernel":"uk.create@1","mass":"conserve","outputs":[{"column":"age","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"gender","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"marital_status","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"hours_worked","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"care_hours","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"is_household_head","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_benunit_head","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_parent","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_uc_claimant","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"self_employment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"private_pension_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"tax_free_savings_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"savings_interest_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"dividend_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"property_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"maintenance_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"miscellaneous_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"private_transfer_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"lump_sum_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"student_loan_repayments","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"statutory_sick_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"statutory_maternity_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"student_loans","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"access_fund","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"education_grants","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"healthy_start_vouchers","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"free_school_breakfasts","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"free_school_fruit_veg","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"free_school_meals","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"maintenance_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"childcare_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"personal_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"salary_sacrifice_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"salary_sacrifice_asked","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"child_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"income_support_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"attendance_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_sc_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_m_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"iidb_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"carers_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"would_claim_carers_allowance","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"sda_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"afcs_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"ssmg_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pension_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"child_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"working_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"state_pension_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"winter_fuel_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"incapacity_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_m_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_dl_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"jsa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"jsa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"bsp_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"frs_benunit_capital","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"is_married","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"dependent_children","dtype":"int64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"region","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"tenure_type","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"accommodation_type","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"num_bedrooms","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_reported","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_band","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_rebate","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_single_adult_raw","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"water_and_sewerage_charges","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"domestic_rates","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"rent","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"subrent","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"mortgage_interest_repayment","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"mortgage_capital_repayment","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"structural_insurance_payments","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"housing_service_charges","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"external_child_payments","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"sample_fraction":1.0,"sample_seed":578,"stage_contract_sha256":"047d30937efb21e39259fdbed5597751e4f5a0c341ea583c7446e4c9c11eba48","time_period":"2024"},"population":null,"sources":["frs"],"structural":"create","weights":null},{"base":"create_uk_frs","citation":"","description":"Ownership boundary for the source-assembling root stage.","id":"frs_spine.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"base":null,"citation":"","description":"Claim the cells assembled by the UK FRS root transform.","id":"frs_spine","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"age","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"gender","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"marital_status","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hours_worked","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"care_hours","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_household_head","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_benunit_head","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_parent","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_uc_claimant","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"self_employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"private_pension_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"tax_free_savings_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"savings_interest_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dividend_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"property_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"maintenance_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"miscellaneous_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"private_transfer_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"lump_sum_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"student_loan_repayments","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"statutory_sick_pay","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"statutory_maternity_pay","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"student_loans","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"access_fund","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"education_grants","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"healthy_start_vouchers","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"free_school_breakfasts","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"free_school_fruit_veg","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"free_school_meals","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"maintenance_expenses","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"childcare_expenses","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"personal_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"salary_sacrifice_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"salary_sacrifice_asked","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"income_support_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"attendance_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_sc_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"iidb_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"carers_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"would_claim_carers_allowance","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"sda_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"afcs_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"ssmg_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"working_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"state_pension_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"winter_fuel_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"incapacity_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_dl_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"bsp_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"frs_benunit_capital","dtype":"float64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_married","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dependent_children","dtype":"int64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"region","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"tenure_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"accommodation_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"num_bedrooms","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_reported","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_band","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_rebate","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_single_adult_raw","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"water_and_sewerage_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"domestic_rates","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"rent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"subrent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_interest_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_capital_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"structural_insurance_payments","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_service_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"external_child_payments","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"}],"params":{},"population":"frs_spine.boundary","sources":[],"structural":"none","weights":null},{"base":"frs_spine.boundary","citation":"","description":"Ownership boundary before age_tail rewrites.","id":"age_tail.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage age_tail.","id":"age_tail","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["external_child_payments","region"],"entity":"household","rows":"all"},{"columns":["gender"],"entity":"person","rows":"all"}],"kernel":"uk.stage.age_tail@1","mass":"conserve","outputs":[{"column":"age","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"age_tail","stage_contract_sha256":"963a34f38caede2525e3b8735f904bb70a1bbef5a272f0eefd4346343444484f","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_relationships.","id":"frs_relationships","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","is_household_head"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_relationships@1","mass":"conserve","outputs":[{"column":"relationship_to_head","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"ons_family_role","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"ons_family_index","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"ons_household_type","dtype":"string","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_relationships","stage_contract_sha256":"30628808d06d9829ebcbcf460cd355a01261e78c7c73e6006c75bec2e3d60c33","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_employment.","id":"frs_employment","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["ons_household_type","region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_employment@1","mass":"conserve","outputs":[{"column":"employment_status","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"employment_sector","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"sic_industry_division","dtype":"int64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_employment","stage_contract_sha256":"ddbefaf05b44788d794a6e4b0e8926c14318d66e50d2f11f50ca549538bbf60c","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_council_tax.","id":"frs_council_tax","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","sic_industry_division"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_council_tax@1","mass":"conserve","outputs":[{"column":"council_tax","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_council_tax","stage_contract_sha256":"3381cd9f7a736514d5073c57480e07f5098890f3ca9ed3865392043594771eb9","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_disability.","id":"frs_disability","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["council_tax","region"],"entity":"household","rows":"all"},{"columns":["afcs_reported","age","attendance_allowance_reported","dla_m_reported","dla_sc_reported","esa_contrib_reported","esa_income_reported","iidb_reported","incapacity_benefit_reported","pip_dl_reported","pip_m_reported","sda_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_disability@1","mass":"conserve","outputs":[{"column":"aa_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_sc_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_m_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_m_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_dl_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"is_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_enhanced_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_severely_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_disability","stage_contract_sha256":"ac500b4cb1bc04d4fb72d198592dd376d61da8c150921a6ac53914ac18be0508","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_education.","id":"frs_education","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","esa_contrib_reported","esa_income_reported","is_severely_disabled_for_benefits","jsa_contrib_reported","jsa_income_reported","universal_credit_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_education@1","mass":"conserve","outputs":[{"column":"current_education","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"highest_education","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"is_in_non_advanced_education","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_in_approved_training","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"age_started_or_accepted_current_education_or_training","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"is_before_universal_credit_qualifying_young_person_terminal_date","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"adult_ema","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"child_ema","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"receives_benefits_in_own_right","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_education","stage_contract_sha256":"836cb0f5a582fee1425190e96c9cb81bdef859bd236c8b6bc27660b6e3d0c2f8","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_legacy_proxies.","id":"frs_legacy_proxies","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_legacy_proxies@1","mass":"conserve","outputs":[{"column":"legacy_jobseeker_proxy","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_health_condition_proxy","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_support_group_proxy","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_legacy_proxies","stage_contract_sha256":"1ecf761cf3fa7180da15659e138e67a8654e58272b4fde28344aa27a874ad0aa","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"age_tail.boundary","citation":"","description":"Ownership boundary before frs_education_grant_split rewrites.","id":"frs_education_grant_split.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_education_grant_split.","id":"frs_education_grant_split","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_education_grant_split@1","mass":"conserve","outputs":[{"column":"disabled_students_allowance_eligible_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"education_grants","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_education_grant_split","stage_contract_sha256":"8718f014b90498c2cc7c754289775ca4a41452d9b978d8cbc5f34b64cbdc7df2","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_take_up.","id":"frs_take_up","inputs":[{"columns":["frs_benunit_capital","is_married"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","child_benefit_reported","education_grants","pension_credit_reported","universal_credit_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_take_up@1","mass":"conserve","outputs":[{"column":"would_claim_child_benefit","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"child_benefit_opts_out","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_pc","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_uc","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_tfc","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_extended_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_universal_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_targeted_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_uc_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"maximum_extended_childcare_hours_usage","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_take_up","stage_contract_sha256":"067d31b7fbe733d1d182588705e9ce79df0c8e5f6e8f519314765af49b1b00d7","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_person_draws.","id":"frs_person_draws","inputs":[{"columns":["frs_benunit_capital","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_person_draws@1","mass":"conserve","outputs":[{"column":"would_claim_marriage_allowance","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"would_claim_scp","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"attends_private_school_random_draw","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"tax_free_childcare_spend_routed_share","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_person_draws","stage_contract_sha256":"5b9b4b7ad87cdc16f96a7f2ee0794946a308011b529956d8adf8ab53d704ede2","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_household_draws.","id":"frs_household_draws","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","tax_free_childcare_spend_routed_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_household_draws@1","mass":"conserve","outputs":[{"column":"household_owns_tv","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"would_evade_tv_licence_fee","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"main_residential_property_purchased_is_first_home","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"property_purchased","dtype":"bool","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_household_draws","stage_contract_sha256":"c74c63b09264319c4bf0049dabba00ecd0ce660beb7b53d6dae71518434bc949","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_brma.","id":"frs_brma","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_brma@1","mass":"conserve","outputs":[{"column":"brma","dtype":"string","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_brma","stage_contract_sha256":"5b8f0b361310c7cd3efbaae3762b347649b7d4afb9e9943131d9bef9266fc3c8","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"frs_education_grant_split.boundary","citation":"","description":"Freeze the assembled-spine gate population.","id":"frs_brma.checkpoint","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_hmrc_spine_leaves.","id":"frs_hmrc_spine_leaves","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["brma","region"],"entity":"household","rows":"all"},{"columns":["age","employment_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_hmrc_spine_leaves@1","mass":"conserve","outputs":[{"column":"hmrc_spi_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_unemployment_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_incapacity_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"ossben_identifiable_subset","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"srp_regular_code5","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"employer_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_hmrc_spine_leaves","stage_contract_sha256":"8cfab42bcc250fdf1161b22e4fd988fbe8a0473af78f99afea4b1017556633ae","time_period":"2024"},"population":"frs_brma.checkpoint","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"frs_brma.checkpoint","citation":"","description":"Run structural UK stage spi_support_channel.","id":"spi_support_channel","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.spi_support_channel@1","mass":"declared","outputs":[],"params":{"expand_cells":[["person","person_source_id","int64"],["person","person_support_channel","string"],["person","person_support_clone_index","int64"],["benunit","benunit_source_id","int64"],["benunit","benunit_support_channel","string"],["benunit","benunit_support_clone_index","int64"],["household","source_household_id","int64"],["household","source_year","int64"],["household","source_household_key","string"],["household","household_source_id","int64"],["household","household_support_channel","string"],["household","household_support_clone_index","int64"],["household","household_is_spi_synthetic","bool"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"spi_support_channel","stage_contract_sha256":"2c682605139c0a8f96e8ba45a2c03f506d3f758ef437841066a8fdb8b74269b6","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by spi_support_channel.","id":"spi_support_channel.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"person_source_id","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"person_support_channel","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"person_support_clone_index","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"benunit_source_id","dtype":"int64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"benunit_support_channel","dtype":"string","entity":"benunit","ownership":"produced","rows":"all"},{"column":"benunit_support_clone_index","dtype":"int64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"source_household_id","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"source_year","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"source_household_key","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"household_source_id","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"household_support_channel","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"household_support_clone_index","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"household_is_spi_synthetic","dtype":"bool","entity":"household","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["person.person_source_id","person.person_support_channel","person.person_support_clone_index","benunit.benunit_source_id","benunit.benunit_support_channel","benunit.benunit_support_clone_index","household.source_household_id","household.source_year","household.source_household_key","household.household_source_id","household.household_support_channel","household.household_support_clone_index","household.household_is_spi_synthetic"]},"population":"spi_support_channel","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"spi_support_channel","citation":"","description":"Run structural UK stage spi_income_band_donors.","id":"spi_income_band_donors","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.spi_income_band_donors@1","mass":"free","outputs":[],"params":{"expand_cells":[["person","person_source_id","int64"],["person","person_support_channel","string"],["person","person_support_clone_index","int64"],["benunit","benunit_source_id","int64"],["benunit","benunit_support_channel","string"],["benunit","benunit_support_clone_index","int64"],["household","household_source_id","int64"],["household","household_support_channel","string"],["household","household_support_clone_index","int64"],["household","household_is_spi_synthetic","bool"],["household","household_is_spi_income_band_donor","bool"],["household","spi_income_band_donor_lower_bound","float64"],["person","person_is_spi_income_band_carrier","bool"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"spi_income_band_donors","stage_contract_sha256":"190a40ab145b001993733553f45b3e8e5adfe999f598231481ab6706447d5f3c","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by spi_income_band_donors.","id":"spi_income_band_donors.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"person_source_id","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"person_support_channel","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"person_support_clone_index","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"benunit_source_id","dtype":"int64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"benunit_support_channel","dtype":"string","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"benunit_support_clone_index","dtype":"int64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_source_id","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_support_channel","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_support_clone_index","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_is_spi_synthetic","dtype":"bool","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_is_spi_income_band_donor","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"spi_income_band_donor_lower_bound","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"person_is_spi_income_band_carrier","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_spi_income_band_donor","household.spi_income_band_donor_lower_bound","person.person_is_spi_income_band_carrier"]},"population":"spi_income_band_donors","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage hmrc_spi_income_spine.","id":"hmrc_spi_income_spine","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","maintenance_expenses","childcare_expenses","salary_sacrifice_reported","salary_sacrifice_asked","ssmg_reported","incapacity_benefit_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","ossben_identifiable_subset","srp_regular_code5","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier"],"entity":"person","rows":"all"}],"kernel":"uk.stage.hmrc_spi_income_spine@1","mass":"conserve","outputs":[{"column":"charitable_investment_gifts","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"gift_aid","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"other_investment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_employment_benefits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_employment_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_other_social_security_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_taxable_termination_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_miscellaneous_employment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_other_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_state_pension_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_employed_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_total_earned_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_total_investment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_assessable_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"self_employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"savings_interest_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dividend_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"private_pension_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"property_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employer_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"personal_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"tax_free_savings_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"income_support_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"working_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"attendance_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"state_pension_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_sc_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_dl_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"sda_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"carers_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"iidb_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"afcs_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"bsp_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"winter_fuel_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hmrc_spi_pay","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hmrc_spi_unemployment_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hmrc_spi_incapacity_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"aa_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_sc_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_m_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_m_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_dl_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_enhanced_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_severely_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"would_claim_carers_allowance","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"hmrc_spi_income_spine","stage_contract_sha256":"834334536c1c6add6648b801272c61ca2f597a54c82553a943667200b9e6c4ce","time_period":"2024"},"population":"spi_income_band_donors","sources":["frs"],"structural":"none","weights":null},{"base":"spi_income_band_donors","citation":"","description":"Ownership boundary before spi_housing_shell rewrites.","id":"spi_housing_shell.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage spi_housing_shell.","id":"spi_housing_shell","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["council_tax_single_adult_raw","household_support_channel","ons_household_type","region"],"entity":"household","rows":"all"},{"columns":["age","dividend_income","employment_income","is_household_head","other_investment_income","private_pension_income","property_income","savings_interest_income","self_employment_income","state_pension_reported","would_claim_carers_allowance"],"entity":"person","rows":"all"}],"kernel":"uk.stage.spi_housing_shell@1","mass":"conserve","outputs":[{"column":"tenure_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"accommodation_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_band","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"num_bedrooms","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_reported","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"rent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_interest_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_capital_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"structural_insurance_payments","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_service_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"water_and_sewerage_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"domestic_rates","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"subrent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_rebate","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"spi_housing_shell","stage_contract_sha256":"809f19fd33ff79ceda005e8cc3fa89ae40b297dceca26a7658b6259267987890","time_period":"2024"},"population":"spi_housing_shell.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage was_wealth.","id":"was_wealth","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.was_wealth@1","mass":"conserve","outputs":[{"column":"owned_land","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"property_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"corporate_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"private_pension_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"gross_financial_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"net_financial_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"main_residence_value","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"other_residential_property_value","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"non_residential_property_value","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"savings","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"num_vehicles","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"cash_isa","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"stocks_and_shares_isa","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"mortgage_debt","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"consumer_debt","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"student_loan_balance","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"was_wealth","stage_contract_sha256":"366e019c41ee2a28bcd61bbaa62a024ed0bb4b30b3ffc4d87ab23b340b4076bf","time_period":"2024"},"population":"spi_housing_shell.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage was_lisa.","id":"was_lisa","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance"],"entity":"person","rows":"all"}],"kernel":"uk.stage.was_lisa@1","mass":"conserve","outputs":[{"column":"has_lifetime_isa","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"lifetime_isa_balance","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"household_lifetime_isa_balance","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"was_lisa","stage_contract_sha256":"f1d0f962e84d32007821b32e4e03a8af9443acc0ed3d1836ff094dcbacb928da","time_period":"2024"},"population":"spi_housing_shell.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"spi_housing_shell.boundary","citation":"","description":"Ownership boundary before regional_property_uprating rewrites.","id":"regional_property_uprating.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage regional_property_uprating.","id":"regional_property_uprating","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["household_lifetime_isa_balance","household_support_channel","region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.regional_property_uprating@1","mass":"conserve","outputs":[{"column":"main_residence_value","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"property_wealth","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"regional_property_uprating","stage_contract_sha256":"acb342e8ca1770556c5682a9411c5ead5d61dd1420cfe6b0305ed14a3407fb25","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage nts_bus_travel.","id":"nts_bus_travel","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance"],"entity":"person","rows":"all"}],"kernel":"uk.stage.nts_bus_travel@1","mass":"conserve","outputs":[{"column":"local_bus_use_band","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"bus_in_london_trips","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"other_local_bus_trips","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"local_bus_trips","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"bus_pass_eligible","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"local_bus_single_fare_share","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"household_local_bus_trips","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"nts_bus_travel","stage_contract_sha256":"59767eeb246755f4636e2abbbc9ab3370a64c6142e01cda21a12cb01be95e535","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage lcfs_consumption.","id":"lcfs_consumption","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.lcfs_consumption@1","mass":"conserve","outputs":[{"column":"food_and_non_alcoholic_beverages_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"alcohol_and_tobacco_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"clothing_and_footwear_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"housing_water_and_electricity_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"household_furnishings_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"health_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"transport_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"communication_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"recreation_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"education_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"restaurants_and_hotels_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"miscellaneous_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"petrol_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"diesel_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"bus_fare_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"domestic_energy_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"electricity_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"gas_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"has_fuel_consumption","dtype":"bool","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"lcfs_consumption","stage_contract_sha256":"f428215b7d66e1297688cfc8dce94e670268fb26a6681ca5f0e6e0641cfaf076","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage etb_vat.","id":"etb_vat","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.etb_vat@1","mass":"conserve","outputs":[{"column":"full_rate_vat_expenditure_rate","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"etb_vat","stage_contract_sha256":"5a2b777d88315bcccb72477c088a136c634131e0bc1a339b7287217ee5cfd8d3","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage etb_services.","id":"etb_services","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.etb_services@1","mass":"conserve","outputs":[{"column":"dfe_education_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"rail_subsidy_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"bus_subsidy_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"rail_usage","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"a_and_e_visits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"admitted_patient_visits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"outpatient_visits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"nhs_a_and_e_spending","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"nhs_admitted_patient_spending","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"nhs_outpatient_spending","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"etb_services","stage_contract_sha256":"991be160f2f2c28cf7eadc911b3214aca8bf5be441676fe21a87ba12888c5c25","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"regional_property_uprating.boundary","citation":"","description":"Ownership boundary before uc_reporter_redraw rewrites.","id":"uc_reporter_redraw.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage uc_reporter_redraw.","id":"uc_reporter_redraw","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.uc_reporter_redraw@1","mass":"conserve","outputs":[{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"uc_reporter_redraw","stage_contract_sha256":"7e94e034cad29c5bae566ffb875c42cddf3a1025c15496907e783ceff1144f8b","time_period":"2024"},"population":"uc_reporter_redraw.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"uc_reporter_redraw.boundary","citation":"","description":"Ownership boundary before uc_capital_coherence rewrites.","id":"uc_capital_coherence.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage uc_capital_coherence.","id":"uc_capital_coherence","inputs":[{"columns":["benunit_support_channel","dependent_children","is_married"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","is_benunit_head","is_parent","person_support_channel","universal_credit_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.uc_capital_coherence@1","mass":"conserve","outputs":[{"column":"uc_reported_capital","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"frs_benunit_capital","dtype":"float64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"would_claim_uc","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"uc_capital_coherence","stage_contract_sha256":"059f27ee687ee15385b246d43068b17aeecbe3645ccc41fd52fcad06a006a081","time_period":"2024"},"population":"uc_capital_coherence.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"uc_capital_coherence.boundary","citation":"","description":"Ownership boundary before pension_credit_take_up rewrites.","id":"pension_credit_take_up.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage pension_credit_take_up.","id":"pension_credit_take_up","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.pension_credit_take_up@1","mass":"conserve","outputs":[{"column":"would_claim_pc","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"pension_credit_take_up","stage_contract_sha256":"5f2f0410ec296be7119d7f58365658e72ee46b5e4397c37f97ce01e911a98960","time_period":"2024"},"population":"pension_credit_take_up.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage uc_deduction_attributes.","id":"uc_deduction_attributes","inputs":[{"columns":["frs_benunit_capital","would_claim_pc"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.uc_deduction_attributes@1","mass":"conserve","outputs":[{"column":"uc_deduction_random_draw","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"uc_deduction_type_random_draw","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"uc_latent_deduction_rate","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"uc_deduction_combination","dtype":"string","entity":"benunit","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"uc_deduction_attributes","stage_contract_sha256":"6bc7e3ec1a5712e23ca1b7d2956e8b17f2526b94431ba3e5a9294b04658e4fe0","time_period":"2024"},"population":"pension_credit_take_up.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"pension_credit_take_up.boundary","citation":"","description":"Run structural UK stage cgt_support_split.","id":"cgt_support_split","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_support_split@1","mass":"conserve","outputs":[],"params":{"expand_cells":[["household","household_is_cgt_support_copy","bool"],["household","cgt_support_copies","int64"],["household","cgt_support_copy_index","int64"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_support_split","stage_contract_sha256":"30b37e1c4e122db18391c5f76f1a6d432aea970275bcda7a52126dacc923490b","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by cgt_support_split.","id":"cgt_support_split.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"household_is_cgt_support_copy","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"cgt_support_copies","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"cgt_support_copy_index","dtype":"int64","entity":"household","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_cgt_support_copy","household.cgt_support_copies","household.cgt_support_copy_index"]},"population":"cgt_support_split","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"cgt_support_split","citation":"","description":"Run structural UK stage cgt_incidence_clone.","id":"cgt_incidence_clone","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_incidence_clone@1","mass":"conserve","outputs":[],"params":{"expand_cells":[["household","household_is_capital_gains_clone","bool"],["person","capital_gains","float64"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_incidence_clone","stage_contract_sha256":"ee30278543cc0297a5366d855b7753aa04af5518971594c193c52be36bf8a7b4","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by cgt_incidence_clone.","id":"cgt_incidence_clone.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"household_is_capital_gains_clone","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"capital_gains","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_capital_gains_clone","person.capital_gains"]},"population":"cgt_incidence_clone","sources":[],"structural":"none","weights":null},{"base":"cgt_incidence_clone","citation":"","description":"Ownership boundary before hmrc_cgt_gains_spine rewrites.","id":"hmrc_cgt_gains_spine.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index","household_is_capital_gains_clone"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending","capital_gains"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage hmrc_cgt_gains_spine.","id":"hmrc_cgt_gains_spine","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["cgt_support_copies","corporate_wealth","gross_financial_wealth","household_is_cgt_support_copy","non_residential_property_value","other_residential_property_value","region"],"entity":"household","rows":"all"},{"columns":["age","dividend_income","employment_income","miscellaneous_income","private_pension_income","property_income","savings_interest_income","self_employment_income","state_pension_reported","tax_free_savings_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.hmrc_cgt_gains_spine@1","mass":"conserve","outputs":[{"column":"capital_gains","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"hmrc_cgt_gains_spine","stage_contract_sha256":"ff740a36b03dcda73b889aa43bde7cff5d178de7b3da38e487320c68342bf9a3","time_period":"2024"},"population":"hmrc_cgt_gains_spine.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage hmrc_cgt_asset_type_spine.","id":"hmrc_cgt_asset_type_spine","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["corporate_wealth","gross_financial_wealth","other_residential_property_value","region","stocks_and_shares_isa"],"entity":"household","rows":"all"},{"columns":["age","capital_gains","dividend_income","property_income","self_employment_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.hmrc_cgt_asset_type_spine@1","mass":"conserve","outputs":[{"column":"capital_gains_asset_type","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"capital_gains_residential_property","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"capital_gains_badr","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"hmrc_cgt_asset_type_spine","stage_contract_sha256":"c918223197c706279b37d1b32d0ff3885fb1cd7b65b2e50d0308ab508ed0ed64","time_period":"2024"},"population":"hmrc_cgt_gains_spine.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"hmrc_cgt_gains_spine.boundary","citation":"","description":"Run structural UK stage cgt_incidence_anchor.","id":"cgt_incidence_anchor","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["household_is_capital_gains_clone","region"],"entity":"household","rows":"all"},{"columns":["age","capital_gains","capital_gains_badr","dividend_income","employment_income","miscellaneous_income","private_pension_income","property_income","savings_interest_income","self_employment_income","state_pension_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_incidence_anchor@1","mass":"conserve","outputs":[],"params":{"expand_cells":[],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_incidence_anchor","stage_contract_sha256":"b79ce806015de2f1e1703fa68bf4ad871c6df54b5b32633c864632ab68919a48","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage salary_sacrifice.","id":"salary_sacrifice","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index","household_is_capital_gains_clone"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending","capital_gains","capital_gains_asset_type","capital_gains_residential_property","capital_gains_badr"],"entity":"person","rows":"all"}],"kernel":"uk.stage.salary_sacrifice@1","mass":"conserve","outputs":[{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"salary_sacrifice","stage_contract_sha256":"b4581ac43437c170dbe6369392630a832c1f18e99c113753984be6e57b5b18cd","time_period":"2024"},"population":"cgt_incidence_anchor","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage student_loans.","id":"student_loans","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","current_education","employment_income","highest_education","student_loan_repayments","student_loans"],"entity":"person","rows":"all"}],"kernel":"uk.stage.student_loans@1","mass":"conserve","outputs":[{"column":"student_loan_plan","dtype":"string","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"student_loans","stage_contract_sha256":"f2936c5e22a0e9b097437e3f3aabaf06c4561d5c25a9b7c05e720a6553f73bbc","time_period":"2024"},"population":"cgt_incidence_anchor","sources":["frs"],"structural":"none","weights":null}],"sources":[{"codec":"csv-tables","description":"Content-bound UK FRS and donor fixture/source bundle.","name":"frs"}]} +{"country":"uk","nodes":[{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Load the source-bound UK FRS root population.","id":"create_uk_frs","inputs":[],"kernel":"uk.create@1","mass":"conserve","outputs":[{"column":"age","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"gender","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"marital_status","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"hours_worked","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"care_hours","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"is_household_head","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_benunit_head","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_parent","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_uc_claimant","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"self_employment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"private_pension_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"tax_free_savings_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"savings_interest_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"dividend_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"property_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"maintenance_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"miscellaneous_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"private_transfer_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"lump_sum_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"student_loan_repayments","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"statutory_sick_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"statutory_maternity_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"student_loans","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"access_fund","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"education_grants","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"healthy_start_vouchers","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"free_school_breakfasts","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"free_school_fruit_veg","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"free_school_meals","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"maintenance_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"childcare_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"personal_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"salary_sacrifice_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"salary_sacrifice_asked","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"child_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"income_support_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"attendance_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_sc_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_m_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"iidb_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"carers_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"would_claim_carers_allowance","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"sda_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"afcs_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"ssmg_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pension_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"child_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"working_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"state_pension_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"winter_fuel_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"incapacity_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_m_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_dl_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"jsa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"jsa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"bsp_reported","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"frs_benunit_capital","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"is_married","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"dependent_children","dtype":"int64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"region","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"tenure_type","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"accommodation_type","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"num_bedrooms","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_reported","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_band","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_rebate","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"council_tax_single_adult_raw","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"water_and_sewerage_charges","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"domestic_rates","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"rent","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"subrent","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"mortgage_interest_repayment","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"mortgage_capital_repayment","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"structural_insurance_payments","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"housing_service_charges","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"external_child_payments","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"sample_fraction":1.0,"sample_seed":578,"stage_contract_sha256":"047d30937efb21e39259fdbed5597751e4f5a0c341ea583c7446e4c9c11eba48","time_period":"2024"},"population":null,"sources":["frs"],"structural":"create","weights":null},{"base":"create_uk_frs","citation":"","description":"Ownership boundary for the source-assembling root stage.","id":"frs_spine.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"base":null,"citation":"","description":"Claim the cells assembled by the UK FRS root transform.","id":"frs_spine","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"age","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"gender","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"marital_status","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hours_worked","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"care_hours","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_household_head","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_benunit_head","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_parent","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_uc_claimant","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"self_employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"private_pension_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"tax_free_savings_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"savings_interest_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dividend_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"property_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"maintenance_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"miscellaneous_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"private_transfer_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"lump_sum_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"student_loan_repayments","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"statutory_sick_pay","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"statutory_maternity_pay","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"student_loans","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"access_fund","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"education_grants","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"healthy_start_vouchers","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"free_school_breakfasts","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"free_school_fruit_veg","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"free_school_meals","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"maintenance_expenses","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"childcare_expenses","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"personal_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"salary_sacrifice_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"salary_sacrifice_asked","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"income_support_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"attendance_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_sc_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"iidb_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"carers_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"would_claim_carers_allowance","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"sda_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"afcs_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"ssmg_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"working_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"state_pension_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"winter_fuel_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"incapacity_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_dl_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"bsp_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"frs_benunit_capital","dtype":"float64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_married","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dependent_children","dtype":"int64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"region","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"tenure_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"accommodation_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"num_bedrooms","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_reported","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_band","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_rebate","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_single_adult_raw","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"water_and_sewerage_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"domestic_rates","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"rent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"subrent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_interest_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_capital_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"structural_insurance_payments","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_service_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"external_child_payments","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"}],"params":{},"population":"frs_spine.boundary","sources":[],"structural":"none","weights":null},{"base":"frs_spine.boundary","citation":"","description":"Ownership boundary before age_tail rewrites.","id":"age_tail.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage age_tail.","id":"age_tail","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["external_child_payments","region"],"entity":"household","rows":"all"},{"columns":["gender"],"entity":"person","rows":"all"}],"kernel":"uk.stage.age_tail@1","mass":"conserve","outputs":[{"column":"age","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"age_tail","stage_contract_sha256":"963a34f38caede2525e3b8735f904bb70a1bbef5a272f0eefd4346343444484f","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_relationships.","id":"frs_relationships","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","is_household_head"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_relationships@1","mass":"conserve","outputs":[{"column":"relationship_to_head","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"ons_family_role","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"ons_family_index","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"ons_household_type","dtype":"string","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_relationships","stage_contract_sha256":"30628808d06d9829ebcbcf460cd355a01261e78c7c73e6006c75bec2e3d60c33","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_employment.","id":"frs_employment","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["ons_household_type","region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_employment@1","mass":"conserve","outputs":[{"column":"employment_status","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"employment_sector","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"sic_industry_division","dtype":"int64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_employment","stage_contract_sha256":"ddbefaf05b44788d794a6e4b0e8926c14318d66e50d2f11f50ca549538bbf60c","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_council_tax.","id":"frs_council_tax","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","sic_industry_division"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_council_tax@1","mass":"conserve","outputs":[{"column":"council_tax","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_council_tax","stage_contract_sha256":"3381cd9f7a736514d5073c57480e07f5098890f3ca9ed3865392043594771eb9","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_disability.","id":"frs_disability","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["council_tax","region"],"entity":"household","rows":"all"},{"columns":["afcs_reported","age","attendance_allowance_reported","dla_m_reported","dla_sc_reported","esa_contrib_reported","esa_income_reported","iidb_reported","incapacity_benefit_reported","pip_dl_reported","pip_m_reported","sda_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_disability@1","mass":"conserve","outputs":[{"column":"aa_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_sc_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"dla_m_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_m_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"pip_dl_category","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"is_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_enhanced_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_severely_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_disability","stage_contract_sha256":"ac500b4cb1bc04d4fb72d198592dd376d61da8c150921a6ac53914ac18be0508","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_education.","id":"frs_education","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","esa_contrib_reported","esa_income_reported","is_severely_disabled_for_benefits","jsa_contrib_reported","jsa_income_reported","universal_credit_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_education@1","mass":"conserve","outputs":[{"column":"current_education","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"highest_education","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"is_in_non_advanced_education","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"is_in_approved_training","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"age_started_or_accepted_current_education_or_training","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"is_before_universal_credit_qualifying_young_person_terminal_date","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"adult_ema","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"child_ema","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"receives_benefits_in_own_right","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_education","stage_contract_sha256":"836cb0f5a582fee1425190e96c9cb81bdef859bd236c8b6bc27660b6e3d0c2f8","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_legacy_proxies.","id":"frs_legacy_proxies","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_legacy_proxies@1","mass":"conserve","outputs":[{"column":"legacy_jobseeker_proxy","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_health_condition_proxy","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"esa_support_group_proxy","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_legacy_proxies","stage_contract_sha256":"1ecf761cf3fa7180da15659e138e67a8654e58272b4fde28344aa27a874ad0aa","time_period":"2024"},"population":"age_tail.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"age_tail.boundary","citation":"","description":"Ownership boundary before frs_education_grant_split rewrites.","id":"frs_education_grant_split.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_education_grant_split.","id":"frs_education_grant_split","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_education_grant_split@1","mass":"conserve","outputs":[{"column":"disabled_students_allowance_eligible_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"education_grants","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_education_grant_split","stage_contract_sha256":"8718f014b90498c2cc7c754289775ca4a41452d9b978d8cbc5f34b64cbdc7df2","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_take_up.","id":"frs_take_up","inputs":[{"columns":["frs_benunit_capital","is_married"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","child_benefit_reported","education_grants","pension_credit_reported","universal_credit_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_take_up@1","mass":"conserve","outputs":[{"column":"would_claim_child_benefit","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"child_benefit_opts_out","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_pc","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_uc","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_tfc","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_extended_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_universal_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_targeted_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"would_claim_uc_childcare","dtype":"bool","entity":"benunit","ownership":"produced","rows":"all"},{"column":"maximum_extended_childcare_hours_usage","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_take_up","stage_contract_sha256":"067d31b7fbe733d1d182588705e9ce79df0c8e5f6e8f519314765af49b1b00d7","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_person_draws.","id":"frs_person_draws","inputs":[{"columns":["frs_benunit_capital","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_person_draws@1","mass":"conserve","outputs":[{"column":"would_claim_marriage_allowance","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"would_claim_scp","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"attends_private_school_random_draw","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"tax_free_childcare_spend_routed_share","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_person_draws","stage_contract_sha256":"e840b69cb05b5cbb76b97b4fcb296244ea843a005e4882459c7132664bb526c2","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_household_draws.","id":"frs_household_draws","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","tax_free_childcare_spend_routed_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_household_draws@1","mass":"conserve","outputs":[{"column":"household_owns_tv","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"would_evade_tv_licence_fee","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"main_residential_property_purchased_is_first_home","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"property_purchased","dtype":"bool","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_household_draws","stage_contract_sha256":"c74c63b09264319c4bf0049dabba00ecd0ce660beb7b53d6dae71518434bc949","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_brma.","id":"frs_brma","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_brma@1","mass":"conserve","outputs":[{"column":"brma","dtype":"string","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_brma","stage_contract_sha256":"5b8f0b361310c7cd3efbaae3762b347649b7d4afb9e9943131d9bef9266fc3c8","time_period":"2024"},"population":"frs_education_grant_split.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"frs_education_grant_split.boundary","citation":"","description":"Freeze the assembled-spine gate population.","id":"frs_brma.checkpoint","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage frs_hmrc_spine_leaves.","id":"frs_hmrc_spine_leaves","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["brma","region"],"entity":"household","rows":"all"},{"columns":["age","employment_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.frs_hmrc_spine_leaves@1","mass":"conserve","outputs":[{"column":"hmrc_spi_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_unemployment_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_incapacity_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"ossben_identifiable_subset","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"srp_regular_code5","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"employer_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"frs_hmrc_spine_leaves","stage_contract_sha256":"8cfab42bcc250fdf1161b22e4fd988fbe8a0473af78f99afea4b1017556633ae","time_period":"2024"},"population":"frs_brma.checkpoint","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"frs_brma.checkpoint","citation":"","description":"Run structural UK stage spi_support_channel.","id":"spi_support_channel","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.spi_support_channel@1","mass":"declared","outputs":[],"params":{"expand_cells":[["person","person_source_id","int64"],["person","person_support_channel","string"],["person","person_support_clone_index","int64"],["benunit","benunit_source_id","int64"],["benunit","benunit_support_channel","string"],["benunit","benunit_support_clone_index","int64"],["household","source_household_id","int64"],["household","source_year","int64"],["household","source_household_key","string"],["household","household_source_id","int64"],["household","household_support_channel","string"],["household","household_support_clone_index","int64"],["household","household_is_spi_synthetic","bool"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"spi_support_channel","stage_contract_sha256":"2c682605139c0a8f96e8ba45a2c03f506d3f758ef437841066a8fdb8b74269b6","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by spi_support_channel.","id":"spi_support_channel.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"person_source_id","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"person_support_channel","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"person_support_clone_index","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"benunit_source_id","dtype":"int64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"benunit_support_channel","dtype":"string","entity":"benunit","ownership":"produced","rows":"all"},{"column":"benunit_support_clone_index","dtype":"int64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"source_household_id","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"source_year","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"source_household_key","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"household_source_id","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"household_support_channel","dtype":"string","entity":"household","ownership":"produced","rows":"all"},{"column":"household_support_clone_index","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"household_is_spi_synthetic","dtype":"bool","entity":"household","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["person.person_source_id","person.person_support_channel","person.person_support_clone_index","benunit.benunit_source_id","benunit.benunit_support_channel","benunit.benunit_support_clone_index","household.source_household_id","household.source_year","household.source_household_key","household.household_source_id","household.household_support_channel","household.household_support_clone_index","household.household_is_spi_synthetic"]},"population":"spi_support_channel","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"spi_support_channel","citation":"","description":"Run structural UK stage spi_income_band_donors.","id":"spi_income_band_donors","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.spi_income_band_donors@1","mass":"declared","outputs":[],"params":{"expand_cells":[["person","person_source_id","int64"],["person","person_support_channel","string"],["person","person_support_clone_index","int64"],["benunit","benunit_source_id","int64"],["benunit","benunit_support_channel","string"],["benunit","benunit_support_clone_index","int64"],["household","household_source_id","int64"],["household","household_support_channel","string"],["household","household_support_clone_index","int64"],["household","household_is_spi_synthetic","bool"],["household","household_is_spi_income_band_donor","bool"],["household","spi_income_band_donor_lower_bound","float64"],["person","person_is_spi_income_band_carrier","bool"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"spi_income_band_donors","stage_contract_sha256":"5145464dd063c46e60ee1d1b2ddfa9cb02ee3c1ad42e1bc3511f6e288e17e67d","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by spi_income_band_donors.","id":"spi_income_band_donors.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"person_source_id","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"person_support_channel","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"person_support_clone_index","dtype":"int64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"benunit_source_id","dtype":"int64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"benunit_support_channel","dtype":"string","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"benunit_support_clone_index","dtype":"int64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_source_id","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_support_channel","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_support_clone_index","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_is_spi_synthetic","dtype":"bool","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"household_is_spi_income_band_donor","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"spi_income_band_donor_lower_bound","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"person_is_spi_income_band_carrier","dtype":"bool","entity":"person","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_spi_income_band_donor","household.spi_income_band_donor_lower_bound","person.person_is_spi_income_band_carrier"]},"population":"spi_income_band_donors","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage hmrc_spi_income_spine.","id":"hmrc_spi_income_spine","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","maintenance_expenses","childcare_expenses","salary_sacrifice_reported","salary_sacrifice_asked","ssmg_reported","incapacity_benefit_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","ossben_identifiable_subset","srp_regular_code5","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier"],"entity":"person","rows":"all"}],"kernel":"uk.stage.hmrc_spi_income_spine@1","mass":"conserve","outputs":[{"column":"charitable_investment_gifts","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"gift_aid","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"other_investment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_employment_benefits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_employment_expenses","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_other_social_security_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_taxable_termination_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_miscellaneous_employment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_other_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_state_pension_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_employed_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_total_earned_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_total_investment_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"hmrc_spi_assessable_income","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"self_employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"savings_interest_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dividend_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"private_pension_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"property_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employer_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"personal_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"tax_free_savings_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pension_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"income_support_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"working_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_tax_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"attendance_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"state_pension_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_sc_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_m_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_dl_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"sda_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"carers_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"iidb_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"afcs_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"bsp_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"winter_fuel_allowance_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"jsa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_contrib_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"esa_income_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hmrc_spi_pay","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hmrc_spi_unemployment_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"hmrc_spi_incapacity_benefit_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"aa_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_sc_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"dla_m_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_m_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"pip_dl_category","dtype":"string","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_enhanced_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"is_severely_disabled_for_benefits","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"would_claim_carers_allowance","dtype":"bool","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"hmrc_spi_income_spine","stage_contract_sha256":"0155bff429399867190021eb7aa5f1cac73b0aea2e2cbddd4e85edc741b9646e","time_period":"2024"},"population":"spi_income_band_donors","sources":["frs"],"structural":"none","weights":null},{"base":"spi_income_band_donors","citation":"","description":"Ownership boundary before spi_housing_shell rewrites.","id":"spi_housing_shell.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage spi_housing_shell.","id":"spi_housing_shell","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["council_tax_single_adult_raw","household_support_channel","ons_household_type","region"],"entity":"household","rows":"all"},{"columns":["age","dividend_income","employment_income","is_household_head","other_investment_income","private_pension_income","property_income","savings_interest_income","self_employment_income","state_pension_reported","would_claim_carers_allowance"],"entity":"person","rows":"all"}],"kernel":"uk.stage.spi_housing_shell@1","mass":"conserve","outputs":[{"column":"tenure_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"accommodation_type","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_band","dtype":"string","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"num_bedrooms","dtype":"int64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_reported","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"rent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_interest_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"mortgage_capital_repayment","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"structural_insurance_payments","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_service_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"water_and_sewerage_charges","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"domestic_rates","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"subrent","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_rebate","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"housing_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"council_tax_benefit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"spi_housing_shell","stage_contract_sha256":"809f19fd33ff79ceda005e8cc3fa89ae40b297dceca26a7658b6259267987890","time_period":"2024"},"population":"spi_housing_shell.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage was_wealth.","id":"was_wealth","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.was_wealth@1","mass":"conserve","outputs":[{"column":"owned_land","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"property_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"corporate_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"private_pension_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"gross_financial_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"net_financial_wealth","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"main_residence_value","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"other_residential_property_value","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"non_residential_property_value","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"savings","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"num_vehicles","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"cash_isa","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"stocks_and_shares_isa","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"mortgage_debt","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"consumer_debt","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"student_loan_balance","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"was_wealth","stage_contract_sha256":"daeb769e5b18086a68d66f288fcc5e3f6180d27d0cb95fea63a0874c6089e876","time_period":"2024"},"population":"spi_housing_shell.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage was_lisa.","id":"was_lisa","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance"],"entity":"person","rows":"all"}],"kernel":"uk.stage.was_lisa@1","mass":"conserve","outputs":[{"column":"has_lifetime_isa","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"lifetime_isa_balance","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"household_lifetime_isa_balance","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"was_lisa","stage_contract_sha256":"46f12fdf9228f721d860ff7858b78b25406bb8737628acf63ac17d44d549f657","time_period":"2024"},"population":"spi_housing_shell.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"spi_housing_shell.boundary","citation":"","description":"Ownership boundary before regional_property_uprating rewrites.","id":"regional_property_uprating.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage regional_property_uprating.","id":"regional_property_uprating","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["household_lifetime_isa_balance","household_support_channel","region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.regional_property_uprating@1","mass":"conserve","outputs":[{"column":"main_residence_value","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"property_wealth","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"owned_land","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"other_residential_property_value","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"},{"column":"non_residential_property_value","dtype":"float64","entity":"household","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"regional_property_uprating","stage_contract_sha256":"10e9b226ee1483c9465bec4c8fff6d2d4387bcdb5611907189c84d1645874ab5","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage nts_bus_travel.","id":"nts_bus_travel","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance"],"entity":"person","rows":"all"}],"kernel":"uk.stage.nts_bus_travel@1","mass":"conserve","outputs":[{"column":"local_bus_use_band","dtype":"int64","entity":"person","ownership":"produced","rows":"all"},{"column":"bus_in_london_trips","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"other_local_bus_trips","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"local_bus_trips","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"bus_pass_eligible","dtype":"bool","entity":"person","ownership":"produced","rows":"all"},{"column":"local_bus_single_fare_share","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"household_local_bus_trips","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"nts_bus_travel","stage_contract_sha256":"59767eeb246755f4636e2abbbc9ab3370a64c6142e01cda21a12cb01be95e535","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage lcfs_consumption.","id":"lcfs_consumption","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.lcfs_consumption@1","mass":"conserve","outputs":[{"column":"food_and_non_alcoholic_beverages_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"alcohol_and_tobacco_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"clothing_and_footwear_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"housing_water_and_electricity_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"household_furnishings_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"health_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"transport_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"communication_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"recreation_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"education_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"restaurants_and_hotels_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"miscellaneous_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"petrol_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"diesel_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"bus_fare_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"domestic_energy_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"electricity_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"gas_consumption","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"has_fuel_consumption","dtype":"bool","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"lcfs_consumption","stage_contract_sha256":"348c64e464d10e32be48843cb0ddf4f357957306439975b860a15679ea3465b4","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage etb_vat.","id":"etb_vat","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.etb_vat@1","mass":"conserve","outputs":[{"column":"full_rate_vat_expenditure_rate","dtype":"float64","entity":"household","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"etb_vat","stage_contract_sha256":"5a2b777d88315bcccb72477c088a136c634131e0bc1a339b7287217ee5cfd8d3","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage etb_services.","id":"etb_services","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share"],"entity":"person","rows":"all"}],"kernel":"uk.stage.etb_services@1","mass":"conserve","outputs":[{"column":"dfe_education_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"rail_subsidy_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"bus_subsidy_spending","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"rail_usage","dtype":"float64","entity":"household","ownership":"produced","rows":"all"},{"column":"a_and_e_visits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"admitted_patient_visits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"outpatient_visits","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"nhs_a_and_e_spending","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"nhs_admitted_patient_spending","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"nhs_outpatient_spending","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"etb_services","stage_contract_sha256":"991be160f2f2c28cf7eadc911b3214aca8bf5be441676fe21a87ba12888c5c25","time_period":"2024"},"population":"regional_property_uprating.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"regional_property_uprating.boundary","citation":"","description":"Ownership boundary before uc_reporter_redraw rewrites.","id":"uc_reporter_redraw.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage uc_reporter_redraw.","id":"uc_reporter_redraw","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.uc_reporter_redraw@1","mass":"conserve","outputs":[{"column":"universal_credit_reported","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"uc_reporter_redraw","stage_contract_sha256":"7e94e034cad29c5bae566ffb875c42cddf3a1025c15496907e783ceff1144f8b","time_period":"2024"},"population":"uc_reporter_redraw.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"uc_reporter_redraw.boundary","citation":"","description":"Ownership boundary before uc_capital_coherence rewrites.","id":"uc_capital_coherence.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage uc_capital_coherence.","id":"uc_capital_coherence","inputs":[{"columns":["benunit_support_channel","dependent_children","is_married"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","is_benunit_head","is_parent","person_support_channel","universal_credit_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.uc_capital_coherence@1","mass":"conserve","outputs":[{"column":"uc_reported_capital","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"frs_benunit_capital","dtype":"float64","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"would_claim_uc","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"uc_capital_coherence","stage_contract_sha256":"059f27ee687ee15385b246d43068b17aeecbe3645ccc41fd52fcad06a006a081","time_period":"2024"},"population":"uc_capital_coherence.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"uc_capital_coherence.boundary","citation":"","description":"Ownership boundary before pension_credit_take_up rewrites.","id":"pension_credit_take_up.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage pension_credit_take_up.","id":"pension_credit_take_up","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.pension_credit_take_up@1","mass":"conserve","outputs":[{"column":"would_claim_pc","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"pension_credit_take_up","stage_contract_sha256":"5f2f0410ec296be7119d7f58365658e72ee46b5e4397c37f97ce01e911a98960","time_period":"2024"},"population":"pension_credit_take_up.boundary","sources":["frs"],"structural":"none","weights":null},{"base":"pension_credit_take_up.boundary","citation":"","description":"Ownership boundary before child_benefit_take_up rewrites.","id":"child_benefit_take_up.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage child_benefit_take_up.","id":"child_benefit_take_up","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.child_benefit_take_up@1","mass":"conserve","outputs":[{"column":"would_claim_child_benefit","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"},{"column":"child_benefit_opts_out","dtype":"bool","entity":"benunit","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"child_benefit_take_up","stage_contract_sha256":"4689e00cfcb038c11d78274e1771719bb61beca2d76966193495c15ab691061f","time_period":"2024"},"population":"child_benefit_take_up.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage uc_deduction_attributes.","id":"uc_deduction_attributes","inputs":[{"columns":["child_benefit_opts_out","frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age"],"entity":"person","rows":"all"}],"kernel":"uk.stage.uc_deduction_attributes@1","mass":"conserve","outputs":[{"column":"uc_deduction_random_draw","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"uc_deduction_type_random_draw","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"uc_latent_deduction_rate","dtype":"float64","entity":"benunit","ownership":"produced","rows":"all"},{"column":"uc_deduction_combination","dtype":"string","entity":"benunit","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"uc_deduction_attributes","stage_contract_sha256":"6bc7e3ec1a5712e23ca1b7d2956e8b17f2526b94431ba3e5a9294b04658e4fe0","time_period":"2024"},"population":"child_benefit_take_up.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"child_benefit_take_up.boundary","citation":"","description":"Run structural UK stage cgt_support_split.","id":"cgt_support_split","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_support_split@1","mass":"conserve","outputs":[],"params":{"expand_cells":[["household","household_is_cgt_support_copy","bool"],["household","cgt_support_copies","int64"],["household","cgt_support_copy_index","int64"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_support_split","stage_contract_sha256":"30b37e1c4e122db18391c5f76f1a6d432aea970275bcda7a52126dacc923490b","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by cgt_support_split.","id":"cgt_support_split.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"household_is_cgt_support_copy","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"cgt_support_copies","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"cgt_support_copy_index","dtype":"int64","entity":"household","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_cgt_support_copy","household.cgt_support_copies","household.cgt_support_copy_index"]},"population":"cgt_support_split","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"cgt_support_split","citation":"","description":"Run structural UK stage cgt_incidence_clone.","id":"cgt_incidence_clone","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_incidence_clone@1","mass":"conserve","outputs":[],"params":{"expand_cells":[["household","household_is_capital_gains_clone","bool"],["person","capital_gains","float64"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_incidence_clone","stage_contract_sha256":"ee30278543cc0297a5366d855b7753aa04af5518971594c193c52be36bf8a7b4","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by cgt_incidence_clone.","id":"cgt_incidence_clone.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"household_is_capital_gains_clone","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"capital_gains","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_capital_gains_clone","person.capital_gains"]},"population":"cgt_incidence_clone","sources":[],"structural":"none","weights":null},{"base":"cgt_incidence_clone","citation":"","description":"Ownership boundary before hmrc_cgt_gains_spine rewrites.","id":"hmrc_cgt_gains_spine.boundary","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index","household_is_capital_gains_clone"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending","capital_gains"],"entity":"person","rows":"all"}],"kernel":"uk.identity@1","mass":"conserve","outputs":[],"params":{},"population":null,"sources":[],"structural":"filter","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage hmrc_cgt_gains_spine.","id":"hmrc_cgt_gains_spine","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["cgt_support_copies","corporate_wealth","gross_financial_wealth","household_is_cgt_support_copy","non_residential_property_value","other_residential_property_value","region"],"entity":"household","rows":"all"},{"columns":["age","dividend_income","employment_income","miscellaneous_income","private_pension_income","property_income","savings_interest_income","self_employment_income","state_pension_reported","tax_free_savings_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.hmrc_cgt_gains_spine@1","mass":"conserve","outputs":[{"column":"capital_gains","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"hmrc_cgt_gains_spine","stage_contract_sha256":"ff740a36b03dcda73b889aa43bde7cff5d178de7b3da38e487320c68342bf9a3","time_period":"2024"},"population":"hmrc_cgt_gains_spine.boundary","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"hmrc_cgt_gains_spine.boundary","citation":"","description":"Run structural UK stage cgt_incidence_anchor.","id":"cgt_incidence_anchor","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["household_is_capital_gains_clone","region"],"entity":"household","rows":"all"},{"columns":["age","capital_gains","dividend_income","employment_income","miscellaneous_income","private_pension_income","property_income","savings_interest_income","self_employment_income","state_pension_reported"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_incidence_anchor@1","mass":"conserve","outputs":[],"params":{"expand_cells":[],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_incidence_anchor","stage_contract_sha256":"b79ce806015de2f1e1703fa68bf4ad871c6df54b5b32633c864632ab68919a48","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":"cgt_incidence_anchor","citation":"","description":"Run structural UK stage cgt_residential_split.","id":"cgt_residential_split","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index","household_is_capital_gains_clone"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","employment_income","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","employee_pension_contributions","pension_contributions_via_salary_sacrifice","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending","capital_gains"],"entity":"person","rows":"all"}],"kernel":"uk.stage.expand.cgt_residential_split@1","mass":"conserve","outputs":[],"params":{"expand_cells":[["household","household_is_cgt_residential_clone","bool"],["household","cgt_residential_clone_index","int64"],["person","cgt_residential_probability","float64"],["person","capital_gains_residential_property","float64"]],"expand_weight_entity":"household","expand_weight_kind":"importance","numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"cgt_residential_split","stage_contract_sha256":"3355ff2f5203104a68b0414b0f5e7c05e5c23e9ff89a88e7deb3e5ed6bf04a20","time_period":"2024"},"population":null,"sources":["frs"],"structural":"expand","weights":null},{"base":null,"citation":"","description":"Own the cells materialized by cgt_residential_split.","id":"cgt_residential_split.owned","inputs":[],"kernel":"uk.claim@1","mass":"conserve","outputs":[{"column":"household_is_cgt_residential_clone","dtype":"bool","entity":"household","ownership":"produced","rows":"all"},{"column":"cgt_residential_clone_index","dtype":"int64","entity":"household","ownership":"produced","rows":"all"},{"column":"cgt_residential_probability","dtype":"float64","entity":"person","ownership":"produced","rows":"all"},{"column":"capital_gains_residential_property","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"materialized_expand_outputs":["household.household_is_cgt_residential_clone","household.cgt_residential_clone_index","person.cgt_residential_probability","person.capital_gains_residential_property"]},"population":"cgt_residential_split","sources":[],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage hmrc_cgt_asset_type_spine.","id":"hmrc_cgt_asset_type_spine","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["corporate_wealth","gross_financial_wealth","other_residential_property_value","region","stocks_and_shares_isa"],"entity":"household","rows":"all"},{"columns":["age","capital_gains","capital_gains_residential_property","dividend_income","property_income","self_employment_income"],"entity":"person","rows":"all"}],"kernel":"uk.stage.hmrc_cgt_asset_type_spine@1","mass":"conserve","outputs":[{"column":"capital_gains_asset_type","dtype":"string","entity":"person","ownership":"produced","rows":"all"},{"column":"capital_gains_badr","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"hmrc_cgt_asset_type_spine","stage_contract_sha256":"5fb69157cf25de5cf64b48da73fd3ece5a390bcb17a6fa345fb494e78a95581e","time_period":"2024"},"population":"cgt_residential_split","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage salary_sacrifice.","id":"salary_sacrifice","inputs":[{"columns":["frs_benunit_capital","is_married","dependent_children","would_claim_child_benefit","child_benefit_opts_out","would_claim_pc","would_claim_uc","would_claim_tfc","would_claim_extended_childcare","would_claim_universal_childcare","would_claim_targeted_childcare","would_claim_uc_childcare","maximum_extended_childcare_hours_usage","benunit_source_id","benunit_support_channel","benunit_support_clone_index","uc_reported_capital","uc_deduction_random_draw","uc_deduction_type_random_draw","uc_latent_deduction_rate","uc_deduction_combination"],"entity":"benunit","rows":"all"},{"columns":["region","tenure_type","accommodation_type","num_bedrooms","council_tax_reported","council_tax_band","council_tax_rebate","council_tax_single_adult_raw","water_and_sewerage_charges","domestic_rates","rent","subrent","mortgage_interest_repayment","mortgage_capital_repayment","structural_insurance_payments","housing_service_charges","external_child_payments","ons_household_type","council_tax","household_owns_tv","would_evade_tv_licence_fee","main_residential_property_purchased_is_first_home","property_purchased","brma","source_household_id","source_year","source_household_key","household_source_id","household_support_channel","household_support_clone_index","household_is_spi_synthetic","household_is_spi_income_band_donor","spi_income_band_donor_lower_bound","owned_land","property_wealth","corporate_wealth","private_pension_wealth","gross_financial_wealth","net_financial_wealth","main_residence_value","other_residential_property_value","non_residential_property_value","savings","num_vehicles","cash_isa","stocks_and_shares_isa","mortgage_debt","consumer_debt","household_lifetime_isa_balance","household_local_bus_trips","food_and_non_alcoholic_beverages_consumption","alcohol_and_tobacco_consumption","clothing_and_footwear_consumption","housing_water_and_electricity_consumption","household_furnishings_consumption","health_consumption","transport_consumption","communication_consumption","recreation_consumption","education_consumption","restaurants_and_hotels_consumption","miscellaneous_consumption","petrol_spending","diesel_spending","bus_fare_spending","domestic_energy_consumption","electricity_consumption","gas_consumption","has_fuel_consumption","full_rate_vat_expenditure_rate","dfe_education_spending","rail_subsidy_spending","bus_subsidy_spending","rail_usage","household_is_cgt_support_copy","cgt_support_copies","cgt_support_copy_index","household_is_capital_gains_clone","household_is_cgt_residential_clone","cgt_residential_clone_index"],"entity":"household","rows":"all"},{"columns":["age","gender","marital_status","hours_worked","care_hours","is_household_head","is_benunit_head","is_parent","is_uc_claimant","self_employment_income","private_pension_income","tax_free_savings_income","savings_interest_income","dividend_income","property_income","maintenance_income","miscellaneous_income","private_transfer_income","lump_sum_income","student_loan_repayments","statutory_sick_pay","statutory_maternity_pay","student_loans","access_fund","education_grants","healthy_start_vouchers","free_school_breakfasts","free_school_fruit_veg","free_school_meals","council_tax_benefit_reported","maintenance_expenses","childcare_expenses","personal_pension_contributions","salary_sacrifice_reported","salary_sacrifice_asked","child_benefit_reported","income_support_reported","housing_benefit_reported","attendance_allowance_reported","dla_sc_reported","dla_m_reported","iidb_reported","carers_allowance_reported","would_claim_carers_allowance","sda_reported","afcs_reported","ssmg_reported","pension_credit_reported","child_tax_credit_reported","working_tax_credit_reported","state_pension_reported","winter_fuel_allowance_reported","incapacity_benefit_reported","universal_credit_reported","pip_m_reported","pip_dl_reported","jsa_contrib_reported","jsa_income_reported","esa_contrib_reported","esa_income_reported","bsp_reported","relationship_to_head","ons_family_role","ons_family_index","employment_status","employment_sector","sic_industry_division","aa_category","dla_sc_category","dla_m_category","pip_m_category","pip_dl_category","is_disabled_for_benefits","is_enhanced_disabled_for_benefits","is_severely_disabled_for_benefits","current_education","highest_education","is_in_non_advanced_education","is_in_approved_training","age_started_or_accepted_current_education_or_training","is_before_universal_credit_qualifying_young_person_terminal_date","adult_ema","child_ema","receives_benefits_in_own_right","legacy_jobseeker_proxy","esa_health_condition_proxy","esa_support_group_proxy","disabled_students_allowance_eligible_expenses","would_claim_marriage_allowance","would_claim_scp","attends_private_school_random_draw","tax_free_childcare_spend_routed_share","hmrc_spi_pay","hmrc_spi_unemployment_benefit_income","hmrc_spi_incapacity_benefit_income","ossben_identifiable_subset","srp_regular_code5","employer_pension_contributions","person_source_id","person_support_channel","person_support_clone_index","person_is_spi_income_band_carrier","charitable_investment_gifts","gift_aid","other_investment_income","hmrc_spi_employment_benefits","hmrc_spi_employment_expenses","hmrc_spi_other_social_security_income","hmrc_spi_taxable_termination_pay","hmrc_spi_miscellaneous_employment_income","hmrc_spi_other_income","hmrc_spi_state_pension_income","hmrc_spi_employed_income","hmrc_spi_total_earned_income","hmrc_spi_total_investment_income","hmrc_spi_assessable_income","student_loan_balance","has_lifetime_isa","lifetime_isa_balance","local_bus_use_band","bus_in_london_trips","other_local_bus_trips","local_bus_trips","bus_pass_eligible","local_bus_single_fare_share","a_and_e_visits","admitted_patient_visits","outpatient_visits","nhs_a_and_e_spending","nhs_admitted_patient_spending","nhs_outpatient_spending","capital_gains","cgt_residential_probability","capital_gains_residential_property","capital_gains_asset_type","capital_gains_badr"],"entity":"person","rows":"all"}],"kernel":"uk.stage.salary_sacrifice@1","mass":"conserve","outputs":[{"column":"pension_contributions_via_salary_sacrifice","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employee_pension_contributions","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"employment_income","dtype":"float64","entity":"person","ownership":"produced","rewrite":true,"rows":"all"},{"column":"salary_sacrifice_pre_conversion_pay","dtype":"float64","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"salary_sacrifice","stage_contract_sha256":"051e5ad02ae8f22986db97f513ad3d7dcc06da81f4cfa7ec5d01428f37748360","time_period":"2024"},"population":"cgt_residential_split","sources":["frs"],"structural":"none","weights":null},{"artifact_outputs":[{"name":"stage_evidence","type":{"name":"microcosm.stage-evidence","schema_version":1}}],"base":null,"citation":"","description":"Run UK spine stage student_loans.","id":"student_loans","inputs":[{"columns":["frs_benunit_capital"],"entity":"benunit","rows":"all"},{"columns":["region"],"entity":"household","rows":"all"},{"columns":["age","current_education","highest_education","salary_sacrifice_pre_conversion_pay","student_loan_repayments","student_loans"],"entity":"person","rows":"all"}],"kernel":"uk.stage.student_loans@1","mass":"conserve","outputs":[{"column":"student_loan_plan","dtype":"string","entity":"person","ownership":"produced","rows":"all"}],"params":{"numerical_dependencies":[["policyengine-uk","2.100.0"],["policyengine-core","3.32.5"],["numpy","2.4.6"],["pandas","3.0.3"],["scikit-learn","1.8.0"],["quantile-forest","1.4.2"]],"stage":"student_loans","stage_contract_sha256":"f2936c5e22a0e9b097437e3f3aabaf06c4561d5c25a9b7c05e720a6553f73bbc","time_period":"2024"},"population":"cgt_residential_split","sources":["frs"],"structural":"none","weights":null}],"sources":[{"codec":"csv-tables","description":"Content-bound UK FRS and donor fixture/source bundle.","name":"frs"}]} diff --git a/test_support/microcosm_build/uk_full_population_graph.py b/test_support/microcosm_build/uk_full_population_graph.py index 87079e5b0..91d985d95 100644 --- a/test_support/microcosm_build/uk_full_population_graph.py +++ b/test_support/microcosm_build/uk_full_population_graph.py @@ -59,6 +59,8 @@ def source_frame(): household["household_is_capital_gains_clone"] = False household["household_is_cgt_support_copy"] = False household["cgt_support_copy_index"] = np.zeros(len(household), dtype="int64") + household["household_is_cgt_residential_clone"] = False + household["cgt_residential_clone_index"] = np.zeros(len(household), dtype="int64") return Frame( tables, original.schema, diff --git a/test_support/microcosm_build/uk_ladder_rowwise_clone.py b/test_support/microcosm_build/uk_ladder_rowwise_clone.py index a9bf4b3fd..de863de2c 100644 --- a/test_support/microcosm_build/uk_ladder_rowwise_clone.py +++ b/test_support/microcosm_build/uk_ladder_rowwise_clone.py @@ -113,6 +113,8 @@ def _spine_household_frame() -> pd.DataFrame: household_is_capital_gains_clone=[False, False, True, False], household_is_cgt_support_copy=[False, False, False, True], cgt_support_copy_index=[0, 0, 0, 1], + household_is_cgt_residential_clone=[False, False, False, False], + cgt_residential_clone_index=[0, 0, 0, 0], ) diff --git a/test_support/microcosm_build/uk_rowwise_candidate.py b/test_support/microcosm_build/uk_rowwise_candidate.py index f09ad6763..bd79c09cf 100644 --- a/test_support/microcosm_build/uk_rowwise_candidate.py +++ b/test_support/microcosm_build/uk_rowwise_candidate.py @@ -282,6 +282,8 @@ def _write_staging_h5( "household_is_capital_gains_clone": [False] * len(household_ids), "household_is_cgt_support_copy": [False] * len(household_ids), "cgt_support_copy_index": [0] * len(household_ids), + "household_is_cgt_residential_clone": [False] * len(household_ids), + "cgt_residential_clone_index": [0] * len(household_ids), } ) person_ids = [10_000 + household_id for household_id in household_ids] diff --git a/tools/build_uk_release_input_coverage_manifest.py b/tools/build_uk_release_input_coverage_manifest.py index e44d8dec7..f719aaa8b 100644 --- a/tools/build_uk_release_input_coverage_manifest.py +++ b/tools/build_uk_release_input_coverage_manifest.py @@ -810,6 +810,12 @@ def build_manifest( stage_name="cgt_incidence_anchor", candidate_source=candidate_source, ), + # The residential split conserves mass like the other CGT + # structural stages (microcosm#1063). + "cgt_residential_split": _source_stage_family_coverage_contract( + stage_name="cgt_residential_split", + candidate_source=candidate_source, + ), "salary_sacrifice": _source_stage_family_coverage_contract( stage_name="salary_sacrifice", candidate_source=candidate_source, @@ -818,6 +824,13 @@ def build_manifest( stage_name="student_loans", candidate_source=candidate_source, ), + # The reserved income rows are a support channel funded from the + # incumbent households (microcosm#1063): the terminal gate demands + # their conserving receipt like any other stage's. + "spi_income_band_donors": _source_stage_family_coverage_contract( + stage_name="spi_income_band_donors", + candidate_source=candidate_source, + ), "hmrc_spi_income": _hmrc_family_coverage_contract( candidate_source=candidate_source ), @@ -915,10 +928,15 @@ def _source_stage_family_coverage_contract( "assert the mass was conserved." ) required_mass_change_reason = declared_reasons[-1] - # Every spine stage conserves household mass since microcosm#1045 retired - # the band-donor stack; the validator keeps ``mass_increasing_support`` - # only until the SPI income-band donors follow (microcosm#1063). - mass_change_semantics = "mass_conserving" + from microcosm.build.uk_runtime.release_input_coverage import ( + UK_FAMILY_MASS_CHANGE_SEMANTICS, + ) + + # Every spine stage conserves household mass: microcosm#1045 retired the + # CGT band-donor stack and microcosm#1063 made the SPI income-band donors + # a funded support channel, so no family adds mass and the validator + # accepts no other semantics. + mass_change_semantics = UK_FAMILY_MASS_CHANGE_SEMANTICS return { "status": "required_at_build", "stage": stage_name, diff --git a/tools/graph_uk_spine_fixture.py b/tools/graph_uk_spine_fixture.py index 87bf7966b..fde303760 100644 --- a/tools/graph_uk_spine_fixture.py +++ b/tools/graph_uk_spine_fixture.py @@ -28,16 +28,15 @@ from microcosm.build.source_manifest import SourceManifest, SourceStageSpec from microcosm.build.uk_runtime.age_tail import UKAgeTailStageTransform from microcosm.build.uk_runtime.cgt_asset_type import ( - CGT_ASSET_TYPE_COLUMN, CGT_ASSET_TYPE_NON_RESIDENTIAL_TYPES, CGT_ASSET_TYPE_RESIDENTIAL, + CGT_RESIDENTIAL_GAINS_COLUMN, HMRC_CGT_TABLE4_BAND_LOWER_BOUNDS, HMRCCGTAssetTypeFacts, HMRCCGTBADRBand, HMRCCGTTable7Type, UKCGTAssetTypeStageTransform, UKCGTBADRParameters, - assign_uk_cgt_asset_types, uk_cgt_badr_parameters, ) from microcosm.build.uk_runtime.cgt_imputation import ( @@ -45,11 +44,18 @@ uk_cgt_policy_parameters, uk_cgt_spine_stage_transform, ) +from microcosm.build.uk_runtime.cgt_residential_split import ( + UKCGTResidentialSplitStageTransform, + split_cgt_residential_households, +) from microcosm.build.uk_runtime.cgt_structure import ( UKCGTIncidenceAnchorStageTransform, UKCGTIncidenceCloneStageTransform, ) from microcosm.build.uk_runtime.cgt_support import UKCGTSupportSplitStageTransform +from microcosm.build.uk_runtime.child_benefit_take_up import ( + UKChildBenefitTakeUpStageTransform, +) from microcosm.build.uk_runtime.content_identity import uk_frame_content_identity from microcosm.build.uk_runtime.etb_services import UKETBServicesStageTransform from microcosm.build.uk_runtime.etb_vat import UKETBVATStageTransform @@ -154,11 +160,11 @@ _SPI_DONOR_SAMPLE_SIZE = 64 #: The packaged FRS spine roster the fixture exercises (manifest minus the #: certified-pair exclusions); moves whenever a spine stage is added. -UK_FIXTURE_STAGE_COUNT = 36 +UK_FIXTURE_STAGE_COUNT = 38 _QRF_ESTIMATORS = 4 # These are the complete object-string surface observed in the unchanged -# legacy 36-stage output. Graph storage uses pandas StringDtype/python. +# legacy 38-stage output. Graph storage uses pandas StringDtype/python. _NORMALIZED_STRING_COLUMNS: Mapping[str, tuple[str, ...]] = { "person": ( "gender", @@ -662,6 +668,7 @@ def _was_person_donor() -> pd.DataFrame: def _was_donor() -> pd.DataFrame: rows = np.arange(_DONOR_ROWS, dtype=float) + position = rows.astype(int) lisa_totals = ( _was_person_donor() .groupby("CASER8")["DVFLISAvR8"] @@ -669,47 +676,89 @@ def _was_donor() -> pd.DataFrame: .reindex(rows + 1.0, fill_value=0.0) .to_numpy() ) + # Tenure in the round-8 codebook: 1 owns outright, 2 buys with a mortgage, + # 3 part rent / part mortgage, 4 rents, 5 lives rent-free. DVPriRntR8 is 1 + # for a private renting household, 2 for a council or housing-association + # tenant and -9 (not applicable) for owners. + tenure = np.take([1, 2, 3, 4, 4, 5], position % 6) + owner = np.isin(tenure, [1, 2, 3]) + mortgaged = np.isin(tenure, [2, 3]) + private_rent = np.where(owner, -9, np.where(position % 6 == 4, 2, 1)) + # The accounting identities of the real tab hold on every row: the + # property total is the sum of the property values, gross financial + # wealth covers the listed assets, and net is gross less the liabilities. + main_residence = np.where(owner, 100_000.0 + rows * 2_000.0, 0.0) + other_houses = np.where(position % 3 == 0, 1_000.0 + rows * 50.0, 0.0) + buildings = np.where(position % 4 == 0, 3_000.0 + rows * 60.0, 0.0) + land = np.where(position % 5 == 0, 10.0 + rows, 0.0) + other_property = np.where(position % 7 == 0, 500.0 + rows * 5.0, 0.0) + employee_shares = 1.0 + rows + uk_shares = 3.0 + rows + stocks_isa = 5.0 + rows + cash_isa = 7.0 + rows + collectives = 9.0 + rows + savings = 500.0 + rows * 20.0 + gross_financial = ( + savings + + cash_isa + + stocks_isa + + employee_shares + + uk_shares + + collectives + + 30.0 + + rows + ) + consumer_debt = (position % 5) * 40.0 + student_loans = np.where(position % 3 == 1, 5_000.0 + rows * 5.0, 0.0) + main_mortgage = np.where(mortgaged, 10_000.0 + rows * 100.0, 0.0) + # A mortgage on other property, held off a mortgaged tenure by a few + # owners outright and renters, as on the real tab. + other_mortgage = np.where( + ~mortgaged & (other_houses > 0.0) & (position % 2 == 0), 2_000.0 + rows, 0.0 + ) return pd.DataFrame( { "CASER8": rows + 1.0, "DVFLISAVR8_aggr": lisa_totals, "R8xshhwgt": 1.0 + rows % 7 / 10.0, - "DVLUKValR8_sum": 10.0 + rows, - "DVPropertyR8": 20_000.0 + rows * 500.0, - "DVFESHARESR8_aggr": 1.0 + rows, - "DVFShUKVR8_aggr": 3.0 + rows, - "DVIISAVR8_aggR": 5.0 + rows, - "DVCISAVR8_aggr": 7.0 + rows, - "DVFCollVR8_aggr": 9.0 + rows, + "DVLUKValR8_sum": land, + "DVPropertyR8": main_residence + + other_houses + + buildings + + land + + other_property, + "DVFESHARESR8_aggr": employee_shares, + "DVFShUKVR8_aggr": uk_shares, + "DVIISAVR8_aggR": stocks_isa, + "DVCISAVR8_aggr": cash_isa, + "DVFCollVR8_aggr": collectives, "totalpenr8_aggr": 100.0 + rows * 10.0, "dvvaldbt_scaper8_aggr": 40.0 + rows, - "NumAdultR8": 1 + rows.astype(int) % 3, - "NumCh18R8": rows.astype(int) % 3, + "NumAdultR8": 1 + position % 3, + "NumCh18R8": position % 3, "DVGIPPENR8_AGGR": 11.0 + rows, "DVGISER8_AGGR": 13.0 + rows, "DVGIINVR8_aggr": 15.0 + rows, "DVGIEMPR8_AGGR": 17.0 + rows, - "HBedRmR8": 1 + rows.astype(int) % 5, - "GORR8": np.take([8, 11, 12, 1], rows.astype(int) % 4), - "DVPriRntR8": 1 + rows.astype(int) % 2, + "HBedRmR8": 1 + position % 5, + "GORR8": np.take([8, 11, 12, 1], position % 4), + "DVPriRntR8": private_rent, "CTAmtR8": 900.0 + rows * 10.0, - "HFINWNTR8_Sum": -50.0 + rows * 4.0, - "HFINWNTR8_exSLC_Sum": 20.0 + rows - rows % 5, - "HMortGR8": np.where( - np.isin(np.take([1, 2, 3, 4], rows.astype(int) % 4), [2, 3]), - 10_000.0 + rows * 100.0, - 0.0, - ), - "Ten1R8": np.take([1, 2, 3, 4], rows.astype(int) % 4), - "HFINWR8_SUM": 30.0 + rows, - "DVhvalueR8": 100_000.0 + rows * 2_000.0, - "DVHseValR8_sum": 1_000.0 + rows * 50.0, - "DVBlDValR8_sum": 3_000.0 + rows * 60.0, + "HFINWNTR8_Sum": gross_financial - consumer_debt - student_loans, + "HFINWNTR8_exSLC_Sum": gross_financial - consumer_debt, + "HMortGR8": main_mortgage + other_mortgage, + "TotMortR8": main_mortgage, + "OthMortR8_sum": other_mortgage, + "Ten1R8": tenure, + "HFINWR8_SUM": gross_financial, + "DVhvalueR8": main_residence, + "DVHseValR8_sum": other_houses, + "DVBlDValR8_sum": buildings, "DVTotinc_bhcR8": 20_000.0 + rows * 1_000.0, - "DVSaValR8_aggr": 500.0 + rows * 20.0, - "vcarnr8": rows.astype(int) % 4, - "Tot_LosR8_aggr": 9_000.0 + rows * 10.0, - "Tot_los_exc_SLCR8_aggr": 4_000.0 + rows * 5.0, + "DVSaValR8_aggr": savings, + "vcarnr8": position % 4, + "Tot_LosR8_aggr": 9_000.0 + rows * 10.0 + student_loans, + "Tot_los_exc_SLCR8_aggr": 9_000.0 + rows * 10.0, } ) @@ -1135,18 +1184,17 @@ def _cgt_asset_type_facts( parameters: UKCGTPolicyParameters, badr_parameters: UKCGTBADRParameters, ) -> HMRCCGTAssetTypeFacts: - """Synthetic Table 4.1/7/8 facts sized to the fixture frame after the redraw. + """Synthetic Table 4.1/7/8 facts sized to the fixture frame after the anchor. The residential targets are a fixed share of the liable mass and of the liable gains, so their mean is the population mean and the logistic solve is always attainable on the tiny fixture; the data-only payload the graph side reads is then exactly these numbers. The Table 4.1 bands - are sized in a second pass: the stage runs once with every band at zero - (so no claim is drawn) to learn the residential flags, which are drawn - first on their own seed and are therefore the same in the real run; each - band then claims a fixed share of its non-residential pool at the pool's - own mean gain, and the open top band the same share at the lifetime - limit. A band with no pool carries zero and is skipped. + are sized in a second pass: the residential split runs once on these + targets (it is deterministic, so its arms are the ones the real run + makes), and each band then claims a fixed share of the non-residential + arms' pool at the pool's own mean gain, the open top band the same share + at the lifetime limit. A band with no pool carries zero and is skipped. """ person = frame.table("person") @@ -1213,14 +1261,22 @@ def facts_with(bands: tuple[HMRCCGTBADRBand, ...]) -> HMRCCGTAssetTypeFacts: ) for lower, upper in zip(lowers, uppers, strict=True) ) - classified, _ = assign_uk_cgt_asset_types( - frame, facts_with(empty), parameters, badr_parameters + split = split_cgt_residential_households( + frame, facts=facts_with(empty), parameters=parameters + ).frame + split_person = split.table("person") + split_household = split.table("household") + split_weights = pd.Series( + split.weights_for("household").values, index=split_household["household_id"] ) - residential = ( - classified.table("person")[CGT_ASSET_TYPE_COLUMN].to_numpy() - == CGT_ASSET_TYPE_RESIDENTIAL + person_weight = ( + split_person["person_household_id"].map(split_weights).to_numpy(dtype=float) ) - pool = liable & ~residential + gains = pd.to_numeric(split_person["capital_gains"], errors="raise").to_numpy( + dtype=float + ) + residential = split_person[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy(dtype=float) > 0.0 + pool = (gains > parameters.annual_exempt_amount) & ~residential limit = float(badr_parameters.lifetime_limit) bands = [] for lower, upper in zip(lowers, uppers, strict=True): @@ -1594,6 +1650,9 @@ def capture_root(frame: Frame) -> Frame: "pension_credit_take_up": UKPensionCreditTakeUpStageTransform( stage=stages["pension_credit_take_up"], engine=engine ), + "child_benefit_take_up": UKChildBenefitTakeUpStageTransform( + stage=stages["child_benefit_take_up"], engine=engine + ), "uc_deduction_attributes": UKUCDeductionAttributesStageTransform( stage=stages["uc_deduction_attributes"] ), @@ -1610,15 +1669,20 @@ def capture_root(frame: Frame) -> Frame: distribution=cgt_distribution, parameters=cgt_parameters, ), + "cgt_incidence_anchor": UKCGTIncidenceAnchorStageTransform( + stage=stages["cgt_incidence_anchor"], parameters=cgt_parameters + ), + "cgt_residential_split": UKCGTResidentialSplitStageTransform( + stage=stages["cgt_residential_split"], + facts=cgt_asset_type_facts, + parameters=cgt_parameters, + ), "hmrc_cgt_asset_type_spine": UKCGTAssetTypeStageTransform( stage=stages["hmrc_cgt_asset_type_spine"], facts=cgt_asset_type_facts, parameters=cgt_parameters, badr_parameters=cgt_badr_parameters, ), - "cgt_incidence_anchor": UKCGTIncidenceAnchorStageTransform( - stage=stages["cgt_incidence_anchor"], parameters=cgt_parameters - ), "salary_sacrifice": UKSalarySacrificeStageTransform( stage=stages["salary_sacrifice"] ), @@ -1634,7 +1698,7 @@ def _run_legacy_plan( stages: Iterable[SourceStageSpec], implementations: Mapping[str, object], ) -> Frame: - """Run the legacy 36-stage StagePlan oracle and return its final frame.""" + """Run the legacy 38-stage StagePlan oracle and return its final frame.""" stages = tuple(stages) committed = load_country_spec("uk") @@ -1752,12 +1816,12 @@ def generate(output: Path) -> None: cgt_parameters=cgt_parameters, cgt_badr_parameters=cgt_badr_parameters, ) - # The asset-type facts are sized to the frame the amounts redraw leaves, - # so run the oracle up to that stage once, derive them, and only then - # run the full plan on fresh transforms. + # The asset-type facts are sized to the frame the amounts redraw and the + # anchor leave, so run the oracle up to the residential split once, derive + # them, and only then run the full plan on fresh transforms. implementations, _ = _build_implementations(**build_kwargs) stage_names = [stage.stage for stage in stages] - prefix = stages[: stage_names.index("hmrc_cgt_asset_type_spine")] + prefix = stages[: stage_names.index("cgt_residential_split")] prefix_names = {stage.stage for stage in prefix} after_redraw = _run_legacy_plan( prefix, diff --git a/tools/uk_residential_readout.py b/tools/uk_residential_readout.py new file mode 100644 index 000000000..c36ea0715 --- /dev/null +++ b/tools/uk_residential_readout.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Read the residential split's by-band identities back on calibrated weights. + +microcosm#1063 review item 4: ``cgt_residential_split`` makes HMRC Table 8a's +residential count and gains identities in every gain band at design weights; +calibration binds only the two national totals, so this reports, per band, +the residential count and gains at design and at calibrated weights next to +the stage's expected values, and the row-level weight shape with the arms in +place (smallest positive weight, positive-weight median, zero-weight rows). + + uk_residential_readout.py [--out readout.json] + +``run dir`` is a national build's output directory holding +``uk.full.dense.solution.artifact`` (household weights keyed by household id); +``spine.h5`` is the spine the build read, with its ``.build.json`` beside it +(the split stage's receipt supplies the bands). Licensed inputs stay where +they are; nothing is written unless ``--out`` is given. +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import numpy as np +import pandas as pd + + +def calibrated_weights(run: Path) -> pd.Series: + solution = np.load(run / "uk.full.dense.solution.artifact", allow_pickle=False) + metadata = json.loads(bytes(solution["metadata"]).decode()) + return pd.Series( + np.asarray(solution["weights"], dtype=float), + index=pd.Index(metadata["entity_ids"], name="household_id"), + ) + + +def main() -> int: + p = argparse.ArgumentParser() + p.add_argument("run", type=Path) + p.add_argument("spine", type=Path) + p.add_argument("--out", type=Path) + a = p.parse_args() + with pd.HDFStore(a.spine, mode="r") as store: + household = store.select( + "/household", + columns=[ + "household_id", + "household_weight", + "household_is_cgt_residential_clone", + "cgt_residential_clone_index", + ], + ) + person = store.select( + "/person", + columns=[ + "person_id", + "person_household_id", + "capital_gains", + "capital_gains_residential_property", + "cgt_residential_probability", + ], + ) + build = json.loads((a.spine.parent / (a.spine.stem + ".build.json")).read_text()) + + def find(o): + if isinstance(o, dict): + if o.get("stage") == "cgt_residential_split" and "bands" in o: + return o + for v in o.values(): + r = find(v) + if r: + return r + elif isinstance(o, list): + for v in o: + r = find(v) + if r: + return r + + receipt = find(build) + bands = receipt["bands"] + design = household.set_index("household_id")["household_weight"].astype(float) + final = calibrated_weights(a.run).reindex(design.index) + missing = int(final.isna().sum()) + person = person.assign( + design=person["person_household_id"].map(design).to_numpy(float), + final=person["person_household_id"].map(final).to_numpy(float), + ) + res = person["capital_gains_residential_property"].to_numpy(float) > 0 + gains = person["capital_gains"].to_numpy(float) + rows = [] + for band in bands: + lo, hi = float(band["gain_lower_bound"]), band.get("gain_upper_bound") + inb = res & (gains >= lo) & ((gains < float(hi)) if hi is not None else True) + rows.append( + { + "gain_lower_bound": lo, + "gain_upper_bound": hi, + "expected_count": band["expected_count"], + "expected_gains": band["expected_gains"], + "design_count": float(person["design"].to_numpy()[inb].sum()), + "design_gains": float((person["design"].to_numpy() * gains)[inb].sum()), + "calibrated_count": float(person["final"].to_numpy()[inb].sum()), + "calibrated_gains": float( + (person["final"].to_numpy() * gains)[inb].sum() + ), + "arms": int(inb.sum()), + } + ) + tot = { + k: float(sum(r[k] for r in rows)) + for k in ( + "expected_count", + "expected_gains", + "design_count", + "design_gains", + "calibrated_count", + "calibrated_gains", + ) + } + arms = household["household_is_cgt_residential_clone"].astype(bool).to_numpy() + pos = final.to_numpy(float) > 0 + out = { + "run": str(a.run), + "spine": str(a.spine), + "households_without_calibrated_weight": missing, + "bands": rows, + "totals": tot, + "calibrated_over_expected": { + "count": tot["calibrated_count"] / tot["expected_count"], + "gains": tot["calibrated_gains"] / tot["expected_gains"], + }, + "row_level_weights": { + "minimum_positive_calibrated": float(final.to_numpy(float)[pos].min()), + "median_positive_calibrated": float(np.median(final.to_numpy(float)[pos])), + "maximum_calibrated": float(final.max()), + "zero_weight_rows": int((~pos).sum()), + "arms": { + "rows": int(arms.sum()), + "minimum_design": float(design.to_numpy()[arms].min()), + "minimum_positive_calibrated": float( + final.to_numpy(float)[arms & pos].min() + ) + if (arms & pos).any() + else None, + "median_positive_calibrated": float( + np.median(final.to_numpy(float)[arms & pos]) + ) + if (arms & pos).any() + else None, + "zero_weight": int((arms & ~pos).sum()), + }, + }, + } + text = json.dumps(out, indent=1) + if a.out: + a.out.write_text(text) + print(text) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/verify_uk_identity_stability.py b/tools/verify_uk_identity_stability.py index a665ac3e3..ff293b14b 100644 --- a/tools/verify_uk_identity_stability.py +++ b/tools/verify_uk_identity_stability.py @@ -42,6 +42,10 @@ load_regional_land_values_resource, uprate_household_property_by_region, ) +from microcosm.build.uk_runtime.spi_band_donors import ( + HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR, + SPI_INCOME_BAND_DONOR_CLONE_INDEX, +) from microcosm.build.uk_runtime.was_wealth import ( allocate_student_loan_balance_to_people, ) @@ -559,7 +563,17 @@ def recompute(person_t, benunit_t, household_t) -> dict[str, pd.DataFrame]: synthetic = household_t["household_is_spi_synthetic"].astype(bool).to_numpy() channel = np.where(synthetic, "spi", "frs") household_out["household_support_channel"] = channel - household_out["household_support_clone_index"] = np.where(synthetic, 1, 0) + # The SPI support copy sits at clone index 1 and the income band donors, + # which carry the same synthetic flag, at their own index; a frame built + # before the donor stage carries no donor flag and no donors. + donor = ( + household_t[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR].astype(bool).to_numpy() + if HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR in household_t.columns + else np.zeros(len(household_t), dtype=bool) + ) + household_out["household_support_clone_index"] = np.where( + synthetic, np.where(donor, SPI_INCOME_BAND_DONOR_CLONE_INDEX, 1), 0 + ) missing_keys = {"source_year", "source_household_id"} - set(household_t.columns) if missing_keys: @@ -1009,10 +1023,511 @@ def recompute(person_t, household_t, weights_t): return receipt +def _e8_residential_split( + frame, + problems: dict[str, object], + *, + parameters, + facts, + permutation_seed: int, +) -> dict[str, object]: + """Recompute the residential split from the folded pre-split frame. + + The split (microcosm#1063) is deterministic - no draw, no seed - so it is + reconstructible from the artifact alone: every arm folded onto the arm + that kept its household's ids gives the household table at the weights + the stage saw, and the stage's own function rerun on that table from the + vendored Table 8 rows and the policy parameters must reproduce the stored + arms (ids, indices and weights), the stored probabilities and the stored + residential gains, in original and permuted person order. The stored + layer must also agree with itself (every arm has a source, flag and index + agree, the mass record conserves the total at declared factor one). + Problems land under ``residential_split_*`` keys. + """ + + from microcosm.build.uk_runtime.cgt_asset_type import ( + CGT_RESIDENTIAL_GAINS_COLUMN, + ) + from microcosm.build.uk_runtime.cgt_residential_split import ( + CGT_RESIDENTIAL_CLONE_INDEX_COLUMN, + CGT_RESIDENTIAL_PROBABILITY_COLUMN, + CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON, + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + split_cgt_residential_households, + ) + from microcosm.build.uk_runtime.national_frame import ( + uk_household_weight_kind, + uk_national_frame, + ) + + household = frame.table("household") + person = frame.table("person") + if HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE not in household.columns: + raise ValueError( + "e8 identity receipt: the artifact carries no " + f"{HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE!r} column, so the CGT " + "residential split layer is absent and there is nothing to receipt. " + "Receipt the artifact with the tool at the commit that built it." + ) + receipt: dict[str, object] = {} + mass_records = [ + record + for record in frame.mass_log + if record.reason == CGT_RESIDENTIAL_SPLIT_MASS_CHANGE_REASON + ] + if not mass_records: + problems["residential_split_mass_record"] = "missing" + else: + record = mass_records[-1] + receipt["mass"] = { + "old_total": float(record.old_total), + "new_total": float(record.new_total), + "declared_factor": record.declared_factor, + } + if record.declared_factor != 1.0 or not np.isclose( + record.new_total, record.old_total, rtol=1e-9, atol=0.0 + ): + problems["residential_split_mass_record"] = [ + float(record.old_total), + float(record.new_total), + record.declared_factor, + ] + try: + lineage = _residential_arm_lineage(person, frame.table("benunit"), household) + except ValueError as error: + problems["residential_split_lineage"] = str(error) + return receipt + receipt["id_multiplier"] = int(lineage.multiplier) + receipt["arms"] = int(len(lineage.arm_positions)) + receipt["households_split"] = int(np.unique(lineage.source_positions).size) + receipt["largest_arm_index"] = ( + int(lineage.arm_index.max()) if lineage.arm_index.size else 0 + ) + + # The frame the split saw: arms folded onto their sources and dropped, + # the stage's own columns removed. + pre = _drop_stacked_layers(frame, [HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE]) + pre_person = pre.table("person").drop( + columns=[CGT_RESIDENTIAL_PROBABILITY_COLUMN, CGT_RESIDENTIAL_GAINS_COLUMN] + ) + pre_household = pre.table("household").drop( + columns=[HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, CGT_RESIDENTIAL_CLONE_INDEX_COLUMN] + ) + + def pre_frame(person_table: pd.DataFrame): + return uk_national_frame( + person=person_table, + benunit=pre.table("benunit"), + household=pre_household, + time_period=uk_time_period(frame), + weight_kind=uk_household_weight_kind(frame), + household_weights=pre.weights_for("household").values, + mass_log=pre.mass_log, + ) + + def recompute(person_table: pd.DataFrame): + result = split_cgt_residential_households( + pre_frame(person_table), facts=facts, parameters=parameters + ) + table = result.frame.table("household") + weights = pd.Series( + np.asarray(result.frame.weights_for("household").values, dtype=float), + index=pd.to_numeric(table["household_id"]).astype("int64").to_numpy(), + ) + index = pd.Series( + table[CGT_RESIDENTIAL_CLONE_INDEX_COLUMN].to_numpy(dtype="int64"), + index=weights.index, + ) + people = result.frame.table("person") + ids = pd.to_numeric(people["person_id"]).astype("int64").to_numpy() + probability = pd.Series( + people[CGT_RESIDENTIAL_PROBABILITY_COLUMN].to_numpy(dtype=float), index=ids + ) + gains = pd.Series( + people[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy(dtype=float), index=ids + ) + return weights, index, probability, gains, result.evidence() + + weights, index, probability, gains, evidence = recompute(pre_person) + receipt["identities"] = dict(evidence["identities"]) + receipt["arm_weights"] = dict(evidence["arm_weights"]) + receipt["concentration"] = dict(evidence["concentration"]) + stored_weights = pd.Series( + np.asarray(frame.weights_for("household").values, dtype=float), + index=lineage.household_ids, + ) + stored_index = pd.Series( + household[CGT_RESIDENTIAL_CLONE_INDEX_COLUMN].to_numpy(dtype="int64"), + index=lineage.household_ids, + ) + missing = int((~stored_weights.index.isin(weights.index)).sum()) + extra = int((~weights.index.isin(stored_weights.index)).sum()) + if missing or extra: + problems["residential_split_arms_stored"] = { + "missing": missing, + "extra": extra, + } + else: + aligned = weights.reindex(stored_weights.index) + receipt["max_abs_weight_diff"] = float( + np.abs(aligned.to_numpy() - stored_weights.to_numpy()).max() + ) + off = ~np.isclose( + aligned.to_numpy(), stored_weights.to_numpy(), rtol=1e-9, atol=1e-6 + ) + if off.any(): + problems["residential_split_weights_stored"] = int(off.sum()) + if not index.reindex(stored_index.index).equals(stored_index): + problems["residential_split_index_stored"] = True + person_ids = pd.to_numeric(person["person_id"]).astype("int64").to_numpy() + stored_probability = pd.Series( + person[CGT_RESIDENTIAL_PROBABILITY_COLUMN].to_numpy(dtype=float), + index=person_ids, + ) + stored_gains = pd.Series( + person[CGT_RESIDENTIAL_GAINS_COLUMN].to_numpy(dtype=float), index=person_ids + ) + if not stored_probability.index.isin(probability.index).all(): + problems["residential_split_people_stored"] = True + else: + if not np.allclose( + probability.reindex(stored_probability.index).to_numpy(), + stored_probability.to_numpy(), + rtol=1e-9, + atol=0.0, + ): + problems["residential_split_probability_stored"] = True + if not np.array_equal( + gains.reindex(stored_gains.index).to_numpy(), stored_gains.to_numpy() + ): + problems["residential_split_gains_stored"] = True + + rng = np.random.default_rng(permutation_seed) + permuted_weights, permuted_index, permuted_probability, permuted_gains, _ = ( + recompute( + pre_person.iloc[rng.permutation(len(pre_person))].reset_index(drop=True) + ) + ) + if not ( + permuted_weights.sort_index().equals(weights.sort_index()) + and permuted_index.sort_index().equals(index.sort_index()) + and permuted_probability.sort_index().equals(probability.sort_index()) + and permuted_gains.sort_index().equals(gains.sort_index()) + ): + problems["residential_split_permutation"] = True + return receipt + + +def _e8_band_donors( + frame, + problems: dict[str, object], + *, + propensity: pd.DataFrame | None, + band_taxpayers: Mapping[int, float] | None, + permutation_seed: int, +) -> dict[str, object]: + """Reconstruct the SPI income band donor seating from ids (microcosm#1063). + + The donors are a funded support channel: each band's donors carry equal + weights and their mass left the incumbent households of each donor's + region in proportion. Folding the CGT layers stacked afterwards gives the + frame the stage wrote; scaling each region's incumbents back to the + region's whole mass gives the weights it read. The stored layer must + agree with itself (one carrier per donor, each source household copied + once, equal weights within a band, the mass record conserving the total) + and, when the tape's band propensities are supplied (they need the + licensed SPI tape), with the stage's own function rerun on the + reconstructed pre-donor frame: the same source households in the same + bands, the same carriers and the same band weights, in original and + permuted row order. The seating is keyed on person ids, so the rerun + needs no stage-time row order. Without the propensities the receipt says + the seating was not recomputed. Problems land under ``band_donor_*`` keys. + """ + + from microcosm.build.uk_runtime.cgt_structure import HOUSEHOLD_IS_CGT_CLONE + from microcosm.build.uk_runtime.cgt_support import HOUSEHOLD_IS_CGT_SUPPORT_COPY + from microcosm.build.uk_runtime.national_frame import ( + uk_household_weight_kind, + uk_national_frame, + ) + from microcosm.build.uk_runtime.spi_band_donors import ( + HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR, + PERSON_IS_SPI_INCOME_BAND_CARRIER, + SPI_INCOME_BAND_DONOR_DRAW_SALT, + SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN, + SPI_INCOME_BAND_DONOR_LOWER_BOUNDS, + SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON, + SPI_INCOME_BAND_DONOR_SEED, + _funding_strata, + stack_spi_income_band_donors, + ) + + if HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR not in frame.table("household").columns: + raise ValueError( + "e8 identity receipt: the artifact carries no " + f"{HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR!r} column, so the SPI income " + "band donor layer is absent and there is nothing to receipt. " + "Receipt the artifact with the tool at the commit that built it." + ) + receipt: dict[str, object] = { + "seed": SPI_INCOME_BAND_DONOR_SEED, + "salt": SPI_INCOME_BAND_DONOR_DRAW_SALT, + } + mass_records = [ + record + for record in frame.mass_log + if record.reason == SPI_INCOME_BAND_DONOR_MASS_CHANGE_REASON + ] + if not mass_records: + problems["band_donor_mass_record"] = "missing" + else: + record = mass_records[-1] + receipt["mass"] = { + "old_total": float(record.old_total), + "new_total": float(record.new_total), + "declared_factor": record.declared_factor, + } + if record.declared_factor != 1.0 or not np.isclose( + record.new_total, record.old_total, rtol=1e-9, atol=0.0 + ): + problems["band_donor_mass_record"] = [ + float(record.old_total), + float(record.new_total), + record.declared_factor, + ] + + # The frame the donor stage wrote: the CGT layers stacked after it folded + # back, so every remaining household carries the weight the stage left. + staged = _drop_stacked_layers( + frame, + [ + flag + for flag in (HOUSEHOLD_IS_CGT_CLONE, HOUSEHOLD_IS_CGT_SUPPORT_COPY) + if flag in frame.table("household").columns + ], + ) + household = staged.table("household") + person = staged.table("person") + benunit = staged.table("benunit") + weights = np.asarray(staged.weights_for("household").values, dtype=float) + is_donor = household[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR].astype(bool).to_numpy() + donor_band = household[SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN].to_numpy(float) + donor_source = ( + pd.to_numeric(household.loc[is_donor, "household_source_id"], errors="raise") + .astype("int64") + .to_numpy() + ) + receipt["donor_households"] = int(is_donor.sum()) + stored_seats = pd.Series(donor_band[is_donor], index=donor_source).sort_index() + + # Stored-layer consistency. + layer: dict[str, int] = {} + if not stored_seats.index.is_unique: + layer["source_households_copied_twice"] = int( + stored_seats.index.duplicated().sum() + ) + undeclared = ~np.isin(donor_band[is_donor], SPI_INCOME_BAND_DONOR_LOWER_BOUNDS) + if undeclared.any(): + layer["donors_in_undeclared_bands"] = int(undeclared.sum()) + if (donor_band[~is_donor] != 0.0).any(): + layer["incumbents_carrying_a_band"] = int((donor_band[~is_donor] != 0.0).sum()) + carriers = person[PERSON_IS_SPI_INCOME_BAND_CARRIER].astype(bool) + carriers_per_household = ( + person.loc[carriers, "person_household_id"].value_counts().to_dict() + ) + donor_ids = household.loc[is_donor, "household_id"].tolist() + without_one = sum( + 1 for value in donor_ids if carriers_per_household.get(value, 0) != 1 + ) + if without_one or len(carriers_per_household) != len(donor_ids): + layer["donors_without_exactly_one_carrier"] = int(without_one) + band_rows: list[dict[str, object]] = [] + unequal = 0 + for band in SPI_INCOME_BAND_DONOR_LOWER_BOUNDS: + band_weights = weights[is_donor & (donor_band == band)] + if band_weights.size and not np.allclose( + band_weights, band_weights[0], rtol=1e-9, atol=1e-6 + ): + unequal += 1 + band_rows.append( + { + "lower_bound": int(band), + "donor_households": int(band_weights.size), + "donor_weight": (float(band_weights[0]) if band_weights.size else None), + "weighted_taxpayers": float(band_weights.sum()), + } + ) + if unequal: + layer["bands_with_unequal_donor_weights"] = unequal + if layer: + problems["band_donor_layer"] = layer + receipt["bands"] = band_rows + + # The weights the stage read: every region's incumbents gave up the + # region's donor mass in proportion, so scaling them back to the region's + # whole mass restores them. + strata = _funding_strata(household).to_numpy() + region_mass = pd.Series(weights).groupby(strata).sum() + incumbent_mass = pd.Series(weights[~is_donor]).groupby(strata[~is_donor]).sum() + if (incumbent_mass.reindex(region_mass.index).fillna(0.0) <= 0.0).any(): + problems["band_donor_funding"] = "a donor stratum has no incumbent mass" + return receipt + factor = incumbent_mass / region_mass.reindex(incumbent_mass.index) + receipt["funding"] = { + str(stratum): float(value) for stratum, value in factor.sort_index().items() + } + pre_weights = weights[~is_donor] / pd.Series(strata[~is_donor]).map( + factor + ).to_numpy(dtype=float) + pre_household = ( + household.loc[~is_donor] + .drop( + columns=[ + HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR, + SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN, + ] + ) + .reset_index(drop=True) + ) + donor_id_set = set(donor_ids) + pre_person_rows = ~person["person_household_id"].isin(donor_id_set) + pre_person = ( + person.loc[pre_person_rows] + .drop(columns=[PERSON_IS_SPI_INCOME_BAND_CARRIER]) + .reset_index(drop=True) + ) + pre_benunit = benunit.loc[ + benunit["benunit_id"].isin(set(pre_person["person_benunit_id"])) + ].reset_index(drop=True) + stored_carriers = set( + pd.to_numeric(person.loc[carriers, "person_source_id"], errors="raise") + .astype("int64") + .tolist() + ) + + if propensity is None or band_taxpayers is None: + receipt["seating_recomputed"] = False + receipt["seating_scope"] = ( + "not recomputed: the propensities come from the licensed SPI tape; " + "pass --spi-tab to rerun the seating from ids" + ) + return receipt + period = int(uk_time_period(frame)) + + def recompute(person_t, household_t, weights_t): + result = stack_spi_income_band_donors( + uk_national_frame( + person=person_t, + benunit=pre_benunit, + household=household_t, + time_period=uk_time_period(frame), + weight_kind=uk_household_weight_kind(frame), + household_weights=weights_t, + ), + propensity=propensity, + band_taxpayers=band_taxpayers, + seed=SPI_INCOME_BAND_DONOR_SEED, + taxpayer_period=period, + ) + table = result.frame.table("household") + seated = table[HOUSEHOLD_IS_SPI_INCOME_BAND_DONOR].astype(bool) + seats = pd.Series( + table.loc[seated, SPI_INCOME_BAND_DONOR_LOWER_BOUND_COLUMN].to_numpy(float), + index=pd.to_numeric(table.loc[seated, "household_source_id"]) + .astype("int64") + .to_numpy(), + ).sort_index() + people = result.frame.table("person") + carrier_ids = set( + pd.to_numeric( + people.loc[ + people[PERSON_IS_SPI_INCOME_BAND_CARRIER].astype(bool), + "person_source_id", + ] + ) + .astype("int64") + .tolist() + ) + return seats, carrier_ids, result.evidence() + + seats, carrier_ids, evidence = recompute(pre_person, pre_household, pre_weights) + receipt["seating_recomputed"] = True + receipt["seating_scale"] = evidence["seating_scale"] + receipt["mass_scale"] = evidence["mass_scale"] + missing = int((~stored_seats.index.isin(seats.index)).sum()) + extra = int((~seats.index.isin(stored_seats.index)).sum()) + if missing or extra: + problems["band_donor_seats_stored"] = {"missing": missing, "extra": extra} + else: + moved = int((seats.to_numpy() != stored_seats.to_numpy()).sum()) + if moved: + problems["band_donor_bands_stored"] = moved + if carrier_ids != stored_carriers: + problems["band_donor_carriers_stored"] = len(carrier_ids ^ stored_carriers) + recomputed_weight = { + int(row["lower_bound"]): float(row["donor_weight"]) for row in evidence["bands"] + } + weight_off = [ + int(row["lower_bound"]) + for row in band_rows + if row["donor_weight"] is None + or not np.isclose( + row["donor_weight"], + recomputed_weight[int(row["lower_bound"])], + rtol=1e-9, + atol=1e-6, + ) + ] + if weight_off: + problems["band_donor_weights_stored"] = weight_off + + # A frame keeps its group tables sorted by id, so the permutation is over + # the person rows, the table the candidates are read from. + rng = np.random.default_rng(permutation_seed) + permuted_seats, permuted_carriers, _ = recompute( + pre_person.iloc[rng.permutation(len(pre_person))].reset_index(drop=True), + pre_household, + pre_weights, + ) + if not (permuted_seats.equals(seats) and permuted_carriers == carrier_ids): + problems["band_donor_seats_permutation"] = True + return receipt + + +def _band_donor_seating_inputs( + frame, spi_tab: Path | None +) -> tuple[pd.DataFrame | None, dict[int, float] | None]: + """The tape's band propensities and the build year's Table 2.5 taxpayers. + + Built exactly as the stage builds them, from the verified licensed tape; + ``(None, None)`` without one. + """ + + from microcosm.build.uk_runtime.spi_band_donors import ( + load_hmrc_itl_band_taxpayers, + spi_income_band_donor_propensity, + ) + from microcosm.build.uk_runtime.spi_income import ( + load_spi_donor_age_model, + verify_spi_donor_identity, + ) + + if spi_tab is None: + return None, None + identity = verify_spi_donor_identity(spi_tab) + period = int(uk_time_period(frame)) + propensity = spi_income_band_donor_propensity( + pd.read_csv(identity.path, delimiter="\t"), + age_model=load_spi_donor_age_model(period), + ) + return propensity, load_hmrc_itl_band_taxpayers(period) + + def e8_identity_receipt( frame, *, permutation_seed: int, + spi_tab: Path | None = None, ) -> dict[str, object]: """Receipt E8 deterministic layers under row permutation by entity id. @@ -1031,7 +1546,21 @@ def e8_identity_receipt( in original and permuted row order, against the stored copies, counts, family weights, flags and mass record; (3) the student-loan plan column recomputed in full (identity-keyed top-ups at the release calibration - year) in original and permuted row order against the stored column. + year) in original and permuted row order against the stored column; + (4) the SPI income band donors (microcosm#1063) - the stored layer's + structure, equal band weights and conserving mass record, and, with the + licensed SPI tape (``spi_tab``), the identity-keyed seating rerun on the + reconstructed pre-donor frame in original and permuted row order; + (5) the CGT residential split (microcosm#1063) recomputed from the folded + pre-split frame with the stage's own function - the solved probability + carried as arm weights - in original and permuted row order against the + stored arms, indices, weights, probabilities and residential gains. The + clone-pair, support-split and band-donor checks run on the frame with the + arms folded onto their sources, the state those layers were written in. + Every CGT layer ran before ``salary_sacrifice``, whose conversion + rewrites a converted record's pay in place, so the recomputes first + reverse it from the stage's pre-conversion pay carrier + (``_pre_salary_sacrifice_frame``). The A&S prior amounts (overwritten by the Table 3 redraw and its sub-AEA remainder mapping), the redraw's seeded within-band draws (covered by the merged #560 embedded published-surface tests), and the @@ -1039,7 +1568,13 @@ def e8_identity_receipt( consumed by the stage) are covered by twin-build determinism. """ + from microcosm.build.uk_runtime.cgt_asset_type import ( + load_hmrc_cgt_asset_type_facts, + ) from microcosm.build.uk_runtime.cgt_imputation import uk_cgt_policy_parameters + from microcosm.build.uk_runtime.cgt_residential_split import ( + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + ) from microcosm.build.uk_runtime.cgt_structure import HOUSEHOLD_IS_CGT_CLONE from microcosm.build.uk_runtime.cgt_support import ( CGT_SUPPORT_COPIES_COLUMN, @@ -1056,31 +1591,61 @@ def e8_identity_receipt( problems: dict[str, object] = {} person = frame.table("person") - household = frame.table("household") + + # The split's income-band proxy, the anchor and the residential split + # read the same policy parameters (the tapered Personal Allowance and + # the annual exempt amount) at the frame's build period. + parameters = uk_cgt_policy_parameters(uk_time_period(frame)) + + # Every CGT layer ran before salary_sacrifice, which rewrites a converted + # record's pay and employee contribution in place; the recomputes band + # on the income those stages saw, restored from the stage's carrier. + pre_sacrifice = _pre_salary_sacrifice_frame(frame) + + # (5) Residential split recomputed from the folded pre-split frame; the + # CGT layers below it are then checked with the arms folded away. + residential_split = _e8_residential_split( + pre_sacrifice, + problems, + parameters=parameters, + facts=load_hmrc_cgt_asset_type_facts(), + permutation_seed=permutation_seed, + ) + cgt_frame = _drop_stacked_layers( + pre_sacrifice, [HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE] + ) + household = cgt_frame.table("household") # (1) Clone-pair structure over every household: the support split's # copies are pre-clone rows with clones of their own. is_clone = household[HOUSEHOLD_IS_CGT_CLONE].astype(bool).to_numpy() if int((~is_clone).sum()) != int(is_clone.sum()): problems["clone_half_counts"] = [int((~is_clone).sum()), int(is_clone.sum())] - # The split's income-band proxy and the anchor read the same policy - # parameters (the tapered Personal Allowance and the annual exempt - # amount) at the frame's build period. - parameters = uk_cgt_policy_parameters(uk_time_period(frame)) - clone_pairs = _e8_clone_pairs(frame, problems, parameters=parameters) + clone_pairs = _e8_clone_pairs(cgt_frame, problems, parameters=parameters) # (2) Support split recomputed from the folded pre-split frame. Same # contract as the E6 NHS check: age_tail runs immediately after # frs_spine, so the split classified each household by its oldest-adult # carrier on the disaggregated age surface stored in the artifact. support_split = _e8_support_split( - frame, + cgt_frame, problems, distribution=load_hmrc_cgt_joint_distribution(), parameters=parameters, permutation_seed=permutation_seed, ) + # (4) SPI income band donors: the seating reconstructed from ids, on the + # arm-free frame (the donor stage ran before the residential split). + propensity, band_taxpayers = _band_donor_seating_inputs(frame, spi_tab) + band_donors = _e8_band_donors( + cgt_frame, + problems, + propensity=propensity, + band_taxpayers=band_taxpayers, + permutation_seed=permutation_seed, + ) + # (3) Student-loan plan recomputed in full. stocks = load_slc_liable_stocks() year = load_uk_frs_release().calibration_year @@ -1091,7 +1656,18 @@ def e8_identity_receipt( .set_index("person_id")["student_loan_plan"] .reindex(stored_plan.index) ) - plan_matches_store = bool(stored_plan.equals(recomputed_plan)) + + # Compared by value: the stored column comes back from HDF5 under one + # string dtype and the recomputed one under another, and ``Series.equals`` + # calls two equal columns of different string dtypes unequal. + def plans_equal(left: pd.Series, right: pd.Series) -> bool: + return bool( + np.array_equal( + left.astype(object).to_numpy(), right.astype(object).to_numpy() + ) + ) + + plan_matches_store = plans_equal(stored_plan, recomputed_plan) permuted_result = assign_student_loan_plans( _reverse_rows(frame), stocks=stocks, year=year ) @@ -1100,7 +1676,7 @@ def e8_identity_receipt( .set_index("person_id")["student_loan_plan"] .reindex(stored_plan.index) ) - plan_permutation_stable = bool(recomputed_plan.equals(permuted_plan)) + plan_permutation_stable = plans_equal(recomputed_plan, permuted_plan) if not plan_matches_store: problems["student_loan_plan_stored"] = True if not plan_permutation_stable: @@ -1114,7 +1690,10 @@ def e8_identity_receipt( key.endswith("_permutation") for key in problems ), "clone_pairs": clone_pairs, + "pre_salary_sacrifice_restored": pre_sacrifice is not frame, "support_split": support_split, + "band_donors": band_donors, + "residential_split": residential_split, "permutation_mismatches": { key: value for key, value in problems.items() @@ -1138,16 +1717,30 @@ def e8_identity_receipt( "the anchor each cost one float rounding generation); " "support-split mass record: new_total against old_total at rtol " "1e-9; support-split families, copy counts, flags and " - "student_loan_plan: exact equality" + "student_loan_plan: exact equality; band-donor seats, bands and " + "carriers: exact equality; band-donor weights within a band and " + "against the rerun: rtol 1e-9 / atol 1e-6; band-donor mass " + "record: new_total against old_total at rtol 1e-9 with declared " + "factor one; residential-split arms, indices and residential " + "gains: exact equality; residential-split weights against the " + "rerun: rtol 1e-9 / atol 1e-6; probabilities: rtol 1e-9; mass " + "record: new_total against old_total at rtol 1e-9 with declared " + "factor one" ), "columns_by_entity": { "household": [ HOUSEHOLD_IS_CGT_CLONE, HOUSEHOLD_IS_CGT_SUPPORT_COPY, CGT_SUPPORT_COPIES_COLUMN, + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + "cgt_residential_clone_index", "household_weight", ], - "person": ["student_loan_plan"], + "person": [ + "student_loan_plan", + "cgt_residential_probability", + "capital_gains_residential_property", + ], }, "qrf_draw_columns_scope": ( "excluded: the A&S prior amounts (overwritten by the Table 3 " @@ -1303,6 +1896,16 @@ def main() -> int: "--check", choices=("e4", "e5", "e6", "e7", "e8", "e9"), default="e4" ) parser.add_argument("--permutation-seed", type=int, default=123) + parser.add_argument( + "--spi-tab", + type=Path, + default=None, + help=( + "Licensed SPI public use tape (put2223uk.tab). With it the e8 " + "receipt reruns the income band donor seating from ids; without " + "it that layer is checked for structure and mass only." + ), + ) args = parser.parse_args() frame, _provenance = load_uk_national_frame(args.input_h5) @@ -1377,6 +1980,7 @@ def main() -> int: receipt = e8_identity_receipt( frame, permutation_seed=args.permutation_seed, + spi_tab=args.spi_tab, ) ok = bool( receipt["identical_under_permutation"] and receipt["matches_stored_columns"] @@ -1403,6 +2007,10 @@ def main() -> int: "household_is_spi_synthetic": "spi_support_channel", "household_is_capital_gains_clone": "cgt_incidence_clone", # E8 clone "household_is_cgt_support_copy": "cgt_support_split", # E8 support copies + # microcosm#1063: the residential split's arms carry their source's clone + # and copy flags, so dropping the clone layer drops them too; dropping + # this layer alone folds each arm onto its source. + "household_is_cgt_residential_clone": "cgt_residential_split", } _STACKED_ROW_FLAGS = tuple(_STACKING_STAGE_BY_FLAG) @@ -1437,22 +2045,175 @@ def _pre_clone_household_weights(frame) -> np.ndarray: without the clone layer is returned unchanged. """ + from microcosm.build.uk_runtime.cgt_residential_split import ( + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + ) from microcosm.build.uk_runtime.cgt_structure import ( HOUSEHOLD_IS_CGT_CLONE, pair_clone_households, ) household = frame.table("household") - weights = np.asarray(frame.weights_for("household").values, dtype=float).copy() + person = frame.table("person") + benunit = frame.table("benunit") + weights = _pre_residential_household_weights(frame) if HOUSEHOLD_IS_CGT_CLONE not in household.columns: return weights - clone_positions, original_positions, _ = pair_clone_households( + # The residential arms (microcosm#1063) are stacked after the clone, so + # the clone pairing runs on the frame without them, their mass already + # folded onto their sources. + keep = np.ones(len(household), dtype=bool) + if HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE in household.columns: + keep &= ~household[HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE].to_numpy(dtype=bool) + if keep.all(): + clone_positions, original_positions, _ = pair_clone_households( + person, benunit, household + ) + else: + kept_household = household.loc[keep].reset_index(drop=True) + kept_ids = set(kept_household["household_id"].tolist()) + kept_person = person.loc[ + person["person_household_id"].isin(kept_ids) + ].reset_index(drop=True) + kept_benunit = benunit.loc[ + benunit["benunit_id"].isin(set(kept_person["person_benunit_id"])) + ].reset_index(drop=True) + clone_kept, original_kept, _ = pair_clone_households( + kept_person, kept_benunit, kept_household + ) + kept_positions = np.flatnonzero(keep) + clone_positions = kept_positions[clone_kept] + original_positions = kept_positions[original_kept] + weights[original_positions] += weights[clone_positions] + return weights + + +def _pre_residential_household_weights(frame) -> np.ndarray: + """Household weights with every residential arm folded onto its source. + + The arms of ``cgt_residential_split`` (microcosm#1063) divide a + household's mass by the solved residential probabilities, so summing + them back onto the arm that kept the household's ids is exact. An + artifact without the arm layer is returned unchanged. The arms keep their + own weights in the returned vector; the scoping that calls this drops + those rows. + """ + + from microcosm.build.uk_runtime.cgt_residential_split import ( + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + ) + + household = frame.table("household") + weights = np.asarray(frame.weights_for("household").values, dtype=float).copy() + if HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE not in household.columns: + return weights + lineage = _residential_arm_lineage( frame.table("person"), frame.table("benunit"), household ) - weights[original_positions] += weights[clone_positions] + np.add.at(weights, lineage.source_positions, weights[lineage.arm_positions]) return weights +class _ResidentialArmLineage(NamedTuple): + """Where every residential arm sits and which source it came from.""" + + household_ids: np.ndarray + pre_split: np.ndarray + arm_positions: np.ndarray + source_positions: np.ndarray + arm_index: np.ndarray + multiplier: int + + +def _residential_arm_lineage( + person: pd.DataFrame, benunit: pd.DataFrame, household: pd.DataFrame +) -> _ResidentialArmLineage: + """Every residential arm's position, its source's position and its index. + + The split offsets ids by ``id_multiplier_for_values`` over the rows it + saw, every household of the frame before it ran: the households whose + arm flag is false. Arm ``j`` of source ``r`` carries ``r + j * M``, so + ``j = id // M`` and ``r = id - j * M``. An arm without a source, or with + index zero, fails closed, and the stored ``cgt_residential_clone_index`` + must agree with the id scheme on every row. + """ + + from microcosm.build.uk_runtime.cgt_residential_split import ( + CGT_RESIDENTIAL_CLONE_INDEX_COLUMN, + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + ) + from microcosm.build.uk_runtime.rowwise_geography import id_multiplier_for_values + + household_ids = ( + pd.to_numeric(household["household_id"], errors="raise") + .astype("int64") + .to_numpy() + ) + if len(np.unique(household_ids)) != len(household_ids): + raise ValueError("CGT residential-arm lineage requires unique household ids.") + is_arm = household[HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE].to_numpy(dtype=bool) + if CGT_RESIDENTIAL_CLONE_INDEX_COLUMN not in household.columns: + raise ValueError( + f"CGT residential-arm lineage requires {CGT_RESIDENTIAL_CLONE_INDEX_COLUMN!r}." + ) + stored_index = ( + pd.to_numeric(household[CGT_RESIDENTIAL_CLONE_INDEX_COLUMN], errors="raise") + .astype("int64") + .to_numpy() + ) + flag_disagreements = int(((stored_index != 0) != is_arm).sum()) + if flag_disagreements: + raise ValueError( + "CGT residential-arm flag and stored arm index disagree on " + f"{flag_disagreements} household(s)." + ) + pre_split = ~is_arm + pre_split_ids = set(household_ids[pre_split].tolist()) + person_pre_split = person["person_household_id"].isin(pre_split_ids).to_numpy() + benunit_pre_split = ( + benunit["benunit_id"] + .isin(set(person.loc[person_pre_split, "person_benunit_id"].tolist())) + .to_numpy() + ) + multiplier = id_multiplier_for_values( + person.loc[person_pre_split, "person_id"], + person.loc[person_pre_split, "person_household_id"], + person.loc[person_pre_split, "person_benunit_id"], + benunit.loc[benunit_pre_split, "benunit_id"], + household_ids[pre_split], + ) + arm_positions = np.flatnonzero(is_arm) + arm_ids = household_ids[arm_positions] + arm_index = arm_ids // multiplier + source_ids = arm_ids - arm_index * multiplier + position_by_id = {int(value): index for index, value in enumerate(household_ids)} + source_positions = np.empty(len(arm_positions), dtype=np.int64) + for slot, (arm_id, index, source_id) in enumerate( + zip(arm_ids.tolist(), arm_index.tolist(), source_ids.tolist(), strict=True) + ): + source = position_by_id.get(int(source_id)) + if index < 1 or source is None or not pre_split[source]: + raise ValueError( + "CGT residential arm without a source household: household_id " + f"{arm_id} (id multiplier {multiplier})." + ) + source_positions[slot] = source + scheme_disagreements = int((stored_index[arm_positions] != arm_index).sum()) + if scheme_disagreements: + raise ValueError( + f"CGT residential arm index stored on {scheme_disagreements} arm(s) " + f"disagrees with the id scheme (id // {multiplier})." + ) + return _ResidentialArmLineage( + household_ids=household_ids, + pre_split=pre_split, + arm_positions=arm_positions, + source_positions=source_positions, + arm_index=arm_index, + multiplier=multiplier, + ) + + class _SupportCopyLineage(NamedTuple): """Where every CGT support copy sits and which root it came from.""" @@ -1679,9 +2440,11 @@ def _frs_only_frame(frame): population the pre-stacking stages actually drew for. Their weights fold each clone back onto its original (exact before and after the #970 anchor) and each CGT support copy back onto its root, so every surviving - row carries its pre-split weight. + row carries its pre-split weight. Every pre-stacking stage precedes + salary_sacrifice, so the conversion's pay rewrite is reversed too. """ + frame = _pre_salary_sacrifice_frame(frame) household = frame.table("household") return _drop_stacked_layers( frame, [flag for flag in _STACKED_ROW_FLAGS if flag in household.columns] @@ -1689,14 +2452,102 @@ def _frs_only_frame(frame): def _frame_as_stage_saw(frame, stage: str): - """Scope the artifact to the rows present when ``stage`` ran.""" + """Scope the artifact to the rows present when ``stage`` ran. + + A stage before ``salary_sacrifice`` also saw every converted record's + pay and employee contribution before the conversion rewrote them. + """ + positions = _roster_positions() + if positions[stage] < positions[_SALARY_SACRIFICE_STAGE]: + frame = _pre_salary_sacrifice_frame(frame) return _drop_stacked_layers( frame, _flags_stacked_after(stage, frame.table("household")) ) +#: The stage whose conversion rewrites pay in place (microcosm#1069 c9). +_SALARY_SACRIFICE_STAGE = "salary_sacrifice" + + +def _pre_salary_sacrifice_frame(frame): + """The artifact with the salary-sacrifice conversion reversed. + + ``salary_sacrifice`` moves a converted record's employee pension + contribution whole into ``pension_contributions_via_salary_sacrifice``, + zeroes the source and lowers ``employment_income`` by the amount moved, + after the capital-gains stages have banded every carrier on that pay + (microcosm#1063: on the 2 October spine the E8 recomputes banded 2,797 + clone carriers on post-sacrifice pay and derived different anchor targets + and a different support family). The stage keeps a converted record's + pre-conversion pay on ``SALSAC_PRE_CONVERSION_PAY_COLUMN`` (zero for + everyone else), so the rewrite reverses exactly: pay back to the carrier + and the employee contribution back from the salary-sacrifice column, + which the conversion set to zero on the record it converted. The stage's + QRF predictions for the records it did not convert stay as stored; no + receipted layer before the stage reads them. An artifact without the + carrier is returned as it is. + """ + + from microcosm.build.uk_runtime.national_frame import ( + uk_household_weight_kind, + uk_national_frame, + ) + from microcosm.build.uk_runtime.salary_sacrifice import ( + SALSAC_OUTPUT, + SALSAC_PRE_CONVERSION_PAY_COLUMN, + ) + + person = frame.table("person") + if SALSAC_PRE_CONVERSION_PAY_COLUMN not in person.columns: + return frame + carrier = pd.to_numeric( + person[SALSAC_PRE_CONVERSION_PAY_COLUMN], errors="raise" + ).to_numpy(dtype=float) + if not np.isfinite(carrier).all() or (carrier < 0.0).any(): + raise ValueError( + f"{SALSAC_PRE_CONVERSION_PAY_COLUMN} must be finite and non-negative." + ) + converted = carrier > 0.0 + if not converted.any(): + return frame + person = person.copy() + pay = pd.to_numeric(person["employment_income"], errors="raise").to_numpy( + dtype=float, copy=True + ) + employee = pd.to_numeric( + person["employee_pension_contributions"], errors="raise" + ).to_numpy(dtype=float, copy=True) + sacrifice = pd.to_numeric(person[SALSAC_OUTPUT], errors="raise").to_numpy( + dtype=float, copy=True + ) + if (employee[converted] != 0.0).any() or (sacrifice[converted] <= 0.0).any(): + raise ValueError( + "A converted salary-sacrifice record must carry a zero employee " + "contribution and a positive salary sacrifice; the artifact's " + f"{SALSAC_PRE_CONVERSION_PAY_COLUMN} disagrees with its columns." + ) + pay[converted] = carrier[converted] + employee[converted] = sacrifice[converted] + sacrifice[converted] = 0.0 + person["employment_income"] = pay + person["employee_pension_contributions"] = employee + person[SALSAC_OUTPUT] = sacrifice + return uk_national_frame( + person=person, + benunit=frame.table("benunit"), + household=frame.table("household"), + time_period=uk_time_period(frame), + weight_kind=uk_household_weight_kind(frame), + household_weights=frame.weights_for("household").values, + mass_log=frame.mass_log, + ) + + def _drop_stacked_layers(frame, flags: Sequence[str]): + from microcosm.build.uk_runtime.cgt_residential_split import ( + HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE, + ) from microcosm.build.uk_runtime.cgt_structure import HOUSEHOLD_IS_CGT_CLONE from microcosm.build.uk_runtime.cgt_support import HOUSEHOLD_IS_CGT_SUPPORT_COPY from microcosm.build.uk_runtime.national_frame import ( @@ -1726,6 +2577,8 @@ def _drop_stacked_layers(frame, flags: Sequence[str]): all_weights = _pre_split_household_weights(frame) elif HOUSEHOLD_IS_CGT_CLONE in flags: all_weights = _pre_clone_household_weights(frame) + elif HOUSEHOLD_IS_CGT_RESIDENTIAL_CLONE in flags: + all_weights = _pre_residential_household_weights(frame) else: all_weights = np.asarray(frame.weights_for("household").values, dtype=float) weights = all_weights[keep.to_numpy()]