Skip to content

UK: port the unblocked uk-data fixes tracked in #1095 - #1100

Merged
juaristi22 merged 20 commits into
mainfrom
uk-data-ports-1095
Oct 5, 2026
Merged

juaristi22 merged 20 commits into
mainfrom
uk-data-ports-1095

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #1095.

Summary

This PR ports to the microcosm UK spine the uk-data fixes from the post-AB2026 wave that #1095 tracks and that need no policyengine-uk change. It covers nine of the tracker's items in 13 commits, plus two register retirements, the review-round fixes and the receipts note. Each commit carries its own changelog fragment with the measured before-state.

It was stacked on #1089. Since #1089 merged, it targets main and is rebased onto 48fa0616.

  • Benefit target scope (uk-data#490 follow-up).
    • 1d1c10da3: the DWP and OBR table 4.9 benefit rows now bind the households their publisher counts. That is Great Britain for the DWP social security rows. It is England and Wales for Attendance Allowance, Carer's Allowance, PIP and Winter Fuel Payment, since their Scottish successors transferred.
    • 3c1b33f34: obr.pip binds the engine's pip + dla, the EFO line's "DLA and PIP".
  • Scottish disability payments (uk-data#500).
    • 04ea9943b: FRS codes 117/118 (Adult Disability Payment) and 121/122 (Child Disability Payment) land in the PIP and DLA columns they replace.
  • FRS dividends (uk-data#498).
    • 90621f74a: the SPI income stage no longer redraws FRS respondents' dividend_income from the SPI forest.
  • Dependants aged 16 to 19 (uk-data#504).
    • f0c30271d: SPI draws go only to FRS claimants and partners (is_uc_claimant), so dependants keep their FRS twin's values.
    • Every other recipient keeps its exact draw.
    • Band-donor carriers are claimants and partners too.
  • Council tax (uk-data#496/Adjudicate the JCX-35-25 OBBBA no-tax anchors: fenced parity exclusion + ledgered probe receipts (#451 items 3-4) #499).
    • 07e5b0f9c: council_tax is the bill before council tax reduction. The Scottish water and sewerage charges are netted at the gross cells.
    • 746940855 and d89a3ddfe: the Scottish water charge is the amount paid. That is the gross charge less the status discount, topped up to the reduction scheme's 35% for council tax reduction recipients (round 1, item 4).
  • SPI benefit coherence (uk-data#514).
    • 790f36ffd: new stage spi_benefit_coherence after uc_reporter_redraw, with the release-blocking gate uk_stage_spi_benefit_coherence.
    • It zeroes the SPI rows' legacy and out-of-work reports, with income-related ESA added in eaf7e80cf (round 1, item 3). It restores IIDB, AFCS and bereavement support from the FRS twin, and re-derives the disability flags and receives_benefits_in_own_right.
    • It redraws would_claim_uc for SPI units as frs_take_up draws it.
  • SPI capital (uk-data#495).
    • c58fc84d3: every SPI benefit unit's TOTCAPB4 is redrawn conditioned on its financial investment income, with declared coarsening.
  • WAS predictors (uk-data#486/UK dense/local epic: rowwise constituency-grain dataset by cloning the national staging pool (US dense-arm analog) #495).
    • adfcc079e: the donor reads DVCTaxAmtAnnualR8 and NumChildR8, the recipient counts come from is_uc_claimant, and a rental_income predictor is added.
  • Buy-to-let (uk-data#501).
    • 49d26eceb: other_residential_property_value includes DVBltValR8_sum.
  • Age-18 child elsewhere (uk-data#486).
    • 39af74a21: the ETB VAT and services adult and child counts come from FRS roles.
    • 8ede090d8: constituency UC child bands use the national UC family-type composition, through one shared helper.
  • Registers.
  • Receipts.
    • 1a4f934ce and 479d923ce: experiments/1095-uk-data-ports-receipts.md.

María's rulings of 2 October are applied:

  • buy-to-let goes into other_residential_property_value;
  • Scottish council tax and its water charges follow two rules, the netting and the charge (the charge rule is refined in round 1, below);
  • obr.pip binds PIP + DLA over England and Wales;
  • the full SPI coherence stage goes in, including the UC take-up redraw.

Review round 1 (2026-10-04)

Vahid's round 1 is addressed at this head:

  1. Rebase. Every commit replays onto main, and the coverage manifest and H2 fixture were regenerated after each one. The gate digests did not move (repin_digests --check: 0 of 12). Of the inherited failures, only the target-references regeneration remains, and it fails on main too (see Notes).
  2. VAT. No change in this PR; the decision is María's (below).
  3. Income-related ESA (eaf7e80cf). It is zeroed on SPI rows with income-based JSA and Income Support, as uk-data#514 does.
    • policyengine-uk pays esa_income_reported as the award behind a capital test alone.
    • On A8 the SPI rows held 29% of its reporters, and 56% of their benefit units had incomes above £12,570.
    • Contributory ESA keeps its draw. It is open to new claims and not income-tested, and on A8 its SPI draw sits at half the FRS rate per working-age adult, with 9.9% of reporters above the permitted-work limit.
  4. Scottish water (d89a3ddfe). The 2021–27 scheme reduces a recipient's charges by R = 35 × (A/B) − D points on top of the status discount D, so the combined reduction is max(D, 35% × A/B).
    • C3b stacked the 35% on the discount: a single person on full reduction paid 48.75% of the gross charge instead of 65%.
    • Every recipient now takes the full-reduction case. 86% of Scotland's recipients receive full reduction, but the FRS-recorded amounts carry none (median £6 a week against the published £16.55 average).
    • The netting of CTANNUAL keeps 0.65.
  5. Arm-to-head mapping. A9 builds d89a3ddfe. The later commits change only a test, the input-mass register and the receipts, not spine or calibration code.

The round also retires three input-mass exclusions (a0c678c21). C6 brings income-based JSA and Working Tax Credit back inside the 452% fence, and A9's calibration no longer pushes the access-fund household to the weight bound. The release-cut gate fails stale entries, so leaving them would block the next release on main.

Not in this PR

Needs María's decision or signature (not committed)

Proposed texts

New entry: council-tax-gross-of-reduction (C3a, commit 07e5b0f)

{
 "id": "council-tax-gross-of-reduction",
 "class": "mechanism_change",
 "scope": {
  "surface": "weighted_totals",
  "columns": ["council_tax"],
  "entities": ["household"]
 },
 "expectation": "column_differs",
 "magnitude_evidence": "FRS 2024-25 CTANNUAL is the annual bill after discounts and the reported council tax reduction (SN 9563 DV summary), while the engine reads council_tax as the liability before the reduction, so the incumbent and our earlier build both entered reduction recipients net of their reduction: 20.6% of English FRS-channel reduction reporters on the 30 September national build had a GBP 0 bill and 25.6% a bill below their own reduction. A household reporting a reduction (CTREB 1) now gets its CTREBAMT added back, a full reduction included; a recipient with no positive amount takes the larger of its bill and its cell mean; the region, band and single-adult cells pool non-recipients only; and Scottish water and sewerage charges are netted at the gross cells, in full for non-recipients and at 65% for recipients, the shares DWP's CTANNUAL carries (uk-data#496/#499). The add-back on the FRS channel is GBP 0.78bn in England, 0.05bn in Wales and 0.06bn in Scotland; on the microcosm#1095 arm spines, council_tax at design weights moves from GBP 42.27bn to 43.57bn across England, Wales and Scotland (England 37.0bn to 38.3bn, Wales 2.08bn to 2.18bn, Scotland 3.21bn to 3.06bn, where the gross water and sewerage charges now come off non-recipients' bills in full), and the reduction reporters with a GBP 0 bill fall from 10.6% to 0.",
 "evidence": "experiments/1095-uk-data-ports-receipts.md#spine-receipts-by-commit",
 "adjudicator": "juaristi22",
 "adjudicated_on": "[on signature]",
 "quantitative": {
  "weighted_totals": {
   "expected_columns": ["council_tax"]
  }
 }
}

Amended entry: scottish-water-sewerage-successor-level (C3b and R2, commits 7469408 and d89a3dd)

Only magnitude_evidence changes. The current text says the amount netted
from council_tax equals the charge. That stops being true at C3a, because
the netting follows DWP's derivation and the charge follows what the
household pays.

Proposed magnitude_evidence:

The Scottish charge is assembled from the successors FRS 2024-25 published for the cells it retired. CWATAMT1 and CSEWAMT1 are the gross charges on the bill; the household pays them less the status discount its council tax records (CTDISC with CT25D50D: 25% or 50%), and for a council tax reduction recipient the Water Charges Reduction Scheme tops that discount up to 35% of the gross charges. The 2021-27 scheme reduces a recipient's charges by R = 35 x (A/B) - D points of the gross charge, unless negative, on top of the discount D, where A is the council tax reduction and B the council tax before it (Scottish Government, Water services - charging principles: 2021 to 2027, Annex A); every recipient takes the full-reduction case A/B = 1, because 86% of Scotland's recipients receive full council tax reduction (March 2025) while none of the FRS 2024-25 Scottish recipients with a recorded CTREBAMT pays nothing after it, so the recorded amount cannot carry the share. A household with no gross water cell pays its recorded CWATAMTD (uk-data#499, microcosm#1095). The earlier rule read CWATAMTD/CWATAMT1 as the household's discount factor, but CWATAMTD sits at a median 0.83 of the gross water charge where no discount applies, which put the charge about 13-16% below the gross charges less the discount. Weighted annual per Scottish household with a charge moves from about GBP 404 to about 474 on the microcosm#1095 arm spines (A3 to A9), against roughly GBP 490 for England and Wales on WATSEWRT; the incumbent sits at zero because of the separate NaN defect. council_tax nets the gross charges at the shares DWP's CTANNUAL carries (in full for non-recipients, 65% for recipients), which is not the amount charged, so the two columns no longer move by one construction; council_tax's own move is signed under council-tax-gross-of-reduction. The nonzero share is unaffected, so this entry deliberately does not sign the share surface.

Scope and quantitative block: unchanged (weighted_totals on
water_and_sewerage_charges and council_tax). Narrowing the columns to
water_and_sewerage_charges alone is an option, since the new entry signs
council_tax; that is María's call.

Amended adjudication: uc_unit_vs_household_grain (C9b, commit 8ede090)

packages/microcosm-build/src/microcosm/build/uk/local_binding_adjudications.json,
key uc_unit_vs_household_grain. Only reason changes; approved_by,
approved_on and expires_on follow the signature. The current text says
the bands use each unit's num_children and match the national re-bind.
They did not match until C9b. Now they match, but no longer through
num_children.

Proposed reason:

Replace the stated basis with the named alternative: the uc_households family binds at benefit-unit grain. uc_households per area counts UC benefit units mapped to their households (the DWP claim-household definition), and the four children bands classify each UC benefit unit by the members DWP's UC family type counts as children (uc_relationships.uc_family_child_member: never a claimant or partner; any UC child or qualifying young person, or any other member under 20), summed benefit unit to household. That is the composition the national dwp.uc.households* re-bind of PR #813 binds (microcosm#1095; before it the bands read the engine's age-18 num_children, which put 2.77% of UC-reporting units in a different band). The household-children basis is retired. Fence evidence: uk_local_target_census.json#/binding_fences/uc_unit_vs_household_grain; runtime: microcosm.build.uk_runtime.local_targets.compute_household_metrics.

Amended census fence rule: uc_unit_vs_household_grain (C9b)

packages/microcosm-build/src/microcosm/build/uk_runtime/local_target_census.py
(the rule of the _UC_GRAIN_FENCE_ID fence; regenerated into
uk/uk_local_target_census.json). Proposed rule:

The runtime maps benefit-unit-level UC receipt and each UC benefit unit's UC family-type child-count band (uc_relationships.uc_family_child_member, the national UC-by-children composition) onto household rows by sum: a physical household containing two UC benefit units contributes 2 to uc_households and one contribution to each unit's child band. This aligns the runtime with DWP UC claim households, which correspond to benefit units rather than physical FRS households; binding must preserve this reviewed grain crosswalk.

Measurement

Ten licensed arms ran. Each builds a spine and runs the national calibration against Chronicle 825406f, with local staging only, and nothing was released. The full receipts are in experiments/1095-uk-data-ports-receipts.md.

  • A0 to A8, on UK spine follow-ups: SPI donors funded, WAS coherence, residential split, Child Benefit, release-cut blockers (#1063) #1089's branch.
  • A9, this head on main (d89a3ddfe).
    • Loss 0.00715, with 99.0% of targets within 10% (12 outside) and ESS 4,828. All 7 calibrated-seam gates pass.
    • The control is main's own final UK spine follow-ups: SPI donors funded, WAS coherence, residential split, Child Benefit, release-cut blockers (#1063) #1089 build at 64c460b66: loss 0.00713, 98.4% within 10% (19 outside), ESS 4,973.
    • Against it, the South East cell is at +11.5% (+30.3% on main, where it is deferred) and obr.council_tax at −9.9% (−11.1%).
    • obr.vat is at +24.4%, against +23.1% on main.
    • With income-related ESA zeroed on SPI rows, design obr.esa falls from £5.73bn to £4.80bn. Calibration restores it to +0.4%, and dwp.esa_income_claimants to −0.1%. dwp.esa_claimants lands at +11.9% (+11.0% on main).
  • Spine receipts (design weights):
    • Scottish PIP daily-living reporters per 1,000 rise from 13.2 to 46.0 (C5).
    • FRS-channel dividends fall from £40.5bn to £5.1bn, the respondents' own reports (C1).
    • SPI dependants aged 16 to 19 with earnings fall from 94% to none, and dependant band carriers from 40 to 0 (C2).
    • SPI rows with legacy or out-of-work reports, 435 carrying £2.50bn, fall to none (C6). The 258 SPI income-related ESA reports, £1.45bn, go too (R1).
    • SPI units whose investment income implies more than £16,000 while they hold no more fall from 1.86m to 0.18m (C7).
    • Rent-receiving households holding other property rise from 11.6% to 88.3% (C8b).
    • The mean Scottish water charge goes from £404 to £474 (C3b, then R2).
  • Release-cut input mass. I ran the certifier's gate functions on A9's calibrated weights; they reproduce main's certified verdict exactly on main's build. They fail only on the three stale exclusions, and with a0c678c21's register the gate passes.

Verification

Notes

  • The etb_services stage notes were a plain YAML scalar containing #890, which YAML reads as a comment, so everything after it was dropped at load. They are now quoted. lcfs_consumption has the same defect and is left for a separate fix.
  • With the pinned feed set, test_uk_target_references.py::test_committed_surfaces_regenerate_from_pinned_feed[national] fails on main as it does here. The committed target_references.json (bf7549…) no longer regenerates (e8f829…). Neither the file nor its generator has changed since UK spine follow-ups: SPI donors funded, WAS coherence, residential split, Child Benefit, release-cut blockers (#1063) #1089's 2fe7d8bbb, and CI skips the test because it has no licensed feed. It belongs in a separate fix.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Automated review pass (Claude Code, high effort) — round 1 at f36af232

Verdict: nothing blocking in the ports themselves. Each of the fourteen commits does what its message says, and the targeted engine-free suites pass. Before this can be reviewed as final it needs a rebase onto #1089's current head, and the VAT gate is close enough to its bound to need a decision before merge.

What I checked. I diffed against the merge base with #1089 (45014f48) and read every runtime change. I ran these suites locally at this head: test_uk_benefit_target_scope, test_uk_spi_spine, test_uk_uc_capital_coherence, test_uk_was_wealth, test_uk_etb_vat, test_uk_etb_services, test_uk_local_targets, test_uk_frs_spine, test_uk_spi_band_donors, test_uk_terminal_gates, test_uk_spi_benefit_coherence, test_uk_frs_council_tax and test_uk_stage_health. All passed. The one failure, test_uk_uc_family (81 against 83), is the inherited pin that #1089's 427bec73 fixes; see item 1.


1. Process: rebase onto #1089's 427bec73. This branch sits on 45014f48, five #1089 commits behind its current head (2af5ad3e, 64c460b6, 1e58ecb3, fd13f71e, 427bec73). Those include #1089's round-2 fixes, so the inherited test_uk_uc_family failure in your verification list goes away after the rebase. git merge-tree shows two content conflicts, both in generated files:

  • release_input_coverage_manifest.json;
  • microcosm-graph/tests/fixtures/parity/uk_spine/uk_spine.json.

Regenerate both, and re-check the gate-battery digest pins in contract.py, which auto-merged. The two test_uk_target_references regeneration failures you list as inherited should be re-checked on the new base too.

2. Should-fix (decision): obr.vat passes by 0.002 points. At the rebased head it calibrates at +24.998% against the 25% bound. It crossed the bound in A3 and A5, and uk_weight_ratio crossed its maximum in A5 (1,161.8 against 1,151.3). The receipts are right that the overshoot predates this PR, since design VAT is 25–28% over the OBR line in every arm. But merging leaves a release-blocking gate on a knife edge, so the next unrelated spine change will block a release on VAT. Either look at the ETB × LCFS VAT imputation before merge, or have María sign a reviewed deferral for obr.vat now with an expiry, so the failure mode is decided in advance.

3. Question: why does income-related ESA keep its SPI draw? spi_benefit_coherence zeroes income-based JSA, Income Support and both tax credits on SPI rows, because each "needs an existing legacy claim, an out-of-work or incapacity test, or a claim the copied household's new incomes cannot carry" (spi_benefit_coherence.py:97-108). Income-related ESA meets the same test: it's a closed legacy benefit, income-tested, with an incapacity test. Yet the docstring keeps it with the disability benefits (:39). Contributory ESA keeping its draw is defensible. For esa_income_reported on rows whose new SPI incomes are taxpayer-level, I'd either zero it alongside JSA-IB and IS, or record why not, given that the commit message measured ESA as 7.5% out of scope on the national build.

4. Nit: the Scottish water reduction is the maximum, applied to every recipient. SCOTTISH_WATER_CHARGES_REDUCTION_RECIPIENT_SHARE = 0.65 (frs_spine.py:959) gives every council tax reduction recipient the Water Charges Reduction Scheme's full 35%. The scheme's reduction is "up to" 35% and tapers with the household's CTR entitlement, so partial recipients pay more than 65%.

  • For netting CTANNUAL, the empirical fit (ratio 1.00 overall, 0.98–1.01 by band) justifies 0.65 as the share DWP's derivation embeds.
  • For the charge a household pays, it understates partial recipients' bills.

Where CTREBAMT and the gross bill are both known, scaling the 35% by the CTR share would be closer. Otherwise, say in the docstring that 0.65 is the full-reduction case.

5. Nit: the rebased-head arm is 9c739408, not this head. The body's A8 measures 9c739408. The two commits after it, the register retirement and the receipts, don't touch the spine, so that's fine, but say so in the receipts so the arm-to-head mapping is explicit.


Checked and correct:

  • Benefit scope (f5fffc55).
    • Every OBR table 4.9 DWP row now carries a GB measurement filter with a matching household condition: PC, ESA, JSA, UC and its cap rows, HB and SMP, plus the DWP caseload facts.
    • AA, CA, PIP and WFP are scoped to England and Wales, and Child Benefit stays UK.
    • The calendar-2025 window (Jan 2025 to Dec 2025) falls after every executive-competence date cited, so a static E&W scope is right for every bound period. No row needs year-varying scope.
    • test_uk_benefit_target_scope refuses any active GB-pinned fact without a GB-or-narrower condition.
  • obr.pip (7acd6f7a) now binds pip + dla over E&W, which matches the OBR line "Disability living allowance and personal independence payments". It removes the solve's PIP inflation from £18.9bn to £36.1bn.
  • ADP and CDP codes (7c44e7f4). 117/118 go into the PIP columns and 121/122 into DLA. Because the PIP, AA and CA rows are now E&W-only, the Scottish payments can't feed a line that excludes them. I didn't open the FRS 2024-25 code frame to confirm the code numbers.
  • Dividends (d90450ae). FRS rows keep their reported dividends, and the stage-parameter assertion refuses a base-channel redraw if one comes back.
  • SPI recipients (6444dc8f) are SPI-channel people aged 16 and over with is_uc_claimant.
    • The forests are still queried over the age domain and the dependants' rows discarded, so every recipient keeps its draw.
    • Band-donor carriers follow the same mask.
    • The rule is the same one policyengine-uk #1983 uses for capital.
  • Council tax (22bfc7e8, 856de4ab). council_tax is the bill before reduction, with CTREBAMT added back. obr.council_tax still binds council_tax_less_benefit, the net receipts concept, so the gross input doesn't inflate the national target.
  • SPI capital (d55f317d) is redrawn for every SPI unit, in cells where reporter status never coarsens.
  • WAS and ETB predictors (606f1379, 9b924307) now measure donor and recipient alike: the annual council tax, NumChildR8, and family-role counts. Buy-to-let (f07c779a) moves out of the remainder without breaking property_wealth's identities.
  • Constituency UC child bands (9c739408) share one helper with the national rows, and the national measurement is byte-identical.
  • Retiring the SE deferral (a4fb0e30) is right, because uk_target_fit fails it as stale at +11.7%.

juaristi22 and others added 17 commits October 4, 2026 18:54
…1095)

OBR table 4.9 reports these lines inside "DWP social security", whose figures
cover Great Britain and residents overseas (DWP benefit expenditure and
caseload tables, Spring 2026, Notes 3), and lists Northern Ireland social
security on rows of its own, but every row bound the UK column. DWP reports
Attendance Allowance, Carer's Allowance, DLA/PIP and Winter Fuel Payment for
England and Wales only since executive competence moved to the Scottish
Government (CA 3 September 2018; DLA, PIP and AA 1 April 2020; WFP 1 April
2024), and the FRS records Scotland's Pension Age Disability Payment and Carer
Support Payment under the AA and CA codes. On the 30 September national build
the out-of-scope share of each fitted UK column was AA 10.8%, CA 12.3%,
PIP 8.9%, ESA 7.5%, PC 4.7% and UC 2.3% (uk-data#490's follow-up note).

Great Britain: obr.pension_credit, obr.esa, obr.jobseekers_allowance,
obr.universal_credit and its two cap rows, obr.housing_benefit,
obr.statutory_maternity_pay, and the Great Britain caseload facts
dwp.esa_claimants, dwp.esa_contrib_claimants, dwp.esa_income_claimants,
dwp.jsa_claimants and dwp.benefit_cap.capped_households. England and Wales:
obr.attendance_allowance, obr.carers_allowance, obr.pip and
obr.winter_fuel_allowance. Each row carries the scope as a uk.geography.country
measurement filter and a household condition on the binding, the form the
Pension Credit, State Pension and pension-age Housing Benefit rows use. Child
Benefit stays UK (an HMRC line covering Northern Ireland).

test_uk_benefit_target_scope pins the scopes and refuses any active row whose
fact is pinned to Great Britain without a Great Britain or narrower condition.
The target references regenerate byte-identical from the pinned feed 825406f
(test_committed_surfaces_regenerate_from_pinned_feed, national and local, run
with CHRONICLE_UK_FACTS); the Chronicle facts still carry K02000001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OBR table 4.9's line is "Disability living allowance and personal independence
payments", but obr.pip bound pip alone, carried as-is for parity with the
incumbent and left for microcosm#622, which closed without a ruling. On the
30 September national build the solve lifted PIP from £18.9bn to £36.1bn to
meet a line that also holds DLA, with the stored DLA (£6.6bn) on top. The row
now binds pip + dla over English and Welsh households (María's ruling of
2026-10-02), the scope DWP reports both benefits on since 1 April 2020.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ine (#1095)

The FRS 2024-25 BENEFITS table records Adult Disability Payment under codes
117 (daily living) and 118 (mobility) and Child Disability Payment under 121
(care) and 122 (mobility), which the spine read nowhere (uk-data#500): on the
raw FRS 215k adults report ADP daily living (£0.98bn a year), 147k ADP
mobility, 51k CDP care and 27k CDP mobility, against 80k Scottish PIP
daily-living reporters, and the 30 September national build carried 24.6 PIP
daily-living reporters per 1,000 Scottish residents against 66.0 in England.
ADP mirrors PIP's components and rates and CDP mirrors DLA's, so each joins the
reported column of the benefit it replaces; the disability categories and flags
follow from the amounts as for PIP and DLA. BENEFIT_CODES now maps each name to
a tuple of codes (the bereavement-support precedent); the recovery codes 69 and
70 stay unmapped. The OBR PIP, AA and Carer's Allowance rows bind English and
Welsh households (previous commits), so the Scottish payments do not enter a
line that excludes them.

Regenerated: the frs_spine scope text in sources.yaml and source_stages.json,
the release input coverage manifest (17 source-manifest digests) and the H2
spine fixture (the frs_spine stage contract). Targeted suites 235 passed; the
H2 parity test passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SPI income stage redrew dividend_income for every FRS-channel adult
from the SPI-trained forest, overwriting what respondents reported. The
redraw operation is gone: FRS rows keep their reported incomes, and the
stage-parameter assertion refuses a base-channel redraw if one comes back
(uk-data#498).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SPI income stage's recipient guard was age-only, so dependent
children aged 16 to 19 in the SPI channel took taxpayer draws. Recipients
are now SPI-channel people aged 16 and over with is_uc_claimant set; one
mask drives both forests' assignments, the State Pension guards, the HMRC
auxiliaries and the disability and carer refreshes, so a dependant keeps
its FRS twin's values. Both forests still query the age domain and discard
the dependants' rows, so every other recipient keeps its draw. Band donor
carriers are claimants and partners too (uk-data#504).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FRS 2024-25 CTANNUAL is the bill after the reported reduction, while the
engine reads council_tax as the liability before it. Households reporting
a reduction get CTREBAMT added back, recipients without an amount take the
larger of their bill and the non-recipient cell mean, and Scottish water
and sewerage charges are netted at the gross cells (in full, or 65% for
recipients) as DWP's derivation embeds them (uk-data#496/#499). The WAS
recipient predictor becomes the bill paid, council_tax less the rebate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spine read each household's discount factor as CWATAMTD/CWATAMT1,
but CWATAMTD sits below the gross charge even without a discount, and
council tax reduction recipients got no water charges reduction. The
charge is now the gross CWATAMT1 + CSEWAMT1 less the CTDISC/CT25D50D
status discount, at 65% for reduction recipients; a household with no
gross water cell pays its recorded CWATAMTD (uk-data#499).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New spine stage spi_benefit_coherence, after uc_reporter_redraw and
before uc_capital_coherence. On SPI-channel rows it zeroes the JSA,
Income Support, tax credit, SDA and Sure Start Maternity Grant reports,
restores IIDB, AFCS and bereavement support from the FRS twin, re-derives
the disability flags and receives_benefits_in_own_right, and redraws
would_claim_uc as frs_take_up draws it for FRS units. FRS rows are never
modified. The release-blocking gate uk_stage_spi_benefit_coherence checks
the structural zeros (uk-data#514).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uc_capital_coherence redrew TOTCAPB4 only for SPI Universal Credit
reporters, so every other SPI unit kept its FRS parent's capital against
SPI-drawn incomes. Every SPI unit is now redrawn from base units with an
available answer, in cells of reporter status by financial investment
income band by couple by children band, widening in a declared order
below 20 donors; reporter status never coarsens (uk-data#495).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The donor's council tax was one CTAmtR8 instalment and its children
NumCh18R8, which overlaps NumAdultR8; the recipient's counts were the
engine's age-18 ones. The donor now reads DVCTaxAmtAnnualR8 and
NumChildR8, the recipient counts its FRS claimants and partners and other
members, and a rental_income predictor pairs the donor's net rent with
the household's property income (uk-data#486/#495).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The WAS stage drew other_residential_property_value from DVHseValR8_sum
alone and left DVBltValR8_sum, buy-to-let worth GBP 573bn at WAS
weights, in the undrawn other-property remainder. The column is now
DVHseValR8_sum + DVBltValR8_sum; the remainder shrinks by the same
amount and property_wealth's identities are unchanged (uk-data#501).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ETB donor's adults and childs count adults and dependent children,
while the recipients summed the engine's age-18 is_adult and is_child.
The recipient counts are now the household's FRS claimants and partners
and its other members (is_uc_claimant). The etb_services notes are quoted
so an unquoted ' #890' no longer truncates them at load (uk-data#486).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The constituency UC child bands read the engine's age-18 num_children,
while the national dwp.uc.households* rows bind DWP's family-type
composition: never a claimant or partner, any UC child or qualifying
young person, or any other member under 20. One helper,
uc_family_child_member, now defines it for both; the national
measurement is byte-identical (uk-data#486).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s make stale (#1095)

The South East Income Tax row for total income GBP 12,570 to 15,000 was
deferred on the #1063 stack because the solver pulled it to +31 %. With
FRS respondents keeping their reported dividends and SPI draws going to
claimants and partners only, it fits at +8.0 % to +12.1 % across the
measurement arms and at +11.7 % on the rebased head, back inside the
25 % bound, so the terminal uk_target_fit gate fails the deferral as
stale. The entry is retired, as in 8e77a90; the register is empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The measurement arms A0-A8 (fit by arm, spine receipts by commit), the
touched-surface verification, the dropped C4c row and the signature
drafts the ports owe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 on #1100, item 3. policyengine-uk pays
esa_income_reported as the award behind a capital test alone
(esa_income), so an SPI row keeping its stage-2 draw was paid its twin's
award whatever its new incomes. On the A8 spine SPI rows held 0.15m of
the 0.53m income-related ESA reporters at design weights (29%, GBP
1.45bn), and 56% of their benefit units had incomes above GBP 12,570.
It is a closed legacy award, income-tested and incapacity-tested, so
spi_benefit_coherence now zeroes it with income-based JSA and Income
Support, as uk-data#514 does.

Contributory ESA keeps its draw, and the stage says why: it is open to
new claims and not income-tested, its incapacity test follows the
person, its SPI draw sits at half the FRS rate per working-age adult,
and 9.9% of its SPI reporters by weight earn above the permitted-work
limit (41% on the uk-data build where #514 zeroes it). The zeroing
reason drops "or incapacity test", which contributory ESA would
otherwise meet; Severe Disablement Allowance stays zeroed as a closed
legacy claim. The gate's zeroed_columns and the stage's written cells
gain the column; digests, coverage manifest and H2 fixture regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1095)

Review round 1 on #1100, item 4. The Water Charges Reduction Scheme for
2021-27 reduces a council tax reduction recipient's charges by
R = 35 x (A/B) - D points of the gross charge, unless negative, on top
of the status discount D, where A is the reduction and B the council
tax before it (Scottish Government, "Water services - charging
principles: 2021 to 2027", Annex A). The combined reduction is the
larger of D and 35% x A/B. The charge paid stacked the 35% on the
discount instead, so a single person on full reduction paid 48.75% of
the gross charge rather than 65%; 207 of the 245 Scottish recipients on
the raw tab carry the 25% discount.

Every recipient takes the full-reduction share, A/B = 1. 86% of
Scotland's recipients receive full reduction (Council Tax Reduction in
Scotland 2024-25, March 2025), while none of the 234 Scottish recipients
with a recorded CTREBAMT on the FRS 2024-25 tab pays nothing after it
(19% of English recipients do) and their amounts have a median of GBP 6
a week against the published GBP 16.55 average, so the recorded amount
cannot carry the share. A recipient now pays the gross charges x
(1 - max(D, 35%)); the netting of CTANNUAL in frs_council_tax keeps
0.65. On the raw tab recipients pay 0.649 of the gross charges instead
of 0.511.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 changed the base branch from uk-spine-followups-1063 to main October 4, 2026 18:29
…ned maximum (#1095)

The ported-values test pinned the Scottish fixture household, a council
tax reduction recipient with the 25% status discount, at the stacked
9 x 0.75 x 0.65 the previous commit corrects; it pays 9 x 0.65.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 marked this pull request as ready for review October 4, 2026 19:13
juaristi22 and others added 2 commits October 4, 2026 20:41
…ale (#1095)

#1089's certifier signed three input-mass exclusions on 2026-10-04:
income-based JSA (+1,058% against the enhanced-FRS reference) and
Working Tax Credit (+747%), both carried by the SPI channel's stage-2
legacy-benefit leaves, and the access fund (+535%), one FRS household
at the calibration's weight bound. spi_benefit_coherence zeroes the two
leaves on SPI rows, and on arm A9 (the review head d89a3dd on main)
the calibrated columns sit at GBP 97.7m (+316%) and GBP 149.6m (+103%)
and the access fund at GBP 258.6m (+62%), its household off the bound:
all inside the 452% fence, so the release-cut input_mass_parity gate
fails the three entries as stale. They are retired, as in 7efa9d0;
dfe_education_spending stays. The evidence pin returns to d28a1fd6...,
the register's digest before the three were signed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hashes on main; the review round's three changes (income-related ESA on
SPI rows, the Scottish water scheme's combined maximum, the stale
input-mass exclusions); arm A9 on the review head against main's own
final build; the verification on the new base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Round 1 addressed at 479d923ce

#1089 merged, so this PR now targets main and gets CI. The branch is rebased onto main (48fa0616), and items 3 and 4 change the spine, so a new arm, A9, measures the rebased head with both changes. The receipts note and the description are updated.

1. Rebase.

  • All fifteen commits replay onto main. The only conflicts were in the two generated files you named.
  • I regenerated the coverage manifest and the H2 fixture after every replayed commit, not only the conflicted ones, so each commit's generated files match its code.
  • Gate digests: after the rebase repin_digests --check reports 0 of 12 moved, so the auto-merged pins in contract.py hold. Item 3's commit moves five, for the gate's new zeroed column, and re-pins them in the same commit.
  • The inherited failures on the new base:

2. obr.vat. A9 measures this head's code and calibrates it at +24.4%, 0.6 points inside the bound. Main's own final #1089 build, without the ports, is at +23.1%.

3. Income-related ESA. Agreed: it is now zeroed alongside income-based JSA and Income Support.

  • On A8's spine at design weights, SPI rows held 0.15m of the 0.53m income-related ESA reporters (29%) and £1.45bn of reports.
  • In 56% of those benefit units the unit's income was over £12,570, and in 43% over £20,000.
  • policyengine-uk 2.100.0 pays esa_income_reported as the award, screened only by the capital test (esa_income), so those units were paid their twin's award whatever their new incomes. uk-data#514 zeroes it too.
  • Contributory ESA keeps its draw, and the stage now says why:
    • it is not income-tested, and the incapacity it needs follows the person, whose health and disability reports the SPI copy keeps;
    • on A8 the SPI draw sits at half the FRS rate per working-age adult;
    • 9.9% of its SPI reporters by weight earn above the permitted-work limit. uk-data's build, where Add JCT tax-expenditure targets for ALD components + CDCC #514 zeroes it, had 41%.
  • The zeroing reason drops "or incapacity test", which would otherwise cover contributory ESA. Severe Disablement Allowance stays zeroed as a closed legacy claim.
  • On A9, design obr.esa falls from £5.73bn to £4.80bn and design dwp.esa_income_claimants from 404k to 313k. Calibration restores both (+0.4% and −0.1%), and dwp.esa_claimants lands at +11.9% (+11.0% on main).

4. Scottish water. You found the smaller of two errors.

  • The Scottish Government's charging principles for 2021–27 (Annex A) set the reduction at R = 35 × (A/B) − D points of the gross charge, unless R is negative. Here A is the council tax reduction, B the council tax before it after discounts, and D the status discount. The reduction comes on top of D, so the combined reduction is max(D, 35% × A/B).
  • So besides the taper, C3b stacked the 35% on the status discount. A single person on full reduction paid 48.75% of the gross charge instead of 65%. That case is 207 of the 245 Scottish recipients on the raw tab.
  • I couldn't scale by the recorded reduction share:
    • the Scottish CTR statistics put 86% of the 458,120 recipients on full reduction in March 2025, with an average award of £16.55 a week;
    • none of the 234 Scottish recipients with a recorded CTREBAMT on the FRS 2024-25 tab pays nothing after it, against 19% of English recipients;
    • their recorded amounts have a median of £6 a week.
  • Every recipient therefore takes the full-reduction case. A recipient pays gross × (1 − max(D, 35%)), and anyone else gross × (1 − D). The docstring says this is the full-reduction case and why. On the raw tab, recipients pay 0.649 of the gross charge instead of 0.511.
  • The netting of CTANNUAL keeps 0.65, as you say.
  • The amended scottish-water-sewerage-successor-level text in the description changes to match.

5. Arm-to-head mapping. A9 measures d89a3ddfe, which is this head's code. The commits after it change only a test pin, the input-mass register and the receipts, none of them spine or calibration code, and the receipts now say so.

Also changed at this head: three input-mass exclusions retired. C6 zeroes income-based JSA and Working Tax Credit on SPI rows, which makes the input-mass exclusions #1089 signed for them stale.

  • I ran the certifier's own input_mass_parity functions on A9's calibrated weights. On main's 64c460b66 build they reproduce the certified verdict exactly.
  • On A9, JSA lands at £97.7m (+316% against the reference) and WTC at £149.6m (+103%). The access fund is at £258.6m (+62%), because its household is no longer at the weight bound.
  • All three are inside the 452% fence, and the gate fails stale entries, so a0c678c21 retires them, as 7efa9d00b did the SE deferral. With that register the gate passes; dfe_education_spending stays.

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review pass (Claude Code, high effort) — round 2 at 479d923c

Verdict: approve. All five round-1 items are closed, CI is 10/10 green, and the branch merges cleanly with current main (e34712cc; main's commits since the rebase base 48fa0616 are US-only). One should-fix below, a tracking issue, which doesn't need to hold the merge.

Item Status Evidence
1. Rebase Closed. The merge base is 48fa0616, after #1089. The coverage manifest and H2 fixture are regenerated per commit. The digest re-pin in eaf7e80c (five pins, for the gate's new zeroed column) is in the same commit as the change. test_contract passes locally.
2. obr.vat Closed by measurement; the margin is thin. A9 (d89a3ddf, the PR's code) calibrates obr.vat at +24.4%, against main's own final #1089 build at +23.1% (experiments/1095-uk-data-ports-receipts.md:143). Earlier arms reached +25.2% (A0 line 116) and +24.998%. So it now passes with 0.6 points to spare. The cause is the engine's microdata_vat_coverage, filed as policyengine-uk#1996, and María's ruling keeps it out of the data PRs. That's fine. If the next national build crosses 25%, the signed deferral should be ready to go.
3. Income-related ESA Closed, code changed. eaf7e80c adds esa_income_reported to SPI_ZEROED_REPORT_COLUMNS and to the gate's zeroed_columns (gates.json:473), with a test. The reason drops "or incapacity test", so contributory ESA's kept draw is now consistent with the stated rule, and the docstring gives the evidence (9.9% above the permitted-work limit, against 41% on uk-data's build). The A9 design and calibrated ESA figures are receipted.
4. Scottish water reduction Closed, and a second error fixed. d89a3ddf now applies max(D, 35%) instead of stacking the 35% on the status discount. A single person on full reduction paid 48.75% before; now 65%. 894a33c1 re-pins the fixture to 9 × 0.65. Applying the full-reduction case to every recipient is argued in the docstring from the published 86% full-reduction share and the unusable CTREBAMT amounts, which is a fair reason not to taper. I didn't open the Scottish Government's Annex A myself, so the R = 35 × A/B − D formula is checked against her citation, not against the source.
5. Arm-to-head mapping Closed. The receipts (lines 95-97) say A9 builds d89a3ddf, and that the three commits after it touch no spine or calibration code. I checked: they are a test pin, the input-mass register and the receipts note.

Should-fix: track the access-fund period bug now that its exclusion is retired. a0c678c2 retires the access_fund input-mass exclusion, along with income-based JSA and WTC. Retiring it is correct, since the gate fails stale entries and A9 puts the column at +62%, inside the 452% fence. But that entry was the only active record of a known data defect. Sernum 14443's £4,500 is coded as monthly and annualised to £54,037, almost certainly an annual award under the wrong period code (receipts 1063-uk-spine-followups.md:304-316). The record is still in the data; it simply sits below the fence now that its household is off the weight bound. Nothing will make anyone fix it. Please open an issue for the period rule, or add it as an item on #1095, so it isn't lost.

Note, outside this PR: test_committed_surfaces_regenerate_from_pinned_feed[national] fails on main when the licensed feed is set, because the committed target_references.json no longer regenerates. CI skips it for want of the feed, so main can't catch this. Agreed that it belongs in its own fix on main; an issue would keep it visible.

Locally at 479d923c: test_uk_spi_benefit_coherence, test_uk_frs_spine, test_uk_weighted_integrity, test_uk_uc_family and the shared test_contract all pass.

juaristi22 added a commit that referenced this pull request Oct 7, 2026
Vahid's should-fix on #1100: retiring the access_fund input-mass
exclusion left the period-code defect it recorded with no active
record. The FRS weeklyises ACCSSAMT from the reported amount and its
period code ACCSSPD, and an access-fund award is paid per academic year
or term. On the 2024-25 tab the 16 awards' calendar-month amounts run up
to the size of a whole annual award, so an award read as monthly can
annualise to more than ten times every annual-coded award; #1089 had to
sign an exclusion when calibration took such a household to the weight
bound.

access_fund_annual reads ACCSSPD: a calendar-month award (code 5) that
annualises to more than twice the largest annual-coded award (code 52)
on the tab is read as its per-period amount, ACCSSAMT x 52/12. On the
2024-25 tab that repairs fewer than 10 awards and leaves large but
plausible monthly amounts as reported, as it does the awards with no
period code. A tab without the ACCSSPD column refuses. The spec names
the rule, the test and H2 fixtures carry ACCSSPD, and the coverage
manifest and H2 fixture are regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 7, 2026
…ar-2025 window (#1095)

DWP's benefit expenditure and caseload tables give each fiscal year its own
measure (expenditure_2024, expenditure_2025), so calendar_year_window read the
two years as two one-year series and refused the window; #1100 dropped the row
for that reason. A window selector can now declare
source_measure_id_by_opening_year, mapping each overlapping year to its
measure: a fact matches only its own year's measure, and the declared measures
form the window's one series. The map's form is checked when the reference is
built; its years are checked when the window resolves, so a compile that
restamps the reference to another period (the 2023 parity receipt) defers it
like any reference without facts.

dwp.hb.amount_pension_age binds the line "Housing Benefit over Pension Credit
qualifying age" at 3/12 of the FY2024-25 outturn (6,851.0m) plus 9/12 of the
FY2025-26 forecast (7,114.7m), 7,048.8m, on housing_benefit over benefit units
whose eldest adult is 66 or over in Great Britain households. It has its own
dwp_benefit_expenditure family, which allows source projections. The national
references, the membership report and the 2025 parity receipt gain the row and
nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 7, 2026
Vahid's should-fix on #1100: retiring the access_fund input-mass
exclusion left the period-code defect it recorded with no active
record. The FRS weeklyises ACCSSAMT from the reported amount and its
period code ACCSSPD, and an access-fund award is paid per academic year
or term. On the 2024-25 tab the 16 awards' calendar-month amounts run up
to the size of a whole annual award, so an award read as monthly can
annualise to more than ten times every annual-coded award; #1089 had to
sign an exclusion when calibration took such a household to the weight
bound.

access_fund_annual reads ACCSSPD: a calendar-month award (code 5) that
annualises to more than twice the largest annual-coded award (code 52)
on the tab is read as its per-period amount, ACCSSAMT x 52/12. On the
2024-25 tab that repairs fewer than 10 awards and leaves large but
plausible monthly amounts as reported, as it does the awards with no
period code. A tab without the ACCSSPD column refuses. The spec names
the rule, the test and H2 fixtures carry ACCSSPD, and the coverage
manifest and H2 fixture are regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 7, 2026
…ar-2025 window (#1095)

DWP's benefit expenditure and caseload tables give each fiscal year its own
measure (expenditure_2024, expenditure_2025), so calendar_year_window read the
two years as two one-year series and refused the window; #1100 dropped the row
for that reason. A window selector can now declare
source_measure_id_by_opening_year, mapping each overlapping year to its
measure: a fact matches only its own year's measure, and the declared measures
form the window's one series. The map's form is checked when the reference is
built; its years are checked when the window resolves, so a compile that
restamps the reference to another period (the 2023 parity receipt) defers it
like any reference without facts.

dwp.hb.amount_pension_age binds the line "Housing Benefit over Pension Credit
qualifying age" at 3/12 of the FY2024-25 outturn (6,851.0m) plus 9/12 of the
FY2025-26 forecast (7,114.7m), 7,048.8m, on housing_benefit over benefit units
whose eldest adult is 66 or over in Great Britain households. It has its own
dwp_benefit_expenditure family, which allows source projections. The national
references, the membership report and the 2025 parity receipt gain the row and
nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants