Take SPI Scottish taxpayer status from SCOT_TXP and keep unknown regions simulable

[MaxGhenis]

Fixes #505.

What and why

create_spi labels SPI records whose GORCODE is not a UK region as UNKNOWN. HMRC's codebook (SN 9422, Annex A) gives these codes as 13 "Address abroad" and 14 "Address unknown or not available", and composite records carry -1. On the 2022-23 public use tape that is 18,859 of 836,850 records: 16,194 abroad, 353 unknown and 2,312 composite. Two things were wrong:

1. The output could not be simulated. Microsimulation(dataset=create_spi(tab, 2022)) failed while the simulation was being built, with ParameterNotFoundError: ... private_rental_prices.UNKNOWN. policyengine-uk 2.104.5 (released 2026-10-02) is the first release with Uprate rent for unknown-region households by the UK-wide index policyengine-uk#1985, which uprates rent for Region.UNKNOWN by the UK-wide index. With it, UNKNOWN simulates as built. UNKNOWN stays the right label: none of these records has a UK region, and relabelling them would distort regional totals.
2. Scottish taxpayer status ignored HMRC's own flag. The tape has SCOT_TXP ("Indicates whether the record is treated as Scottish, for tax purposes"), and its liabilities (TOTTAX_DEVO_TXP, TAX_ST_*_DEVO_TXP) are "based on SCOT_TXP or WELSH_TXP". The dataset didn't carry the flag, so policyengine-uk derived pays_scottish_income_tax from the region. The two disagree for 2,000 records. HMRC notes that the region reflects residence at the end of the tax year, which can differ from Scottish taxpayer status. Records with an address abroad and composite records can also be Scottish taxpayers. HMRC built composites from Scottish, Welsh or other taxpayers only, never a mix.

Changes:

• datasets/spi.py:
  • Set pays_scottish_income_tax from SCOT_TXP. HMRC documents the flag as . or 1, but the 2022-23 tape holds 0 or 1; both parse. WELSH_TXP is not used, because policyengine-uk has no separate Welsh rates of income tax (no Welsh parameters under gov.hmrc.income_tax in 2.93.0 or 2.121.0).
  • The REGION_MAP comment now gives the codes correctly.
  • Add model_simulates_unknown_region(). It checks that the imported policyengine-uk can build a simulation of one SPI-shaped household labelled UNKNOWN, once per process (a few seconds). A failure means "cannot simulate" only if the same household labelled SOUTH_EAST builds, which is exactly when relabelling helps. Otherwise the SOUTH_EAST household's error is raised.
  • The probe establishes that the representative household can be built, not that every SPI dataset will simulate. The dataset-shaped income-tax tests check the real behaviour on the supported releases and years.
  • The household mirrors the fixture that policyengine-uk's own test_rent_uprating.py simulates with UNKNOWN, so the model keeps supporting that shape.
  • The probe checks the imported code rather than the installed version, because the two can differ: make data-local puts a policyengine-uk checkout on PYTHONPATH ahead of the installed release.
• utils/incomes_projection.py:
  • load_spi_dataset relabels UNKNOWN as SOUTH_EAST only when the imported model cannot simulate UNKNOWN. This replaces create_spi output cannot be simulated: unknown regions crash policyengine-uk, and Scottish taxpayer status ignores SCOT_TXP #505's to-do ("remove the relabelling once the lock reaches #1985"): the relabel switches itself off at the relock, with no follow-up PR.
  • ensure_spi_dataset also rebuilds a cached spi_2022_23.h5 that predates the flag. Without that, a stale cache would silently fall back to deriving the flag from the region.

Horizon. policyengine-uk carries a single-year dataset's inputs forward to 2030 (extend_single_year_dataset, end_year=2030, in 2.93.0 and 2.121.0). From 2031 it computes pays_scottish_income_tax from the region again, as it does for any input variable that has a formula. A two-person synthetic check on both releases confirmed this: the input held in 2029 and 2030, and the region rule replaced it in 2031 and 2032. The tests cover 2022, 2026 and 2030.

Evidence

Everything here is aggregate. No count under 10 records is shown. Each count is at least 10, and so is its complement within its group. A count of zero counts as under 10, so no figure here implies one by differencing.

• HMRC tape documentation for 2022-23 (SN 9422) and 2020-21 (SN 9121) covers:

  • GORCODE codes 1-14;
  • the SCOT_TXP and WELSH_TXP definitions;
  • how composite records are built;
  • the note that Scottish taxpayer status can differ from residence.

  SCOT_TXP is on both tapes.

• Tape counts (2022-23):

  • GORCODE 11 but not a Scottish taxpayer: 1,394 records (90,000 weighted).
  • Scottish taxpayers outside GORCODE 11: 606 records (33,000 weighted).

• Historical income-tax evidence against HMRC's own TOTTAX_DEVO_TXP, on the full 2022-23 tape, collected before this refresh. Both datasets ran under policyengine-uk 2.104.5, because the pre-batch create_spi output could not be simulated under the then-locked 2.93.0. "main" in this historical table is create_spi at b45c373. Groups overlap. The within-£10 column is a weighted share (by FACT), not a share of records.

Records	Count	Weighted	Weighted share within £10 of HMRC: main → this PR	PolicyEngine tax: main → this PR	HMRC tax
GORCODE 11, not a Scottish taxpayer	1,394	0.090m	65.9% → 98.2%	£0.227bn → £0.215bn	£0.215bn
Scottish taxpayer, GORCODE not 11	606	0.033m	39.1% → 81.3%	£1.479bn → £1.519bn	£1.528bn
Flag disagrees with region (the two rows above)	2,000	0.123m	58.7% → 93.7%	£1.705bn → £1.733bn	£1.743bn
GORCODE 13, 14 or -1	18,859	1.024m	87.1% → 87.5%	£42.635bn → £42.670bn	£42.911bn
Known region, flag agrees with region	816,231	49.581m	96.69% → 96.69%	£201.167bn → £201.167bn	£201.765bn
Flag agrees with region, any region	834,850	50.591m	96.52% → 96.52%	£242.408bn → £242.408bn	£243.238bn
All	836,850	50.714m	96.43% → 96.51%	£244.114bn → £244.142bn	£244.981bn

• Of the 2,000 records where the flag and the region disagree, 1,074 have different income tax under this PR in 2022, and 1,166 in 2026 (same tape and release).
• policyengine-uk releases: #1985 merged as 1c78908e on 2026-10-02. git tag --contains puts it first in 2.104.5, and 2.104.4 does not contain it. PyPI shows 2.104.4 uploaded at 11:12 UTC and 2.104.5 at 11:15 UTC that day. The latest release when these tests ran (2026-10-05) was 2.122.0.

Invariants

1. The region follows GORCODE alone. Codes 1-12 map to HMRC's regions, and every other code becomes UNKNOWN. Every label is a Region member, and Scottish taxpayers with codes 13, 14 or -1 are not relabelled SCOTLAND.
2. pays_scottish_income_tax is true exactly when SCOT_TXP parses to 1, whatever the GORCODE and however "no" is written (., 0 or blank).
3. With equal incomes, income tax depends only on SCOT_TXP. This holds in the data year, after uprating (2026), and in 2030, the last year policyengine-uk carries dataset inputs to.
4. The region label does not move income tax. Building with UNKNOWN or with the legacy SOUTH_EAST gives equal income tax record for record, in 2022, 2026 and 2030.
5. Records whose SCOT_TXP agrees with their region get exactly the income tax of the old region-derived rule, so only records where the two disagree can change. This is tested in 2022, 2026 and 2030.
6. model_simulates_unknown_region():
   • is false exactly when the UNKNOWN household fails and the same household labelled SOUTH_EAST simulates;
   • raises the SOUTH_EAST household's error when both fail;
   • for a final release not installed from a direct URL, whose imported economic_assumptions.py is the installed file and matches its RECORD, is true exactly when that release is 2.104.5 or later.
7. load_spi_dataset changes a region label only when the imported model cannot simulate UNKNOWN. A cached H5 is reused only if it has the release year and the flag.

Tests

All in tests/test_spi_build.py. SCOT_TXP is added to the fake-tape columns there and in tests/test_spi_allowance_deductions.py.

• test_create_spi_region_and_scottish_taxpayer_invariants[0|.|blank] (invariants 1-2): exhaustive over GORCODE ∈ {-1, 1…14, 99} × SCOT_TXP ∈ {no, 1}, for each encoding of "no". It checks against HMRC's codebook mapping written out in the test, not REGION_MAP.
• test_create_spi_scottish_taxpayer_status_survives_h5_round_trip: the flag and UNKNOWN survive save and load.
• test_spi_income_tax_follows_scottish_taxpayer_flag (invariant 3): uses the legacy label, so it also runs under the historical 2.93.0 lock.
• test_spi_income_tax_changes_only_where_scottish_flag_and_region_disagree (invariant 5): compares against the same dataset with the flag column dropped, which is main's rule. It uses UNKNOWN where the model can simulate it, and SOUTH_EAST otherwise.
• test_create_spi_output_with_unknown_region_can_be_simulated (invariant 4): uses the default UNKNOWN label. Where the imported model cannot simulate UNKNOWN, the build must raise ParameterNotFoundError for private_rental_prices.UNKNOWN and nothing else, and the test reports an xfail. Otherwise it runs in full.
• Invariant 6:
  • test_unknown_region_probe_reads_the_simulation[…]: three cases. It simulates; the rent index is missing; or a non-parameter error that relabelling cures. In each case the test checks which households were simulated.
  • test_unknown_region_probe_raises_failures_relabelling_would_not_cure[…]: an unrelated parameter ending in .UNKNOWN, an unrelated parameter, and a schema error, each also hitting the SOUTH_EAST household. Each must be raised as the SOUTH_EAST household's own error, with the UNKNOWN household's distinct error as its context.
  • test_unknown_region_probe_agrees_with_policyengine_uk_release. It skips:
    • pre-release, dev and local versions;
    • direct-URL and editable installs;
    • an imported tree that is not the installed one;
    • an economic_assumptions.py (the module #1985 changed) that does not match the release's RECORD.
• Invariant 7:
  • test_income_projection_loads_local_h5_dataset[False|True]
  • test_income_projection_rebuilds_spi_dataset_without_scottish_flag, which uses real H5 files.

Historical results from before this refresh (both files, TESTING=1, old PR head 4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c; these were not rerun in this refresh session):

policyengine-uk imported	Installed metadata	Result
2.93.0 (historical lock)	2.93.0	32 passed, 1 xfailed
2.104.4 (last release without #1985)	2.104.4	32 passed, 1 xfailed
2.104.5 (first with #1985)	2.104.5	33 passed
2.121.0	2.121.0	33 passed
2.122.0 (latest on 2026-10-05)	2.122.0	33 passed
2.104.5 checkout on PYTHONPATH	2.93.0	32 passed, 1 skipped (the release test, by design)
2.104.4 checkout on PYTHONPATH	2.121.0	31 passed, 1 skipped (the same), 1 xfailed

The last two rows are the version skew the first review found; the old gate read the metadata there and got the answer wrong both ways. The RECORD check was also exercised directly on a 2.121.0 install. It reports a mismatch after a one-line edit to economic_assumptions.py, and a match once the edit is removed.

Historical mutation checks from before this refresh. Each had to fail at least one test; all ran under the then-locked 2.93.0 unless noted. These mutation checks were not rerun in this refresh session.

Mutation	Tests that fail
Scottish status from GORCODE == 11 (main's rule)	6
Probe always true	9
Probe always false (run on 2.104.5)	8
Probe without the known-region control	5
Control household labelled UNKNOWN	5
Probe reads only parameter errors	1
Cache ignores the missing flag	1
Relabel unconditionally	1
Relabel Scottish taxpayers as SCOTLAND	5
Treat any non-zero SCOT_TXP as Scottish	2
Compare SCOT_TXP without parsing it	1
REGION_MAP[8] set to LONDON (survived round 1)	3
Unknown codes labelled with a non-region (round 1's xfail accepted the resulting error)	6
Raise the UNKNOWN household's error instead of the control's (survived round 3)	3

policyengine-uk lock

The 10/8 release batch landed on 10/7 as uk-data 1.58.0. Its main head 4cbedbecca352f52dffe47e849553c2759a00711 pins policyengine-uk 2.122.2 and policyengine-core 3.32.13. This refresh inherits those lock and dependency changes by merging main. A separate one-line fix aligns the lock's local-project version with pyproject.toml at 1.58.0; no dependency version changes.

2.122.2 is newer than the first UNKNOWN-capable release, 2.104.5. The refreshed validation below confirmed that the unknown-region simulation test passes in full, model_simulates_unknown_region() is true, and load_spi_dataset keeps UNKNOWN without the legacy SOUTH_EAST relabel. The compatibility fallback remains for imported models that cannot simulate UNKNOWN.

Impact: pending a real rebuild after the refresh

No dataset was rebuilt in this refresh. The other item owed after the batch remains a hub-scheduled rebuild of storage/spi_2022_23.h5 and the enhanced FRS, compared with the 1.58.0 batch outputs. From reading the code, no change to the enhanced FRS is expected:

• create_datasets.py does not call create_spi, ensure_spi_dataset, load_spi_dataset or model_simulates_unknown_region.
• The income imputation reads the SPI tab directly in datasets/imputations/income.py. It imports AGE_RANGES, REGION_MAP and the release names from spi.py, and none of them changes in this PR. The enhanced-FRS synthetic rows retain the FRS donor's geography.
• The standalone dataset this PR changes is storage/spi_2022_23.h5. Only ensure_spi_dataset (through load_spi_dataset) and spi.py's __main__ build it, and no step of the enhanced FRS build reads it.

The real rebuild should confirm this expectation and quantify the refreshed standalone SPI impact. It remains pending on the hub's schedule, with one build at a time and more than 60 GB free on the host.

Post-batch refresh (10/7)

• New head: 4fbd8be43b7dc550d9f6c2871ddc7270fec38c01, a fast-forward descendant of 4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c, with main 4cbedbecca352f52dffe47e849553c2759a00711 as the merge commit's other parent.
• Resolved the single end-of-file conflict in tests/test_spi_build.py by keeping both the PR's region/flag invariants and unknown-region simulation tests and Supply is_claimant_or_partner from the FRS adult table #524's test_create_spi_marks_every_taxpayer_as_their_benefit_units_claimant. The shared fake-tape helper supplies both SCOT_TXP and main's SEINC_NUM.
• Supply is_claimant_or_partner from the FRS adult table #524's SPI claimant input is retained alongside the independent SCOT_TXP-based Scottish flag. Draw SPI incomes within earnings groups set by FRS employment status #529's income draws and caches retain their earnings-group semantics. Stop SPI-synthetic rows carrying benefit claims nobody observed #514's SPI-synthetic reported-benefit rules remain intact. These downstream imputations preserve the supplied Scottish flag and household geography; none re-derives Scottish status from region. The documented model input horizon remains 2030.
• Every merge-touched file was audited at merge commit a22fb5daffa29d0ff28dc27d70775285202ff3f2: all 70 main-only paths matched main exactly, all three PR-only paths matched the old PR head, and both shared files preserved each branch's changes. No main or PR hunk was dropped; no conflict markers remain. The only follow-up changes uv.lock's local-project version from 1.57.4 to main's released 1.58.0, fixing uv lock --check without changing any dependency pins.
  Ran the following under policyengine-uk 2.122.2 / policyengine-core 3.32.13, with TESTING=1, sequentially in the foreground, one file per pytest process, -p no:cacheprovider and no xdist. All paths are under policyengine_uk_data/tests/.

Test file	Passed	Skipped	Failed	Xfailed
test_spi_build.py	33	0	0	0
test_spi_allowance_deductions.py	1	0	0	0
test_spi_donor_benefit_rules.py	13	0	0	0
test_spi_income_earnings_groups.py	15	1	0	0
test_claimant_or_partner.py	11	2	0	0

Total: 73 passed, 3 skipped, 0 failed, 0 xfailed. The three skips need a built FRS/enhanced FRS. The unknown-region simulation test passed in full, rather than xfailing. A separate synthetic-H5 check used the real capability probe: it returned true, load_spi_dataset kept UNKNOWN without relabelling to SOUTH_EAST, and the Scottish flag was preserved. uv lock --check passed after the one-line metadata fix; the four PR Python files pass ruff format --check.

The initial harness attempt had 15 passes and 18 setup errors because the configured temporary-directory parent was missing. Creating it fixed the harness; the final SPI run above has no errors. No source changes were needed for that rerun.

• The standalone SPI and enhanced-FRS rebuild remains pending for the hub after this refresh; no real dataset builds were run here.

Review

Independent GPT-6.1 Sol reviews.

Round 1 (9fc0e29): REQUEST CHANGES.

1. The version gate could misread the imported model. It is now a capability probe on the imported code.
2. The tests could accept broken behaviour. They now:
   • carry HMRC's mapping in the test;
   • xfail only on the missing private_rental_prices.UNKNOWN index;
   • include an invariant 5 test.
3. The agreement row was narrower than its label. It is relabelled, and an any-region row has been added.
4. The flag input holds only through 2030. This is documented, and the tests run to 2030.
5. The supporting cross-tab's suppression was incomplete. That local output has been deleted.

Round 2 (3c37ef8): REQUEST CHANGES. Round 1 findings 2-4 resolved; 1 and 5 partly.

1. Blocking: the exact full-tape equality statements implied complementary counts of zero. They are removed from this body, which keeps only the aggregate table and counts whose complements are at least 10. The validation script now applies that secondary suppression to its own output. Invariants 4 and 5 are claimed from the synthetic tests only.
2. The probe's error filter matched any .UNKNOWN' in the message. It now uses a known-region control instead of matching text.
   • The reviewer suggested matching error.name, but that cannot work: AttributeError.__init__ resets name to None, so every ParameterNotFoundError carries name=None (checked on Python 3.13).
   • The control also covers the reviewer's schema concern. The household is the shape policyengine-uk itself tests with UNKNOWN, and a failure that the SOUTH_EAST household shares is raised.
3. The differential test treated any matching path as a final release. It now skips pre-release, dev, local, direct-URL and edited installs, as listed under Tests.
4. Nit: the within-£10 column is labelled as weighted, and "latest" is dated.

Round 3 (c65d381): APPROVE WITH CHANGES. Round 2 findings 1, 2 and 4 resolved; 3 partly. All seven result rows were independently reproduced. The disclosure audit passed (93 assertions). Three nits remained, all fixed at 4f86cee:

1. The release-provenance wording claimed more than the checks show. The test and this body now describe the checks exactly: final release, not a direct-URL install, and economic_assumptions.py installed and matching RECORD.
2. The probe's guarantee is now scoped to building the representative household.
3. The shared-failure test now gives the two households distinct errors and requires the control's. The reviewer's surviving mutant (re-raise the UNKNOWN error) now fails 3 tests.

Round 4 (4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c): APPROVE. This was the last independent Sol review before the batch refresh, as recorded by the hub; CI was green at that head. No new independent review is claimed for the refreshed head. The hub dispatches the delta review after the refresh push.

Release

Held out of the 10/8 batch, which landed on 10/7 as uk-data 1.58.0. This PR is for the next uk-data batch, on Max's go (d833). The refresh does not merge the PR or request or dispatch reviews; the hub dispatches the delta review after the push.

This PR was rescued from an interrupted session's uncommitted patch and finished here. Changes from that patch:

• The minimum model version is 2.104.5 rather than 2.104.1. The patch's version would have run, and failed, the test on 2.104.1-2.104.4. It is now checked through what the imported model can do.
• The relabel in the loader is gated on that check, rather than left as a to-do.
• Stale-cache handling is new.
• The .-encoding test is folded into the exhaustive invariant test.
• The income-tax tests are new: they run under the lock and in uprated years.

axiom: n/a: SPI dataset construction, no policy rule changes

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the UK hub

• Head: 4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c.
• Independent review: four rounds of GPT-6.1 Sol on Subfleet. Rounds 1 and 2 requested changes, round 3 approved with changes (three nits), and round 4 APPROVE at this head, with all three nits resolved and nothing else changed. The PR body's "Review" section summarises each round. The reviews are kept with the hub's records.
• Passed at this head:
  • gh pr checks, all green. CI ran 646 passed, 3 skipped and 2 xfailed; one xfail is test_create_spi_output_with_unknown_region_can_be_simulated under the 2.93.0 lock, as designed.
  • mergeable is MERGEABLE.
  • The targeted SPI tests pass on policyengine-uk 2.93.0, 2.104.4, 2.104.5, 2.121.0 and 2.122.0, and in both version-skew directions.
  • All 14 mutations fail at least one test.
• Pending: nothing on this PR. It waits for the uk-data batch, d833: not in the 10/8 batch, and it merges after it on the uk-data release rules.
• Depends on: nothing open. Uprate rent for unknown-region households by the UK-wide index policyengine-uk#1985 is merged and first released in 2.104.5.
• Follow-ups owed:
  1. The relock to policyengine-uk 2.104.5 or later belongs to the batch, not this PR. At that relock the unknown-region test runs in full, and load_spi_dataset stops relabelling with no code change.
  2. After the 10/8 batch, run the rebuild command in the PR body. More than 60 GB must be free, and only one build may run at a time. It should confirm that the enhanced FRS is unchanged, and it rebuilds storage/spi_2022_23.h5.
  3. FYI, not this PR: the CI Test job failed once at 9fc0e29, in test_cgt_band_donors.py::test_built_total_gains. That test reads the enhanced_frs fixture, which this PR does not touch, and it passed on the three later heads.
  4. FYI for policyengine-uk: the model carries a single-year dataset's inputs only to 2030 (extend_single_year_dataset, end_year=2030). From 2031, any input variable that has a formula falls back to that formula. That applies to this PR's pays_scottish_income_tax and to any other dataset that sets such an input.