Stop counting FRS property losses as income; count sub-let rent for every tenure

[MaxGhenis]

Summary

Stacked on #503: its two commits appear here until it merges, after which this branch is rebased onto main. Review only the last two commits.

frs_property_income (added in #503) builds each person's property_income from two FRS amounts: ROYYR1 (rent from other property) and SUBRENT (rent from letting part of the home). Three problems found while reviewing #503:

1. Property losses counted as income. ROYYR1 is always stored as a positive amount, and a separate answer (RENTPROF) says whether it is a profit or a loss. The build ignored RENTPROF. A loss now counts as zero.
2. SUBRENT counted only for owner-occupiers. The FRS asks every household about sub-letting, whatever its tenure. The owner-only restriction is removed.
3. SUBRENT before or after expenses (SUBALLOW). No change. SUBRENT stays as reported, for the reasons below, and the helper's docstring says so.

The helper also now floors each amount at zero separately. Negative FRS values are missing-value codes (−1 to −9), not amounts. Flooring the sum let a missing ROYYR1 code cancel the household reference person's SUBRENT, and that case now arises for renting households as well. FRS 2024-25 has no negative codes in either field, so this part changes nothing in the current build.

Evidence

Sources: the documentation shipped with FRS 2024-25 (UK Data Service SN 9563), the older FRS derived-variable specification (SN 4149), and aggregates from four raw FRS years (2020-21, 2022-23, 2023-24 and 2024-25). Cells under 10 survey records are suppressed throughout.

Losses (ROYYR1, RENTPROF)

• Questionnaire. PropRent asks for rent received from other property in the last 12 months, before tax and after allowable expenses. The instructions say the questionnaire cannot accept negative amounts, so RentProf then asks whether the amount is a profit or a loss (1 = profit, 2 = loss). Tax paid (Proptax) is asked only after a profit.
• Routing, in all four years. Every adult with ROYAL1 = 1 (rent from property) answers RENTPROF, nobody else does, and nobody else has ROYYR1 above zero. No loss-coded adult reports tax paid on the rent.
• DWP's own derivation. In the released data, DWP's derived "remaining income" (INRINC) is negative for nearly all loss-coded adults with ROYYR1 > 0, and equal to minus ROYYR1. For every profit-coded adult with ROYYR1 > 0 it is at least ROYYR1.

FRS year	Loss-coded adults with ROYYR1 > 0	People (survey weights)	Annualised
2020-21	22	40k	£0.115bn
2022-23	39	49k	£0.253bn
2023-24	38	59k	£0.164bn
2024-25	42	66k	£0.317bn

Why zero and not a negative amount.

• policyengine-uk has no property loss input; its only loss inputs are for trading (trading_loss, loss_relief).
• property_income must be non-negative (tests/test_non_negative_incomes.py).
• For income tax in the year, a property business loss does not reduce other income. It is carried forward against later profits of the same property business (ITA 2007 s. 118). It is set against general income only when it has a capital allowances or agricultural connection (s. 120), which the FRS does not record.
• The loss is not set against the same person's SUBRENT. In none of the four years does a household with SUBRENT > 0 have a reference person reporting a property loss.

Sub-letting (SUBLET, SUBRENT)

• Questionnaire. SubLet is in the tenure section. It asks whether the household has a formal arrangement to let or sub-let any part of the accommodation to someone who is not a member of the household. SubRent follows a yes and asks for the rent received in the last 12 months, before income tax.
• Routing, in all four years. Every household answers SubLet, whatever its tenure: owners, renters, and rent-free households. SUBRENT is above zero only where SUBLET = 1.
• DWP's own derivation. The derived-variable specification takes SUBLTAMT ("rent received from subletting", on benefit unit 1) from SUBRENT whenever SUBLET = 1. It adds SUBRENT to the head of household's INRINC on the same condition. Neither has a tenure condition. In the released data, SUBLTAMT equals SUBRENT in every household with SUBRENT > 0, and the reference person's INRINC is at least SUBRENT.
• Renting and rent-free households. Pooled over the four years, 30 households reported SUBRENT > 0. That is about 21k households and £0.05bn a year on average (survey weights). Single years are not shown, because some fall under 10 records.

No documentation supports the owner-only restriction, so it is removed. SUBRENT still goes to the household reference person, as before.

SUBRENT before or after expenses (SUBALLOW)

SubAllow asks whether the SubRent amount is before or after allowable expenses (1 = before, 2 = after). ROYYR1 is always after expenses.

FRS year	SUBRENT > 0, before expenses	after expenses
2022-23	44 households	14
2023-24	27	13
2024-25	23	15

SUBRENT stays as reported, whichever answer was given:

• The FRS collects no expense amount, so any deduction from the before-expenses answers would be an invented ratio.
• DWP's derived SUBLTAMT and INRINC use SUBRENT as reported, whatever SUBALLOW says.
• What matters downstream is how policyengine-uk treats sub-letting income. Move SUBRENT from property_income to sublet_income, after policyengine-uk reads sublet_income #508 moves SUBRENT to sublet_income once policyengine-uk reads that variable, for rent-a-room relief and the means-test disregards. There, the SUBALLOW = 1 answers are gross receipts and the SUBALLOW = 2 answers are net, and Move SUBRENT from property_income to sublet_income, after policyengine-uk reads sublet_income #508 records that distinction. Until then, before-expenses SUBRENT sits in property_income next to after-expenses ROYYR1. policyengine-uk then deducts the £1,000 property allowance from the total; policyengine-uk#1900 tracks that allowance being applied to an already-net amount.

Invariants and tests

frs_property_income(person, household) should satisfy these for every valid pair of tables, meaning every household in household has exactly one person with hrpid == 1, as in the FRS. "Floored" below means floored at zero, which is how missing-value codes are handled.

• Non-negative. Every result is at least zero.
• Conservation. Total property income = WEEKS_IN_YEAR × (floored SUBRENT summed over all households + floored ROYYR1 summed over everyone not coded as a loss). Each amount is counted exactly once.
• Locality. Raising one person's floored ROYYR1 by £x changes only that person's result: by WEEKS_IN_YEAR × £x, or not at all if the amount is coded as a loss. Raising one household's floored SUBRENT by £x changes only that household's reference person's result, by WEEKS_IN_YEAR × £x.
• Loss removal. Coding one person's amount as a loss lowers only that person's result, by the amount it previously counted: WEEKS_IN_YEAR × floored ROYYR1 if it was counted, otherwise nothing.
• Independence. The result does not depend on CVPAY, TENTYP2 or SUBALLOW.

tests/test_frs_property_income.py checks each invariant on 200 seeded random sets of households. The random tables include missing-value codes, losses and all eight tenures. Hypothesis is not a dependency of this repo, so the tests follow the file's existing seeded style. A differential test checks the vectorised helper against a one-person-at-a-time loop version of the same rules on the same 200 sets. Example tests cover:

• a lodger's rent;
• rent from other property for a non-HRP adult;
• a loss, which neither counts nor reduces SUBRENT;
• an amount with no profit-or-loss answer, which still counts;
• SUBRENT for each of the eight tenures;
• SUBRENT whatever SUBALLOW says;
• missing-value codes;
• the repeated index labels that create_frs produces.

Built dataset against raw tables. test_built_frs_matches_the_raw_tables reads ROYYR1, RENTPROF, HRPID and SUBRENT straight from the raw tables and joins them on SERNUM. It then compares the result with property_income in the built base FRS. This covers the call site in create_frs and the raw column names and codes. It runs in CI, which builds the dataset, and its assertion carries no values. Locally, it fails on the base FRS built from #503's head, fails on a build with only the loss fix, and passes on this branch.

Mutation check. Each of seven deliberate bugs makes between 424 and 665 of the file's 1,218 synthetic tests fail:

• ignoring RENTPROF;
• restoring the owner-only rule;
• netting losses against SUBRENT;
• dropping before-expenses SUBRENT;
• giving SUBRENT to every adult;
• treating the profit code as the loss code;
• flooring the sum rather than each amount.

Measured impact

All figures are aggregates. Simulations use policyengine-uk main (da947eee). The baseline is #503's head (0812cd1). The impact of #503 itself is in that PR.

Base FRS 2024-25 (survey weights, no imputation or calibration)

I built the base FRS three times with create_frs: on #503's head, on this branch, and on this branch with the owner-only SUBRENT rule put back (the loss fix alone). I compared every column of every table.

• Only property_income changes. It is the only column that differs between any two of the three builds.
• Loss fix. 42 records lose their property income, each falling to zero. Stored property_income falls from £16.18bn to £15.86bn (−£0.32bn). People with any property income fall from 1.87m to 1.80m.
• Sub-letting change. It adds property_income to fewer than 10 records in 2024-25, so its 2024-25 effect is not shown. That includes any total from which it could be worked out by subtraction. The pooled four-year figure above gives its scale.

Loss fix alone, 2026:

2026	#503 head	loss fix	change	survey units that change
Property income	£17.57bn	£17.22bn	−£0.34bn	42 people
Taxable property income	£15.80bn	£15.52bn	−£0.29bn	29 people
Income tax	£297.14bn	£297.05bn	−£0.09bn	27 people
Household net income	£1,417.03bn	£1,416.80bn	−£0.23bn	38 households
Housing Benefit, Pension Credit			none	0

Universal Credit, council tax reduction and poverty change for fewer than 10 survey units each, so those changes are not shown.

Enhanced FRS (full rebuild)

I built the enhanced FRS on #503's head (0812cd1) and on this branch, twice each, with two different seeds. The branch builds are at be4d935 (seed 0) and the current head cc937e0 (seed 1). Those two commits give an identical 2024-25 base FRS in every column. All builds used production settings (512 epochs, PE_UK_DATA_OA_CLONES=1), the repo's uv.lock (policyengine-uk 2.93.0), the same raw inputs and the same cached trained models, in worktrees set up the same way.

Two outside inputs are pinned, by a wrapper that changes no repo code:

• The random seed. The calibration's weight dropout draws from torch's global generator, which the build does not seed (Seed the calibration weight dropout so builds are reproducible #507 fixes that). The wrapper seeds it.
• Web downloads. The calibration targets are downloaded at build time, and a failed download changes the targets without stopping the build. One of my unpinned builds lost its ONS tenure targets to an HTTP 429. The wrapper caches each response, so every build calibrates to the same targets.

Check that this makes builds comparable. I built #503's head twice with seed 0 in two worktrees, one with live downloads and one from the cache. The two builds are identical in every column of every table, household weights included.

So within each seed, the difference between the baseline and the branch is caused by the code change. It does not come only through the records whose income changed. Two steps before calibration propagate it, and neither depends on the seed, so the same differences appear in both pairs:

• Capital gains donors. stack_cgt_band_donors draws 270 donor households with a fixed seed and probabilities that depend on total income. Baseline and branch share 82 of the 270.
• Second-stage imputation. frs_only.py refits a model on the FRS persons in every build, with property_income among its predictors. Among the 52,576 non-donor households, which are the same in both builds, 33 person columns differ. For example, reported Universal Credit differs on 894 person rows and capital gains on 23,801.

Calibration then reweights every household, and that step does depend on the seed.

2026	#503 head, seed 0	change, seed 0	change, seed 1
Property income	£59.49bn	+£0.06bn	−£0.02bn
Property income, FRS half	£45.22bn	−£0.19bn	−£0.20bn
People with property income	2.555m	+4k	+3k
Income tax	£313.88bn	+£0.35bn	+£0.00bn
Universal Credit	£80.01bn	−£0.49bn	−£0.49bn
Housing Benefit	£14.10bn	−£0.04bn	−£0.01bn
Pension Credit	£6.94bn	−£0.02bn	−£0.01bn
Council tax reduction	£2.18bn	+£0.02bn	+£0.02bn
Household net income	£1,764.0bn	+£1.5bn	+£0.1bn
Poverty BHC, all people	14.64%	+0.06pp	+0.07pp
Child poverty BHC	16.60%	+0.14pp	+0.15pp
Child poverty AHC	23.73%	+0.37pp	+0.39pp
Effective sample size, households	1,221	+8	+8

• Property income. The total barely moves in either seed, because calibration reweights to the HMRC property income targets. The FRS half loses about £0.2bn in both.
• The same in both seeds: Universal Credit (−£0.49bn), council tax reduction, the poverty rates and the effective sample size. These come through the donor draw and the refitted imputation, which are identical across seeds. They are consequences of the rebuild, not of the corrected incomes directly: on the base FRS, the loss fix changes Universal Credit, council tax reduction and poverty for fewer than 10 survey units each.
• Different between the seeds: income tax (+£0.35bn and +£0.00bn) and household net income (+£1.5bn and +£0.1bn). That is calibration noise of the size Stop counting rent paid by boarders and lodgers as their property income #503's unseeded builds showed: four builds of main there spanned £2.9bn of household net income.

Calibration targets

Each build's fit to the national targets at the calibration year (2025), from get_loss_results. Every build is scored against the same cached targets.

	#503 head, seed 0	branch, seed 0	#503 head, seed 1	branch, seed 1
Mean absolute relative error, 636 targets	0.1750	0.1766	0.1737	0.1750
Targets within 10%	441	435	440	435
Mean absolute relative error, 26 HMRC property income targets	0.1683	0.1702	0.1641	0.1651
Property income targets within 10%	21	21	22	22
Property income, sum over the amount targets (target £55.78bn)	£57.29bn	£57.31bn	£56.96bn	£56.92bn
People with property income, sum over the count targets (target 2.351m)	2.480m	2.480m	2.463m	2.462m

No property income target's relative error moves by more than 0.025 in either seed. The largest moves in both seeds are the same targets: three DWP Universal Credit payment bands and three HMRC capital gains bands.

Not in this PR

• Moving SUBRENT to sublet_income. That is Move SUBRENT from property_income to sublet_income, after policyengine-uk reads sublet_income #508, which has to follow a policyengine-uk release that reads sublet_income.
• Losses as negative income. DWP's derived INRINC subtracts property losses. Doing the same in household income measures, or carrying losses forward for income tax, needs a property loss input that policyengine-uk does not have.

Merge note

This PR lands only in the single uk-data release batch that Max approves (d833), after #503 in that batch. Every merge to uk-data main builds and publishes a release, so nothing here merges on its own.

If #503 lands as a merge commit, this branch needs no rebase. If it is squash-merged, drop its two commits rather than replaying them (git rebase --onto origin/main 0812cd1), then re-run CI on the rebased head.

axiom: n/a: data pipeline change (FRS variable mapping), no rule change.

🤖 Generated with Claude Code

[MaxGhenis]

Handing this PR to the UK hub; it lands only in Max's uk-data release batch (d833).

• Head: cc937e0555d7ead58fc980b41b1959510709b1db.
• Review: the UK hub's independent review, APPROVE WITH NITS at this head (~/reviews/uk-hub/batch-reviews/ud509/review.md). Both nits were in the PR body, not the code (state the invariants' preconditions; make the merge note follow d833). Both are fixed. No commits since the review.
• Passing: gh pr checks exits 0 (4 checks), MERGEABLE, out of draft.
• Impact: in the body. The base FRS shows the loss fix on its own. The enhanced FRS has two seeded rebuild pairs: the changes that repeat in both seeds come through the capital-gains donor draw and the refitted stage-2 imputation, and the rest is calibration noise.
• Depends on Stop counting rent paid by boarders and lodgers as their property income #503. This branch carries Stop counting rent paid by boarders and lodgers as their property income #503's two commits. If Stop counting rent paid by boarders and lodgers as their property income #503 lands as a merge commit, no rebase is needed. If it is squash-merged, run git rebase --onto origin/main 0812cd1, then re-run CI.
• Conflict with Keep FRS-reported dividends and key them on person_id #498, only at the frs.py call site (trial merge 2026-10-04). Keep both one-line calls:
  • pe_person["dividend_income"] = frs_dividend_income(account, person.person_id)
  • pe_person["property_income"] = frs_property_income(person, household)
• Pending: only the d833 batch go and the merge order.
• Follow-ups, not owed here:
  • Move SUBRENT from property_income to sublet_income, after policyengine-uk reads sublet_income #508 moves SUBRENT to sublet_income once policyengine-uk reads it.
  • The same fixes for microcosm's FRS spine are a separate task already in progress.
• Evidence: ~/reviews/frs-property-losses-subrent-2026-10-01/ (STATUS.md).

[MaxGhenis]

Queued in release PR #544 for the 10/8 uk-data batch. It lands only on Max's go (d833).