Derive property_wealth from its uprated components instead of saving the WAS total

[MaxGhenis]

Draft. This changes the built data, so it lands only in a batched uk-data release, on Max's go (decision d833). Please don't merge or release it on its own.

Part of the constituency-level LVT analysis: household land is property_wealth × regional intensity + owned_land, so a frozen property_wealth freezes household land.

The bug

• impute_wealth saves the WAS total DVPropertyR8 as a household column, property_wealth (datasets/imputations/wealth.py:50, 66, written by the loop at :343-350 on main).
• uprate_property_by_region multiplies owners' property_wealth by the same regional HPI factor as main_residence_value (regional_property_uprating.py:58-59, 80 on main).
• policyengine-uk defines property_wealth with adds = ["residential_property_value", "non_residential_property_value"], and residential_property_value adds main_residence_value and other_residential_property_value. That holds at the locked 2.93.0 and at main 4ec7ef1.
• A saved column overrides the formula. I checked this on a three-household synthetic dataset: with a saved property_wealth of [1, 2, 3], the model returned [1, 2, 3] in 2024 and in 2026. Without the column it returned the component sums, uprated.
• No uprating. property_wealth has no uprating attribute. It appears in neither policyengine-uk's data/uprating_indices.yaml (2.93.0 or main) nor uk-data's storage/uprating_factors.csv, which create_policyengine_uprating_factors_table generates from those attributes. The three components are in both.
• Result on release 1.57.4 (enhanced_frs_2024_25.h5, time_period 2024, policyengine-uk 2.93.0):
  • The saved value is identical per household in 2024, 2025 and 2026: the largest absolute difference is £0.
  • Over the same years, the sum of the components grows by 7.43% for every household with property.
  • property_wealth < main_residence_value for 1,291, then 28,937, then 29,677 of 35,672 owner households (2024, 2025, 2026).

Is the saved value the sum of its components in the base year?

No. Measured on 1.57.4 in 2024, before any uprating:

	Households	Weighted share	Saved = sum (±£1)	Saved > sum	Saved < sum	Weighted saved − sum
All	52,846	100%	76.4%	16.5%	7.1%	−£807.2bn
Owners with only a main residence	32,857	55.7%	74.9%	22.4%	2.7%	+£293.6bn
Owners with other property	2,815	7.3%	0.7%	10.1%	89.2%	−£1,280.8bn
Non-owners with other property	1,276	3.4%	26.1%	47.8%	26.1%	+£55.8bn
No property components	15,898	33.6%	96.9%	3.1%	0%	+£124.3bn

• Totals: the weighted saved total is £8,126.9bn and the component sum is £8,934.1bn. The components are main_residence_value £6,792.1bn, other_residential_property_value £1,895.9bn and non_residential_property_value £246.1bn.
• By region: the ratio of saved total to component sum runs from 0.698 (Yorkshire) to 1.031 (London).
• Earlier release: 1.56.14 shows the same pattern (−£734.5bn).

What the saved value carries that the components do not

• It is a separate draw. In microimpute 1.8.1, the version uk-data locks, each output is predicted from the predictors plus every earlier output (_get_sequential_predictors). property_wealth comes second, before the three components, so they are drawn conditional on it. Nothing makes them sum to it.
• Where the two disagree, the saved value mostly leaves property out. For 89.2% of owners with other property it is below the component sum, by £1.28tn in total.
• Where it is higher, it holds value that no saved component identifies:
  • 497 households (£124.3bn weighted) have a saved total but no property components;
  • owners with only a main residence add +£293.6bn.
• The WAS definition of DVPropertyR8 is UNVERIFIED. I could not read the WAS data dictionary in this task, so I can't say whether it is net of mortgages or includes land or overseas property. ONS describes published "net property wealth" as "the value of all properties minus mortgage debt". I have not confirmed that DVPropertyR8 is that variable. One pointer: the saved total is at least the main residence for 96.4% of owners in 2024, which would be unusual for a total net of mortgages. That is an inference from the imputed file, not a check against WAS.

Why (a), dropping the saved column, rather than (b), uprating it

• (b) cannot satisfy P4. The saved total is already below main_residence_value for 1,291 owners in the base year, and uprating does not change that.
• (b) keeps a total that disagrees with its own parts for 12,476 of 52,846 households.
• (b) also needs a policyengine-uk change. policyengine-uk uprates at load time from its own list, and uk-data's factor table is generated from policyengine-uk's uprating attributes.
• The model's definition is the gross sum of the three components. policyengine-uk documents total_wealth as gross assets: net_wealth subtracts mortgage_debt from it (main 4ec7ef1).
• Every other rule that reads property already uses the components, listed below. Land outside the three components is modelled separately: household_land_value adds owned_land, the WAS DVLUKValR8_sum.

What this PR does

• The WAS total stays in IMPUTE_VARIABLES. Removing it would change the predictor set, and so the draws, of every later output.
• It is listed in a new CONDITIONING_ONLY_VARIABLES and is no longer written; the write loop moves into store_wealth_predictions.
• uprate_property_by_region no longer touches it.
• No policyengine-uk change is needed. 2.93.0 already sums and uprates the components when the column is absent, as measured above.

Side effects

Consumers in policyengine-uk (main 4ec7ef1, from git grep)

• Read property_wealth, so they change:
  • household_land_value (property_wealth × intensity[region] + owned_land) → land_value → LVT.
  • total_wealth → household_wealth_decile, wealth_tax (ubi_center), non_primary_residence_wealth_tax (cec) and net_wealth. net_wealth exists on main but not in 2.93.0.
  • LVT, wealth_tax and non_primary_residence_wealth_tax feed household_tax and gov_tax, and LVT also feeds hbai_household_net_income. All three have default rates of 0, so baseline taxes don't move; reforms that set those rates do.
• Don't read it, so they're unaffected:
  • The UC, ESA, JSA, housing benefit, income support and pension credit capital rules. Their capital/sources.yaml lists other_residential_property_value and non_residential_property_value and excludes the main residence.
  • high_value_council_tax_surcharge (main_residence_value).
  • SDLT purchase values (main_residence_value, other_residential_property_value, non_residential_property_value).
  • corporate_land_value and the employer NI capital incidence (corporate_sector_wealth).

Measured on 1.57.4's weights with the column dropped (no recalibration)

Year	property_wealth	household_land_value	land_value	total_wealth
2024	£8,126.9bn → £8,934.1bn (+9.9%)	£5,113.9bn → £5,471.5bn (+7.0%)	£7,172.5bn → £7,530.1bn (+5.0%)	£20,468.0bn → £21,275.2bn (+3.9%)
2025	£8,185.4bn → £9,392.6bn (+14.7%)	£5,165.4bn → £5,752.3bn (+11.4%)	£7,224.0bn → £7,810.9bn (+8.1%)	£21,159.8bn → £22,367.0bn (+5.7%)
2026	£8,216.5bn → £9,703.6bn (+18.1%)	£5,195.3bn → £5,942.7bn (+14.4%)	£7,253.9bn → £8,001.4bn (+10.3%)	£21,620.5bn → £23,107.5bn (+6.9%)

household_wealth_decile changes for 10.3% of households (weighted) in both 2025 and 2026: 6.5% move up and 3.8% move down.

Calibration targets

• No target reads property_wealth directly. mhclg_regional_land.py builds its own property_wealth column from dwellings × average price, for regional shares only.
• Three groups of targets read household_land_value:
  • ons/household_land_value;
  • ons/land_value, through land_value;
  • 11 regional targets ons/household_land_value/{REGION}.
• They are wired in targets/build_loss_matrix.py:306-315 and targets/compute/other.py:65-74, and the national matrix is used by both the constituency and local-authority calibrations.
• The target values do not change. What changes is the model value the calibration fits to them. At 1.57.4's weights in 2025, the calibration year:
  • national household land is £5,752.3bn against a £5,042.9bn target, up from £5,165.4bn;
  • total land is £7,810.9bn against £7,100.0bn, up from £7,224.0bn.
• By region, household land moves:

Region	Change	Before → after (target)
London	+1.3%	£1,191.0bn → £1,206.3bn (target £1,188.2bn)
East of England	+1.5%	£548.9bn → £557.1bn (£542.7bn)
South East	+5.3%	£899.5bn → £947.6bn (£885.8bn)
South West	+6.2%	£468.5bn → £497.3bn (£456.7bn)
Wales	+11.1%	£186.8bn → £207.6bn (£179.3bn)
West Midlands	+16.6%	£370.8bn → £432.4bn (£364.3bn)
East Midlands	+17.1%	£310.4bn → £363.6bn (£304.8bn)
North West	+19.4%	£432.3bn → £516.0bn (£422.9bn)
North East	+29.9%	£119.6bn → £155.3bn (£119.6bn)
Yorkshire	+36.3%	£305.7bn → £416.8bn (£298.1bn)
Scotland	+40.7%	£274.9bn → £386.9bn (£280.4bn)

• The next build will reweight to close these gaps, most in Scotland, Yorkshire and the North East. How that trades off against other targets is known only after a full build. This PR's CI build uses TESTING=1 (32 epochs, 2 OA clones).
• The existing land tests still pass with the column dropped: test_land_value_targets.py (±65%) and test_household_land_less_than_property_wealth, both at 1.57.4's weights.

Invariants and how they're tested

Invariant	Scope	Tests
I1. The built dataset saves no property_wealth column	Imputation step; built file	test_impute_wealth_does_not_save_property_wealth (impute_wealth with a stub model); the Hypothesis property; test_built_dataset_does_not_save_property_wealth
I2. property_wealth = main_residence_value + other_residential_property_value + non_residential_property_value per household (rtol 1e-5, atol £1)	Base year, calibration year and base+2 (2024, 2025, 2026); also after uk-data's uprate_dataset to the calibration year	Hypothesis property; test_built_property_wealth_is_the_sum_of_its_components[year]
I3 (P4). property_wealth ≥ main_residence_value per household	Same years	Hypothesis property; test_built_property_wealth_at_least_main_residence[year]
I4. Per household with property, property_wealth(2026)/property_wealth(2024) equals the same ratio for the component sum (rtol 1e-5)	2024 to 2026	Hypothesis property; test_built_property_wealth_grows_with_its_components
I5. The WAS total is imputed before the three components, so they stay conditioned on it	IMPUTE_VARIABLES order	test_property_wealth_is_imputed_before_its_components

• The Hypothesis property is test_saved_property_wealth_follows_its_components, 25 examples:
  • 1–8 households in any GB region;
  • each component is 0 or £1k–£5m;
  • plus an arbitrary WAS total;
  • each example runs store_wealth_predictions, then uprate_property_by_region, then policyengine-uk.
• The built-dataset tests run after make data in CI and skip when the file is absent. Their failure messages give counts and sums only, never individual records.
• test_regional_property_uprating.py: the proportional property_wealth test is replaced by test_only_main_residence_value_is_rescaled.
• Scope: these are properties of this pipeline with policyengine-uk's current definitions. If policyengine-uk ever gives the components different uprating indices, I4 still compares against the component sum, not one index.

Commands run and results

All commands ran in a fresh clone of b45c373 on Python 3.13.

• uv sync --frozen --all-extras --python 3.13: OK. uv lock --python 3.13 added hypothesis 6.168.5, plus sortedcontainers as its dependency.
• uv run --frozen pytest policyengine_uk_data/tests/test_property_wealth_components.py policyengine_uk_data/tests/test_regional_property_uprating.py policyengine_uk_data/tests/test_student_loan_balance.py: 15 passed, 8 skipped (the skips are the built-dataset tests, since no local build exists).
• Mutation check. With CONDITIONING_ONLY_VARIABLES = (), so that the column is saved again, the Hypothesis property and test_property_wealth_is_imputed_before_its_components both fail.
• Built-dataset tests against the unmodified 1.57.4 file: all 8 fail, as intended:
  • "saves a property_wealth column";
  • the sum differs for 12,476 / 37,445 / 37,445 of 52,846 households in 2024 / 2025 / 2026;
  • P4 fails for 1,291 / 28,937 / 29,677 owners;
  • growth fails for 36,948 of 36,948 households with property: mean 1.0000 against 1.0743.
• Built-dataset tests against 1.57.4 with only the property_wealth column dropped, plus test_land_value_targets.py: 18 passed.
• uv run --frozen ruff format --check . (ruff 0.15.5): 217 files already formatted.
• uv run --frozen pytest policyengine_uk_data/tests (full local suite, no local build): 524 passed, 121 skipped, 1 xfailed. The skips are the tests that need built or private data. CI builds the dataset and runs them; see the checks on this PR.
• Diagnostics (aggregates only; the scripts are not in this PR): base-year comparison, effect at released weights, and the wealth-decile shift. Each was run on 1.57.4_enhanced_frs_2024_25.h5; the base-year comparison also ran on 1.56.14_enhanced_frs_2024_25.h5.

Not done

• No full release build. The post-calibration effect on weights, on the other targets and on household land by region and year is not measured; the CI TESTING build is the only rebuild.
• The WAS definition of DVPropertyR8 is unverified (see above).
• Releases up to 1.57.4 keep the saved column, so anyone on them keeps the frozen value until a new release. policyengine-uk could ignore a saved property_wealth defensively; that is not done here.
• Some other property is not rescaled. The regional HPI rescale applies to main_residence_value only. Before this PR it also scaled the WAS total for owners. other_residential_property_value and non_residential_property_value are not rescaled, and whether they should be is a separate methodological question.
• The effect on published LVT figures is not computed.
• Overlap with Calibrate DWP and OBR welfare targets on the countries their source covers #542, which also adds hypothesis to the dev extras: whichever merges second needs uv lock rerun.

🤖 Generated with Claude Code

[juaristi22]

Review of af64d3757c2648e820219faee96186d3d7cd9a63

Review status: PARTIAL
Review severity: COMMENT
Confirmed critical count: 0

No confirmed code or policy defects were found in the changed behavior and affected dependencies. The persistence filter follows the locked model's property-wealth definition, leaves sequential predictions and other saved outputs unchanged, and remains effective through calibration and export. One material source-coverage question remains unresolved.

Evidence gap: property asset coverage

At wealth.py:289–290, omitting the saved total activates a sum of DVhvalueR8, DVHseValR8_sum and DVBlDValR8_sum (source mapping at lines 102–104). The unresolved question is whether those source fields cover every property category intended for downstream total wealth, and what property categories the discarded DVPropertyR8 previously carried. ONS methodology, table 1 includes other land/property in household gross property wealth; the round-8 guide, physical page 39 separately explains net property wealth. Neither establishes the exact field-level mapping. In locked UK 2.93.0, owned_land denotes land-only plots and enters household_land_value, but does not enter total_wealth. The missing source definitions therefore leave unresolved whether activating the sum loses a category from total wealth, or changes overlap in the land calculation; this matters for wealth deciles and nonzero wealth/LVT reforms. Obtain the round-8 derived-variable specification for the four property fields and DVLUKValR8_sum to resolve this. This is not a confirmed omission/double-counting bug, and not merely a failure to corroborate background PR narrative about whether DVPropertyR8 is net or gross.

Validation

Verified CI run 37410613766 at the reviewed head: the TESTING=1 dataset build succeeded, followed by 642 passed, 3 skipped, 1 xfailed. All 11 property-component tests (including all 8 built-dataset checks), 6 regional tests, 6 student-loan tests and 8 land-target tests passed. Lint, changelog, documentation and bundle-manifest checks also passed.

Local tests were not run because the available environments lacked the required dependencies; none were installed. The locked policyengine-uk 2.93.0 source was independently verified against its published wheel. The arithmetic and persistence tests do not resolve the source-field coverage question above.

Scope and limitations

• Filtering occurs after model.predict, so this edit preserves the sequential predictor set and later draws. The newly calculated total follows the existing model definition and its uprated components.
• National and regional land targets are unchanged; their model contributions change as disclosed. No full production recalibration was run, so effects on production weights, distributional results and published LVT estimates remain unmeasured. This is a release-impact limitation, not a separate confirmed defect.
• This PR changes future builds. Existing published datasets retain the saved column; no migration or defensive fix for those releases is claimed.
• The unknown gross/net definition of DVPropertyR8 alone does not establish a defect. The unresolved question is the coverage of the newly activated aggregate.

Primary evidence: ONS wealth methodology, WAS round-8 guide, and policyengine-uk 2.93.0. The exact WAS derived-variable specification was not obtained.