Impute additional-dwelling and non-residential purchases, calibrated to HMRC's HRAD statistics

[MaxGhenis]

Draft for the d833 batch, landing only on Max's go.

Fixes #556. Pairs with PolicyEngine/policyengine-uk#2241, which makes additional-dwelling and non-residential purchases explicit inputs. This supplies independent annual purchases after wealth, geography and calibration-year uprating exist, before weight calibration. Each purchase is positive, at least £40,000 and no greater than its household's corresponding property stock.

This must land in the same batch as #569, which targets household SDLT at the OBR total less corporate SDLT. Without these purchases, calibration would push main-home SDLT to absorb the missing higher-rates share. #569's SDLT target test should be checked on the combined rebuilt batch; this PR leaves that test and its target logic unchanged. create_datasets.py is touched only to insert the purchase stage and record its metadata.

Methodology choices for Max to review:

• Household HRAD transactions: 186,588.8624 in England and Northern Ireland. This assumes HRAD has the same individuals' transaction share as all residential buyers: 211,700 × 925,100 / (925,100 + 124,500). This is an assumption, not a published household HRAD count. The independently valid lower bound is 87,200.
• Household HRAD purchase value: £68.805261767bn, allocating £77.97bn by individuals' residential value share: 335,515 / (335,515 + 44,690). The independent lower bound is £33.280bn. The proportional gross-receipts benchmark is £4.790657831bn; the independent gross-receipts lower bound is £4.205bn. Receipts are checked rather than fitted.
• Event propensity: stock-proportional purchase hazard, exponent 1, with its seeded threshold fitted to the closest attainable weighted transaction count. Larger portfolios are assumed more likely to trade. A uniform owner draw on the supplied older dataset selected only £48.968bn of HRAD stock against the £68.805bn value target; this choice makes the value target feasible without tuning seeds. Households below £40,000 stock and zero-weight donors cannot purchase. At most one purchase of each type occurs per household each year.
• Purchase prices: seeded draws of band means, followed by a common price multiplier fitted to the weighted value target, with the £40,000 floor and stock cap applied throughout. Residential bands use all-buyer Table 6b as a proxy for household prices; non-residential bands use individuals' own Table 8b distribution. This capped and scaled method changes the published band distribution; it is not an exact conditional draw. Rounded band counts are normalised by their own sums.
• Non-residential targets: Table 8b individuals: 24,800 transactions / £6.955bn / £230m gross SDLT for 2023-24; 25,000 / £7.640bn / £245m for 2024-25. These replace the older household figures in policyengine-uk's statistics parameter.
• Periods and geography: HRAD assumptions are held flat across survey and calibration years; the latest non-residential assumptions carry forward after 2024. This is an explicit extrapolation, not a forecast. England/NI fitting uses household residence as the transaction-jurisdiction proxy; the fitted stock hazard also applies in Scotland/Wales without devolved targets. The mean fitted 2024 event rates in this local check are 12.4534% for HRAD and 5.6170% for non-residential owners; they are recomputed from the dataset's weighted owners rather than fixed national rates.
• Seeds: residential event/price 55601/55602; non-residential event/price 55603/55604. They do not use the main-residence property_purchased draw or global NumPy state. Purchase prices uprate on exactly the corresponding stock indices, including calibration-to-base-year materialisation and table generation on older model versions.

Sources: HMRC UK Stamp Tax Statistics 2024-25, corrected tables, Tables 6b/8a/8b. HMRC Table 6b's all-buyer HRAD totals are 211,700 transactions, £77.970bn value and £5.435bn gross SDLT (£5.44bn rounded); its surcharge-only receipts are £2.790bn.

Read-only validation applied the imputation to the supplied enhanced_frs_2023_24.h5 household table (source year 2023), projecting stocks and weights with the #2241 model's dataset indices and re-imputing for each year. The script reads only four household columns and prints weighted aggregates and boolean checks. This is an annual imputation check on an already-built older dataset, not a full recalibrated 2024-25 release.

Metric, England and Northern Ireland	Selected household target/reference	Imputed 2024	Imputed 2025	HMRC 2024-25 all buyers
HRAD transactions	186,589	186,447	186,737	211,700
HRAD purchase value	£68.805bn	£68.805bn	£68.805bn	£77.970bn
Gross SDLT on HRAD purchases	£4.791bn reference	£4.051bn	£4.052bn	£5.435bn
Additional HRAD liability from +2 percentage points	—	£1.376bn	£1.376bn	—
Non-residential transactions	25,000	25,291	25,474	—
Non-residential purchase value	£7.640bn	£7.640bn	£7.640bn	—
Non-residential gross SDLT	£245m reference	£121.785m	£119.448m	—

For source-year 2023, HRAD is 186,994 transactions / £68.805bn / £4.041bn gross SDLT; non-residential is 25,024 / £6.955bn / £90.673m, against 24,800 / £6.955bn / £230m. Every annual run was reproducible, only eligible stock owners purchased, every price met the floor/cap, counts were within 5% and values matched the targets numerically.

Gross SDLT uses the actual #2241 progressive scale at 1 January (including standard rates), rather than just the surcharge. HMRC receipts span a financial year, policy changes and replacement-home refund timing. Residential receipts are 15.4% below the proportional reference in 2024; the +2pp exposure is non-zero. Non-residential receipts are about half the household reference and are not calibrated. The capped price distribution needs review before batch release. On this older table, 67.3% of weighted HRAD purchases and 22.5% of non-residential purchases hit the stock cap in 2024; fitted band-price multipliers are 2.4275 and 3.5540 respectively. These aggregate checks establish exposure, not a final receipts fit or the result of rebuilding the current survey.

Validation and tests:

• 18 purchase-imputation unit tests passed, including reproducibility independent of global RNG, separate property-type seeds, main-purchase independence, ownership/minimum/stock bounds, weighted counts and numerically fitted values, and infeasible inputs.
• Consolidated source verification: 44 passed, 6 integration cases deselected (18 purchase-imputation, 10 uprating regression, 16 existing stochastic-parameter cases). New-file lint, repository formatting (296 files) and git diff --check passed.
• Six integration cases require a rebuilt enhanced FRS and remain unrun locally. They check saved inputs, stocks and weighted count/value targets in both release years (20% tolerance after weight calibration), and the direct gross HRAD SDLT benchmark (30%, reflecting the price and policy-period assumptions). Pure synthetic counts use 10% tolerance; fitted values use relative 1e-8. The 20% post-calibration tolerance is provisional until the combined batch is built.
• Local source tests use a namespace bootstrap because the supplied model-fixes venv lacks unrelated dataset-build dependencies; the imputation and uprating source functions are unmodified. The aggregate validation used the real #2241 model parameters. No other datasets were downloaded. No upload pipeline or data visibility changes.

🤖 Generated with Claude Code