Split VAT grossing into household share and survey coverage

[vahid-ahmadi]

Fixes #1996.

What was wrong

vat and baseline_vat divide household VAT by gov.simulation.microdata_vat_coverage, a single value of 0.38 from 2010 with no derivation. It came in as a hard-coded constant in January 2023 ("Adjust VAT to hit admin targets", 787e6b9), set against the LCFS-based consumption imputation of the time, and moved to a parameter file in #1139.

On today's datasets it no longer reproduces receipts. With 0.38, 2024-25 VAT is £211.8bn on Microcosm UK 2024-25 (24% above OBR's £171.0bn outturn) and £292.1bn on Enhanced FRS 2024-25 from policyengine-uk-data 1.56.16 (71% above).

Fix

The issue's option 3: the scalar is split into its two parts, so only one of them moves with the data.

• gov.simulation.vat.household_share_of_receipts = 0.7. HMRC's VAT gap estimates put household consumption at around 70% of the VAT total theoretical liability; the rest is government, charities, VAT-exempt businesses and new housing. This is a published quantity and doesn't depend on the dataset.
• gov.simulation.vat.survey_consumption_coverage = 0.67. This is the share of household VAT liability that the dataset's consumption captures, derived on the certified Microcosm UK 2024-25 release.
• vat = statutory VAT on recorded consumption ÷ (coverage × household share). The product is 0.469, against 0.38 before.
• gov.simulation.microdata_vat_coverage is kept as a deprecated parameter, so existing reforms to it still work. Its value is now the product of the two factors (0.469). If a reform sets it to any other value, vat and baseline_vat use that value in place of the two factors (policyengine_uk/utils/vat.py); otherwise they use the two factors. test_vat_grossing_legacy.py covers both cases.

The derivation is in docs/engineering/vat-coverage.md and the parameter's metadata: dataset revisions and SHA-256 hashes, the OBR and HMRC inputs, the formula, per-year results for each dataset, a bound, and a snippet that reruns it.

Derivation

Dataset	Year	Pre-scaling VAT	OBR VAT (EFO March 2026, table 3.8)	Implied product	Implied coverage
Microcosm UK 2024-25 (f9d1922c)	2024-25	£80.48bn	£171.0bn (outturn)	0.471	0.672
Microcosm UK 2024-25	2025-26	£83.82bn	£180.2bn	0.465	0.665
Microcosm UK 2024-25	2026-27	£86.07bn	£187.7bn	0.459	0.655
Enhanced FRS 2024-25, uk-data 1.56.16	2024-25	£111.00bn	£171.0bn (outturn)	0.649	0.927
Enhanced FRS 2024-25, uk-data 1.58.0	2024-25	£115.72bn	£171.0bn (outturn)	0.677	0.967

Pre-scaling VAT = 0.20 × standard-rated consumption + 0.05 × reduced-rated consumption, weighted (on Microcosm in 2024-25, 0.20 × £396.15bn + 0.05 × £25.02bn). Both datasets are in private Hugging Face repositories; the doc gives the revisions and hashes, so anyone with PolicyEngine access can rerun it, and the aggregates let anyone else check the arithmetic.

Independent check: HMRC's ready reckoner (June 2025) gives £8.8bn for one point on the standard rate in 2026-27. On Microcosm that needs a product of 0.481, within 3% of the level-based value.

Bound. Total VAT depends only on the product. On Microcosm it lies between 0.459 (receipts, 2026-27) and 0.481 (ready reckoner), so the coverage lies between 0.655 and 0.688 at a household share of 0.7. The value used, 0.67 (product 0.469), is inside that range.

The implied product moves by under 3% across 2024-25 to 2026-27, so the factors aren't dated by year. A change of default dataset can't be expressed through policy dates, so the doc says to re-derive the coverage when the default dataset changes.

Decision to confirm: which dataset the coverage targets

There's no default dataset in policyengine-uk. policyengine.py 6.2.1's managed UK default is still Enhanced FRS 2024-25, while Microcosm UK 2024-25 is the certified release the data work is moving to. I set the coverage on Microcosm. Compared with 0.38, that improves both datasets:

Total VAT vs OBR	Before (0.38)	After (0.469)
Microcosm UK 2024-25, 2024-25	+23.9%	+0.4%
Microcosm UK 2024-25, 2025-26	+22.4%	−0.8%
Microcosm UK 2024-25, 2026-27	+20.7%	−2.2%
Enhanced FRS 2024-25 (uk-data 1.56.16), 2024-25	+70.8%	+38.4%
Enhanced FRS 2024-25 (uk-data 1.58.0), 2024-25	+78.1%	+44.3%

The factor applies to Microcosm UK 2024-25 only. On Enhanced FRS it still overstates VAT, by 38% to 44% in 2024-25 depending on the release.

Targeting Enhanced FRS instead (coverage 0.93 on 1.56.16, 0.97 on 1.58.0) would fix it there but put Microcosm 28% to 30% below receipts.

Effects

• Every household's vat falls by 19% (0.38 / 0.469), on any dataset. baseline_vat falls the same way, so vat_change scales too.
• VAT reform costings fall by the same 19%. The microsimulation fixture for a 2pp standard-rate rise moves from 21.3 to 17.3 (an exact rescaling; that suite isn't run in CI).
• policyengine-uk-data's reform-impact test "Raise VAT standard rate by 2pp" (£43bn after its own changelog entry) will need the same rescaling on its next policyengine-uk bump.
• Datasets that calibrate to OBR VAT (policyengine-uk-data, Microcosm) no longer start 24% to 71% above the target.

Interaction with #2166

#2166 adds an explicit domestic fuel and power VAT base from electricity_consumption and gas_consumption. This PR doesn't touch the energy side of vat.py. When #2166 lands, an energy base already calibrated to an aggregate shouldn't be divided by the survey coverage again, and the reconciled energy share of reduced_rate_vat_consumption comes out of the base this coverage was derived on. The coverage should be re-derived then.

Tests

• New gov/hmrc/vat/vat_grossing.yaml:

  • grossing by both factors (2,050 / 0.469 = 4,371.00 for vat and baseline_vat);
  • both factors at one returns the statutory liability;
  • the household share alone.

  The first fails on main (5,394.74 there).

• YAML suite: 2,661 passed.

• pytest (code_health, core and top-level tests, excluding the dataset-download tests): 3,187 passed, 2 skipped, 2 xfailed.

• ruff format and ruff check are clean, and the Towncrier draft renders the fragment under "Changed".

Sources

• HMRC, Second estimate of the VAT gap (tax year 2024 to 2025), 3 March 2026.
• HMRC, Measuring tax gaps methodological annex: VAT.
• OBR, Economic and fiscal outlook, March 2026, detailed forecast tables (receipts), table 3.8.
• HMRC, Direct effects of illustrative tax changes bulletin (June 2025).

axiom: n/a: statistical calibration of the VAT grossing factors to OBR receipts; no VAT rate, base or rule changes. rulespec-uk's uk/policies/govuk/vat.yaml takes the coverage factor as a supplied simulation input, and the deprecated microdata_vat_coverage path still works.

🤖 Generated with Claude Code

[juaristi22]

Review of #2187 — Split VAT grossing into household share and survey coverage

Base repository: PolicyEngine/policyengine-uk
PR number: 2187
Reviewed head SHA: eac21df
Merge base SHA: ba06cec
Mode: incremental from c30a6b1
Scope: changed behavior and affected dependencies
Source evidence: Official sources are linked below; local source copies and validation artifacts were retained.
Review status: PARTIAL

Source Documents

• policy-source-vat-gap.html.
• policy-source-vat-method.html.
• policy-source-vat-ready.html.

Critical

None.

Should Address

None.

Suggestions

None.

Evidence Gaps

• Private Microcosm calibration totals and exact cash-basis OBR benchmark remain unreproduced; cross-dataset applicability is an explicit assumption.
• Role checks incomplete: current-pr-2187-review-policy.md; see recovered report.

Notes

• Incremental from c30a6b1; unaffected code validation retained. Private-dataset calibration gap belongs to policy role.
• Incremental delta retains 0.7 and 0.67 factors and restores deprecated combined-factor reform compatibility; no new policy value change.
• Captured base branch: main. Actual fetched base SHA: f1a9a3c. Incremental scope contains the delta from the earlier reviewed head.

Validation Summary

8 passed, 1 warning in 3.92s, exit 0, bounded elapsed 8.75s. Existing Python 3.13 with exact snapshot PYTHONPATH and bytecode disabled; run_bounded.py --seconds 540 review-code-test-runner.py current-pr-2187-code-tests-config.json. Exact selected paths and pytest cache in config/runner; output current-pr-2187-code-tests.log.

Captured incremental diff read. Three same-worktree HMRC originals/extracts checksum verified; current-pr-2187-review-sources.json and current-pr-2187-review-source-cache-check.json. No repeated blocked searches. Code behavior checked by uk_code.

CI at eac21df: Install + smoke-import (py3.11)=SUCCESS, Test documentation builds=SUCCESS, Install + smoke-import (py3.12)=SUCCESS, Install + smoke-import (py3.13)=SUCCESS, Install + smoke-import (py3.14)=SUCCESS, Lint=SUCCESS, Test=SUCCESS.

Detailed code, policy, source and reproduction artifacts were retained locally.

Timing

elapsed seconds: 1151.70s; setup seconds: 270.63s

Review Severity

COMMENT. Open findings: 0 critical, 0 should address, 0 suggestions.

[vahid-ahmadi]

Rebased onto main (3381500f4). New head is 691267f5c.

• Conflict: only myst.yml. Main now lists policy pages by folder (Pick up new documentation pages by folder #2194), so I kept that and added the single docs/engineering/vat-coverage.md entry, since docs/engineering isn't matched by a pattern. The diff against main is otherwise unchanged (12 files, +275/−7).
• Tests (clean venv, policyengine-core 3.33.2): ruff format --check and ruff check clean; VAT pytest (test_vat_grossing_legacy, test_historic_vat_rates) 5 passed; gov/hmrc/vat YAML 4 passed; full YAML policy suite 3,062 passed, 0 failed (main: 3,058).

@juaristi22 your 8 Oct review had no blockers. Could you approve when you have a moment? #2189 is stacked on this and rebased too.

🤖 Generated with Claude Code

[juaristi22]

Re-reviewed 691267f5cce6dc755fc51649ec6b5c8421ecc35d. The rebase changed only the documentation navigation entry; VAT code, parameters and tests match the prior reviewed patch. Eight focused VAT tests pass, and the documentation build passed. I verified the OBR cash-basis VAT forecast and HMRC household-share/ready-reckoner inputs.

The remaining material evidence gap is the private Microcosm and Enhanced FRS pre-scaling VAT totals behind the 0.67 survey-coverage calibration and its use across datasets. Please document the inputs and calculation sufficiently to reproduce or bound that factor, including which datasets it applies to. I cannot approve that quantitative assumption yet.

Please also update the PR description: it says the legacy combined factor was removed, while this head retains it as deprecated, and it needs the required axiom: line (for example, a specific n/a reason if this is purely statistical calibration). The remaining current-head code workflow jobs were pending at review time.

[vahid-ahmadi]

@juaristi22 thanks. Pushed 4054bf17c.

• (a) docs/engineering/vat-coverage.md now gives the inputs (dataset revisions and SHA-256 hashes, OBR table 3.8 receipts, HMRC household share and ready reckoner), the formula, per-year results per dataset and a bound. I reran it: Microcosm UK 2024-25 (f9d1922c) reproduces £80.48bn and 0.672 exactly. The factor applies to that release only. On Microcosm the product lies between 0.459 and 0.481 (coverage 0.655 to 0.688), and 0.67 sits inside that. One correction: the old Enhanced FRS figure (£111.0bn, 0.927) was uk-data 1.56.16. On 1.58.0 it is £115.72bn (0.967), so 0.67 puts 2024-25 VAT 44% above receipts there, not 38%. Both releases are now labelled. The datasets are private, so the aggregates are in the doc for anyone without access.
• (b) The description now says microdata_vat_coverage is kept as a deprecated product (0.469), which overrides the two factors only when a reform sets it.
• (c) Added axiom: n/a: statistical calibration only; no VAT rate, base or rule changes.

Ruff is clean, and the VAT pytest and gov/hmrc/vat YAML tests pass. #2189 is rebased onto this head.