Split VAT grossing into household share and survey coverage

changelog.d/1996.changed.md

@@ -0,0 +1 @@
+Split the undated VAT grossing factor `gov.simulation.microdata_vat_coverage` (0.38) into two documented factors: `gov.simulation.vat.household_share_of_receipts` (0.7, HMRC's share of VAT liabilities on household consumption) and `gov.simulation.vat.survey_consumption_coverage` (0.67, derived on Microcosm UK 2024-25 against OBR VAT receipts). Total VAT on that dataset now matches 2024-25 receipts instead of overshooting by 24%. `microdata_vat_coverage` is kept as a deprecated parameter equal to their product (0.469); a reform that sets it to another value still takes effect.

docs/engineering/vat-coverage.md

@@ -0,0 +1,156 @@
+# VAT grossing to receipts
+
+`vat` and `baseline_vat` compute the statutory liability on each household's
+recorded consumption (the standard rate on `full_rate_vat_consumption` plus
+the reduced rate on `reduced_rate_vat_consumption`) and divide it by two
+factors, so that total VAT matches receipts:
+
+- `gov.simulation.vat.household_share_of_receipts` (0.7): the share of VAT
+  liabilities that falls on household final consumption. HMRC's VAT gap
+  estimates put household consumption at around 70% of the VAT total
+  theoretical liability; the rest falls on government, charities, VAT-exempt
+  businesses and new housing. This is a published quantity and does not depend
+  on the dataset.
+- `gov.simulation.vat.survey_consumption_coverage` (0.67): the share of
+  household VAT liability that the dataset's consumption captures. This
+  depends on the dataset's consumption imputation and must be re-derived when
+  the default dataset changes.
+
+The incidence assumption is that every pound of receipts, including VAT that
+government, charities and exempt businesses pay on their inputs, is passed on
+to households in proportion to their own VAT-liable spending.
+
+These two factors replace `gov.simulation.microdata_vat_coverage`, a single
+undated 0.38 added in January 2023 against an older LCFS-based consumption
+imputation, with no derivation recorded. Its product with today's datasets put
+2024-25 VAT 24% above OBR receipts on Microcosm UK 2024-25 and 71% above on
+Enhanced FRS 2024-25 (policyengine-uk-data 1.56.16).
+
+`gov.simulation.microdata_vat_coverage` is kept as a deprecated parameter so
+that existing reforms to it still work. Its value is now the product of the two
+factors (0.469). If a reform sets it to any other value, `vat` and
+`baseline_vat` use that value in place of the two factors
+(`policyengine_uk/utils/vat.py`); otherwise they use the two factors. New
+reforms should change the two factors instead.
+
+## Which datasets the factor applies to
+
+The survey coverage of 0.67 is calibrated to one dataset: the certified
+Microcosm UK 2024-25 national release. It does not carry over to other
+datasets, because their consumption imputations capture a different share of
+household VAT. On Enhanced FRS 2024-25 it puts VAT 35% to 44% above receipts,
+depending on the release and year (see the tables below). Re-derive the coverage, with
+the calculation below, whenever the dataset it should match changes.
+
+## Derivation (October 2026)
+
+### Inputs
+
+| Input | Value | Source |
+|---|---|---|
+| Microcosm UK 2024-25 national release | `microcosm_uk_2024_25.h5`, Hugging Face `policyengine/populace-uk-private`, tag `microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68` (revision `f9d1922c`), SHA-256 `aa31bdf67c977927ea2b325567d1cf7a79d94381239bc79918a0a0fc9c9588af` | Private repository |
+| Enhanced FRS 2024-25 | `enhanced_frs_2024_25.h5`, Hugging Face `policyengine/policyengine-uk-data-private`, policyengine-uk-data 1.56.16 (revision `a9e52499`, SHA-256 `e433e532b17bd8ce76030156285816e33d44e93edabd2204adbef71d19a68712`) and 1.58.0 (revision `47bbb58d`, SHA-256 `5ba399ea5ab9a6c7d4e9aaf6c623080beb34d9ad5f330f6244dbad72b8d960b6`) | Private repository |
+| VAT receipts | £170.994bn (2024-25 outturn), £180.169bn (2025-26), £187.704bn (2026-27) | OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8, cash-basis VAT |
+| Household share of VAT liabilities | 0.7 | HMRC VAT gap estimates (around 70% of the VAT total theoretical liability); `household_share_of_receipts` |
+| Ready reckoner | £8.8bn for one point on the standard rate, 2026-27 | HMRC, Direct effects of illustrative tax changes (June 2025) |
+| Statutory rates | 20% standard, 5% reduced | `gov.hmrc.vat` |
+
+Model year 2024 is the 2024-25 fiscal year. All dataset totals are weighted
+sums from `Microsimulation`, computed with policyengine-uk at this branch and
+policyengine-core 3.33.2.
+
+### Formula
+
+```
+pre-scaling VAT = 0.20 x sum(full_rate_vat_consumption)
+                + 0.05 x sum(reduced_rate_vat_consumption)
+product         = pre-scaling VAT / OBR VAT receipts      (same year)
+survey coverage = product / household share (0.7)
+```
+
+`vat` divides each household's statutory VAT by the product, so total `vat`
+equals receipts when the product matches the dataset.
+
+### Results
+
+| Dataset | Year | Standard-rated base | Reduced-rated base | Pre-scaling VAT | Receipts | Product | Survey coverage |
+|---|---|---|---|---|---|---|---|
+| Microcosm UK 2024-25 (`f9d1922c`) | 2024-25 | £396.15bn | £25.02bn | £80.48bn | £171.0bn | 0.471 | 0.672 |
+| Microcosm UK 2024-25 (`f9d1922c`) | 2025-26 | £412.57bn | £26.06bn | £83.82bn | £180.2bn | 0.465 | 0.665 |
+| Microcosm UK 2024-25 (`f9d1922c`) | 2026-27 | £423.66bn | £26.76bn | £86.07bn | £187.7bn | 0.459 | 0.655 |
+| Enhanced FRS 2024-25, uk-data 1.56.16 | 2024-25 | £546.96bn | £32.21bn | £111.00bn | £171.0bn | 0.649 | 0.927 |
+| Enhanced FRS 2024-25, uk-data 1.56.16 | 2025-26 | £569.63bn | £33.55bn | £115.60bn | £180.2bn | 0.642 | 0.917 |
+| Enhanced FRS 2024-25, uk-data 1.56.16 | 2026-27 | £584.95bn | £34.45bn | £118.71bn | £187.7bn | 0.632 | 0.904 |
+| Enhanced FRS 2024-25, uk-data 1.58.0 | 2024-25 | £570.29bn | £33.17bn | £115.72bn | £171.0bn | 0.677 | 0.967 |
+| Enhanced FRS 2024-25, uk-data 1.58.0 | 2025-26 | £593.92bn | £34.54bn | £120.51bn | £180.2bn | 0.669 | 0.956 |
+| Enhanced FRS 2024-25, uk-data 1.58.0 | 2026-27 | £609.89bn | £35.47bn | £123.75bn | £187.7bn | 0.659 | 0.942 |
+
+The parameter takes the Microcosm 2024-25 value, 0.67 (product 0.469), the
+latest outturn year.
+
+With that product, total `vat` against receipts is:
+
+| Dataset | 2024-25 | 2025-26 | 2026-27 |
+|---|---|---|---|
+| Microcosm UK 2024-25 (`f9d1922c`) | £171.6bn (+0.4%) | £178.7bn (-0.8%) | £183.5bn (-2.2%) |
+| Enhanced FRS 2024-25, uk-data 1.56.16 | £236.7bn (+38.4%) | £246.5bn (+36.8%) | £253.1bn (+34.8%) |
+| Enhanced FRS 2024-25, uk-data 1.58.0 | £246.7bn (+44.3%) | £257.0bn (+42.6%) | £263.9bn (+40.6%) |
+
+### Bound
+
+Total VAT depends only on the product, so the household share being a
+rounded "around 70%" changes how the product is split between the two factors,
+not the totals. On Microcosm the product is bounded by:
+
+- the receipts route across 2024-25 to 2026-27: 0.459 to 0.471 (coverage 0.655
+  to 0.672);
+- the ready reckoner, which does not use the level of receipts: one point on
+  the standard rate raises £4.24bn before scaling in 2026-27 (1% of
+  £423.66bn), so matching HMRC's £8.8bn needs a product of 0.481 (coverage
+  0.688).
+
+So on Microcosm the product lies between 0.459 and 0.481, and the coverage
+between 0.655 and 0.688 at a household share of 0.7. The value used, 0.469
+(0.67), sits inside that range, and the totals it gives are within 2.2% of
+receipts from 2024-25 to 2026-27. The implied product moves by less than 3%
+over those years, so the factors are not dated by year.
+
+### Reproducing
+
+The two datasets are in private Hugging Face repositories. Anyone with
+PolicyEngine access can rerun the calculation; anyone else can check the
+arithmetic from the published aggregates in the results table.
+
+```python
+from huggingface_hub import hf_hub_download
+from policyengine_uk import Microsimulation
+
+path = hf_hub_download(
+    "policyengine/populace-uk-private",
+    "microcosm_uk_2024_25.h5",
+    repo_type="dataset",
+    revision="f9d1922cddab6b54a0dd37794a9bac74e3780c88",
+)
+sim = Microsimulation(dataset=path)
+# OBR March 2026 EFO, table 3.8, cash-basis VAT receipts
+receipts = {2024: 170.994e9, 2025: 180.169e9, 2026: 187.704e9}
+for year, obr in receipts.items():
+    p = sim.tax_benefit_system.parameters(f"{year}-01-01").gov.hmrc.vat
+    full = sim.calculate("full_rate_vat_consumption", year).sum()
+    reduced = sim.calculate("reduced_rate_vat_consumption", year).sum()
+    raw = full * p.standard_rate + reduced * p.reduced_rate
+    print(year, raw / 1e9, raw / obr, raw / obr / 0.7)
+```
+
+For Enhanced FRS, download `enhanced_frs_2024_25.h5` from the model repository
+`policyengine/policyengine-uk-data-private` at the release tag (for example
+`revision="1.58.0"`).
+
+## Interaction with an explicit energy VAT base
+
+A domestic fuel and power VAT base built from `electricity_consumption` and
+`gas_consumption` (PolicyEngine/policyengine-uk#2166) changes what the survey
+coverage applies to. If those energy inputs are already calibrated to an
+aggregate, they should not be divided by the survey coverage again, and the
+reconciled energy part of `reduced_rate_vat_consumption` should come out of the
+base used to derive it.

myst.yml

@@ -33,6 +33,7 @@ project:
       children:
       - file: docs/engineering/excise-duties.md
       - file: docs/engineering/bus-fares.md
+      - file: docs/engineering/vat-coverage.md
       - pattern: docs/book/policy/*.md
     - title: Programmes
       children:

policyengine_uk/parameters/gov/simulation/microdata_vat_coverage.yaml

@@ -1,9 +1,11 @@
-description: LCFS (from which ETB data is derived) microdata is known to under-report household consumption. We scale up the VAT liability to hit HMRC-reported VAT receipts (following the approach of IFS' TAXBEN, though they might have better coverage anyway from access to the non-EUL dataset). For HMRC statistics see https://www.gov.uk/government/statistics/value-added-tax-vat-annual-statistics.
+description: Deprecated. Combined VAT grossing factor, kept so that existing reforms to this path still work. Its value is gov.simulation.vat.survey_consumption_coverage times gov.simulation.vat.household_share_of_receipts (0.67 x 0.7). If a reform sets it to a different value, vat uses it in place of the two factors; otherwise vat uses the two factors. New reforms should change the two factors instead.
 metadata:
   unit: /1
   period: year
+  economy: false
+  deprecated: true
   reference:
-    - title: Value Added Tax (VAT) annual statistics
-      href: https://www.gov.uk/government/statistics/value-added-tax-vat-annual-statistics
+    - title: docs/engineering/vat-coverage.md (derivation of the two factors this combines)
+      href: https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/engineering/vat-coverage.md
 values:
-  2010-01-01: 0.38
+  2010-01-01: 0.469

policyengine_uk/parameters/gov/simulation/vat/household_share_of_receipts.yaml

@@ -0,0 +1,12 @@
+description: Share of VAT liabilities that falls on household final consumption. The rest falls on government, charities, VAT-exempt businesses and new housing. Household VAT is grossed up by this share so that total VAT matches receipts.
+metadata:
+  unit: /1
+  period: year
+  economy: false
+  reference:
+    - title: HMRC, Second estimate of the VAT gap (tax year 2024 to 2025), 3 March 2026 (household consumption is around 70% of the VAT total theoretical liability)
+      href: https://www.gov.uk/government/statistics/vat-gap-estimates/second-estimate-of-the-vat-gap-tax-year-2023-to-2024
+    - title: HMRC, Measuring tax gaps methodological annex, VAT (components of the VAT total theoretical liability)
+      href: https://www.gov.uk/government/statistics/measuring-tax-gaps-methodological-annex/2-tax-gaps-value-added-tax
+values:
+  2010-01-01: 0.7

policyengine_uk/parameters/gov/simulation/vat/survey_consumption_coverage.yaml

@@ -0,0 +1,29 @@
+description: Share of household VAT liability captured by consumption in the microdata. Household VAT computed from survey consumption is divided by this share and by the household share of VAT liabilities to match receipts.
+metadata:
+  unit: /1
+  period: year
+  economy: false
+  derivation: >-
+    Calibrated to the certified Microcosm UK 2024-25 national release only
+    (policyengine/populace-uk-private, tag
+    microcosm-uk-2024-25-national-20261002T230158Z-5c6b3f68, revision
+    f9d1922c). Pre-scaling VAT (standard rate times full_rate_vat_consumption
+    plus reduced rate times reduced_rate_vat_consumption) on that release is
+    0.20 x £396.15bn + 0.05 x £25.02bn = £80.48bn in 2024. OBR VAT receipts
+    for 2024-25 (outturn, March 2026 EFO table 3.8, cash basis) are £170.994bn.
+    80.48 / 170.994 = 0.471 for the product of the two factors; divided by the
+    household share of 0.7 gives 0.67. Across 2024-25 to 2026-27 the product is
+    0.459 to 0.471, and the HMRC ready reckoner (£8.8bn for one point on the
+    standard rate in 2026-27) implies 0.481, so the coverage lies between 0.655
+    and 0.688. On Enhanced FRS 2024-25 the same calculation gives a coverage of
+    0.927 (policyengine-uk-data 1.56.16) or 0.967 (1.58.0), so this value puts
+    VAT on that dataset 38% or 44% above 2024-25 receipts. Re-derive when the
+    dataset it should match changes, with docs/engineering/vat-coverage.md,
+    which lists the inputs, file hashes and per-dataset results.
+  reference:
+    - title: OBR, Economic and fiscal outlook March 2026, detailed forecast tables, receipts, table 3.8
+      href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/
+    - title: HMRC, Direct effects of illustrative tax changes bulletin (June 2025)
+      href: https://www.gov.uk/government/statistics/direct-effects-of-illustrative-tax-changes/direct-effects-of-illustrative-tax-changes-bulletin-january-2025
+values:
+  2010-01-01: 0.67

policyengine_uk/tests/microsimulation/reforms_config.yaml

@@ -29,7 +29,7 @@ reforms:
   parameters:
     gov.hmrc.national_insurance.class_1.rates.employee.main: 0.1
 - name: Raise VAT standard rate by 2pp
-  expected_impact: 21.3
+  expected_impact: 17.3
   tolerance: 1.5
   parameters:
     gov.hmrc.vat.standard_rate: 0.22

policyengine_uk/tests/policy/baseline/gov/hmrc/vat/vat_grossing.yaml

@@ -0,0 +1,43 @@
+- name: VAT is grossed up by survey coverage times the household share of liabilities
+  period: 2025
+  absolute_error_margin: 0.01
+  input:
+    full_rate_vat_consumption: 10_000
+    reduced_rate_vat_consumption: 1_000
+  output:
+    # (10,000 x 20% + 1,000 x 5%) / (0.67 x 0.7) = 2,050 / 0.469
+    vat: 4_371.00
+    baseline_vat: 4_371.00
+
+- name: With both factors at one, VAT is the statutory liability on recorded consumption
+  period: 2025
+  absolute_error_margin: 0.01
+  input:
+    gov.simulation.vat.survey_consumption_coverage: 1
+    gov.simulation.vat.household_share_of_receipts: 1
+    full_rate_vat_consumption: 10_000
+    reduced_rate_vat_consumption: 1_000
+  output:
+    vat: 2_050
+
+- name: The household share alone grosses survey-complete VAT up to receipts
+  period: 2025
+  absolute_error_margin: 0.01
+  input:
+    gov.simulation.vat.survey_consumption_coverage: 1
+    full_rate_vat_consumption: 7_000
+    reduced_rate_vat_consumption: 0
+  output:
+    # 7,000 x 20% / 0.7
+    vat: 2_000
+
+- name: A reform to the deprecated combined factor still takes effect
+  period: 2025
+  absolute_error_margin: 0.01
+  input:
+    gov.simulation.microdata_vat_coverage: 0.5
+    full_rate_vat_consumption: 10_000
+    reduced_rate_vat_consumption: 1_000
+  output:
+    # 2,050 / 0.5
+    vat: 4_100

policyengine_uk/tests/test_vat_grossing_legacy.py

@@ -0,0 +1,56 @@
+import pytest
+from policyengine_core.reforms import Reform
+
+from policyengine_uk import Simulation
+from policyengine_uk.model_api import Scenario
+from policyengine_uk.system import system
+from policyengine_uk.utils.vat import LEGACY_COVERAGE_DEFAULT
+
+LEGACY = "gov.simulation.microdata_vat_coverage"
+SITUATION = {
+    "people": {"adult": {"age": {2025: 40}}},
+    "benunits": {"benunit": {"members": ["adult"]}},
+    "households": {
+        "household": {
+            "members": ["adult"],
+            "full_rate_vat_consumption": {2025: 10_000},
+            "reduced_rate_vat_consumption": {2025: 1_000},
+        }
+    },
+}
+RAW_VAT = 10_000 * 0.2 + 1_000 * 0.05
+
+
+def test_legacy_default_is_the_product_of_the_two_factors():
+    p = system.parameters.gov.simulation
+    for year in range(2015, 2031):
+        product = p.vat.survey_consumption_coverage(
+            year
+        ) * p.vat.household_share_of_receipts(year)
+        assert p.microdata_vat_coverage(year) == pytest.approx(product)
+    assert LEGACY_COVERAGE_DEFAULT == pytest.approx(0.469)
+
+
+def test_reform_dict_to_legacy_path_changes_vat():
+    reform = Reform.from_dict({LEGACY: {"2010-01-01.2100-12-31": 0.5}}, "uk")
+    sim = Simulation(situation=SITUATION, reform=reform)
+    assert sim.calculate("vat", 2025)[0] == pytest.approx(RAW_VAT / 0.5)
+
+
+def test_scenario_to_legacy_path_changes_vat():
+    scenario = Scenario(parameter_changes={LEGACY: {"year:2025:1": 0.5}})
+    sim = Simulation(situation=SITUATION, scenario=scenario)
+    assert sim.calculate("vat", 2025)[0] == pytest.approx(RAW_VAT / 0.5)
+
+
+def test_untouched_legacy_path_uses_the_two_factors():
+    reform = Reform.from_dict(
+        {
+            "gov.simulation.vat.survey_consumption_coverage": {
+                "2010-01-01.2100-12-31": 1
+            }
+        },
+        "uk",
+    )
+    sim = Simulation(situation=SITUATION, reform=reform)
+    assert sim.calculate("vat", 2025)[0] == pytest.approx(RAW_VAT / 0.7)

policyengine_uk/utils/vat.py

@@ -0,0 +1,34 @@
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+_LEGACY_PATH = (
+    Path(__file__).parents[1]
+    / "parameters"
+    / "gov"
+    / "simulation"
+    / "microdata_vat_coverage.yaml"
+)
+# The declared value of the deprecated combined factor. A reform that moves the
+# parameter away from it is honoured; otherwise the two factors are used.
+LEGACY_COVERAGE_DEFAULT = float(
+    list(yaml.safe_load(_LEGACY_PATH.read_text())["values"].values())[-1]
+)
+
+
+def vat_grossing_factor(simulation_parameters):
+    """Factor that household VAT on recorded consumption is divided by.
+
+    The product of survey consumption coverage and the household share of VAT
+    liabilities, unless a reform has set the deprecated combined parameter
+    `microdata_vat_coverage` to another value, in which case that value is used
+    so that existing reforms keep working.
+    """
+    legacy = simulation_parameters.microdata_vat_coverage
+    if not np.isclose(legacy, LEGACY_COVERAGE_DEFAULT):
+        return legacy
+    return (
+        simulation_parameters.vat.survey_consumption_coverage
+        * simulation_parameters.vat.household_share_of_receipts
+    )

policyengine_uk/variables/gov/hmrc/baseline_vat.py

@@ -1,4 +1,5 @@
 from policyengine_uk.model_api import *
+from policyengine_uk.utils.vat import vat_grossing_factor
 
 
 class baseline_vat(Variable):
@@ -16,4 +17,7 @@ def formula(household, period, parameters):
             full_rate_consumption * p.hmrc.vat.standard_rate
             + reduced_rate_consumption * p.hmrc.vat.reduced_rate
         )
-        return raw_vat / p.simulation.microdata_vat_coverage
+        # Survey consumption under-records spending, and households bear only
+        # part of VAT liabilities, so household VAT is grossed up to receipts.
+        coverage = vat_grossing_factor(p.simulation)
+        return raw_vat / coverage