Write policyengine-uk's alcohol, tobacco and gambling duty inputs from the LCFS stage

changelog.d/1359-uk-excise-quantities.added.md

@@ -0,0 +1,13 @@
+The UK LCFS stage now writes policyengine-uk's excise inputs, so alcohol, tobacco and gambling duty no longer score zero on Microcosm data (#1359). The chain draws five shares on the donor, appended after its twenty targets so every existing draw is unchanged: tobacco within P602's off-trade lines, on-trade alcohol within P611, gambling stakes within P609, and within the stakes the National Lottery's share and the betting and gaming share of the rest. The new `level_excise_quantities` step then shares FY2024-25 national figures out to households in proportion to that spending, at prior weights:
+- HMRC's alcohol clearances by product over the engine's strengths, with beer and cider draught shares from the draught-relief clearances;
+- HMRC's tobacco clearances;
+- each gambling duty's dutiable base: HMRC's receipts, or Machine Games Duty's liabilities, over the engine's rate. Lottery Duty's base follows the National Lottery stakes and every other base the betting and gaming stakes, so the society and Irish lotteries carry none. General Betting Duty's base is split by the Gambling Commission's gross gambling yield.
+
+No spending column moves, the VAT base included. `household_tax` rises by the three duties, about £24.8bn at FY2024-25 (£13.06bn alcohol, £8.17bn tobacco and £3.62bn gambling), so `household_net_income` falls by the same, and with it every net-income, decile and inequality output on the data; HBAI incomes do not move. The levels are national while spending and incidence stay at diary level, so a household's alcohol or tobacco duty is one multiple of its own spending: about 2.2 times its tobacco spending, against 0.41 times its alcohol spending.
+
+The gambling inputs follow keys that the new `top_up_gambling_participation` step brings to the Gambling Commission's Gambling Survey for Great Britain 2025, since the diary records a stake for about a fifth of the households that play. 62.7% of households now carry Lottery Duty and 29.8% the other gambling duties, against 10.7% and 2.0% on the diary's draws:
+- each adult's chance of playing in four weeks and in twelve months is GSGB's rate for its sex and age band, and a household's combines its adults at a theta fitted on the donor (0.9 for the National Lottery, 0.08 for betting and gaming);
+- within nine household strata, households without a stake are added in an order a donor propensity sets, until each stratum's four-week and then twelve-month share is met;
+- an added household's stake is drawn from the donor's positive stakes, and an occasional player's is a tenth of that. Households the diary saw keep their stakes, and no spending column moves.
+
+The stage notes give these and the other known biases: fewer households carry tobacco duty than smoke, other tobacco includes e-cigarette refills, the draught shares are national, alcohol volume follows spending, which leans it towards pub drinkers, and the top-up assumes the diary misses gamblers evenly and takes GSGB's participation as the level, though the Commission found that its estimates move with how the survey invitation is worded. The gambling inputs need policyengine-uk 2.151.0 or later. The new stage-health gates `uk_stage_lcfs_consumption_excise_quantities` and `uk_stage_lcfs_consumption_gambling_participation` recompute every level and rate from the vendored `excise_facts.json` and `gambling_participation_facts.json`.

changelog.d/1359-uk-feed-repin-200bf6d.changed.md

@@ -0,0 +1,8 @@
+The UK Chronicle feed is re-pinned to `200bf6d` (PolicyEngine/chronicle#384), which adds the excise facts for #1359 and replaces the June 2026 ONS Consumer Trends edition with September's.
+
+The September revisions move these UK build values:
+- **Household road fuel.** Calendar-2024 road fuel (COICOP 07.2.2) falls from £37.025bn to £35.460bn (−4.2%). This figure both sets the LCFS stage's petrol and diesel level and binds `ons.household_road_fuel_expenditure`, so household road-fuel spending and fuel duty fall with it.
+- **Measured diagnostics.** The calendar-2025 electricity row moves from £26.5bn to £28.8bn and the gas row from £15.3bn to £19.0bn.
+- **Report-only receipts.** The division capture and the uncarried liquid and solid fuels take the September values.
+
+No other compiled target moves on either surface.

changelog.d/1359-uk-feed-repin-272c987.changed.md

@@ -0,0 +1 @@
+The UK Chronicle feed is re-pinned to `272c987` (PolicyEngine/chronicle#391), which adds the Gambling Commission's Gambling Survey for Great Britain 2025 participation tables for #1359's gambling top-up. The new vendored `gambling_participation_facts.json` carries the 426 Great Britain participation rows the LCFS stage reads. No compiled target moves on either surface.

changelog.d/1383-withholding-review-followups.fixed.md

@@ -0,0 +1 @@
+The UK withholding check refuses a local row that names no contract target, where it used to leave the row out of every family block. A contract test keeps each parentless local family free of a higher-grain row of its measurement (Vahid's third review of #1383, item 33). A build now computes the withheld rows once. The national path's diagnostics take the national gates' set, and the Orrery result summary takes the set that a gate on the same result recorded. A local surface's withholding check logs how long it took (item 32).

docs/evidence/uk-excise-quantities-1359/README.md

@@ -0,0 +1,96 @@
+# Excise quantities: measurement scripts and outputs
+
+Evidence for microcosm#1359, behind `experiments/1359-excise-quantities-receipts.md`. `receipts.json` holds the digests of the files here and of the spine the stage runs used.
+
+## What was measured
+
+- **Stage-only runs** (`scripts/stage_dev_run.py`, `scripts/run_stage_dev.sh`). The `lcfs_consumption` stage was re-run on a finished spine: microcosm#1333's `spine-b145-treat.h5`, 78,440 households. Both runs used policyengine-uk 2.151.0, and every LCFS stage-health gate ran on each run's receipts. The two runs:
+  - `outputs/stage_dev_run.base-74a2a378e.json`: microcosm#1383's head `74a2a378e`, the branch base, on the `abf0921` feed;
+  - `outputs/stage_dev_run.head-restack.json`: the branch head on the `200bf6d` feed, with the excise step declared.
+
+  `outputs/stage_dev_run.comparison.json` (`scripts/compare_runs.py`) lists which of the base's 22 columns the head leaves byte-identical (13), and each other column's prior-weighted total in both runs. The other nine move through the re-pin, not the step:
+  - petrol and diesel, by exactly the Consumer Trends revision (−4.227%);
+  - the seven divisions #1383 holds at the shared frame-to-donor ratio, which the lower road fuel lowers (−0.149% each).
+
+  The head adds the 24 excise inputs.
+
+  A third run, `outputs/stage_dev_run.head-split.json`, is the head once the gambling stakes are split (Vahid's review): Lottery Duty's base follows the National Lottery stakes and every other gambling base the betting and gaming stakes. `outputs/stage_dev_run.split-comparison.json` sets it against the restacked head: 33 of the 46 columns are byte-identical, and only the 13 gambling inputs move. The run also records the share of households with each written column positive, at prior weights.
+
+  A fourth run, `outputs/stage_dev_run.head-main-rebase.json`, is the head rebased onto main after microcosm#1383 merged. `outputs/stage_dev_run.rebase-comparison.json` sets it against the split head: 26 of the 46 columns are byte-identical. #1383's last levelling rule moves the seven levelled divisions, recreation among them, and the 13 gambling inputs follow recreation; the alcohol and tobacco inputs do not move.
+- **The national reconciliation** (`scripts/national_reconciliation.py`, `outputs/national_reconciliation.json`). One household holds every national level the committed declaration resolves to, run through policyengine-uk 2.151.0. Every duty is linear in its inputs, so this is the frame's duty at prior weights. Each OBR row is read through its own binding's variables against the value the national compile-parity receipt records from the pinned feed. Public figures only.
+- **Who carries the duty** (`scripts/excise_incidence.py`, `outputs/excise_incidence.json`). From the rebased run, the reconciliation and the LCFS 2024-25 donor:
+  - each duty over the spending that shares it out, and the share of households that carry it;
+  - the share of donors and of frame households with each parent total and each part positive, and each part's rate within its parent, which shows where the frame loses incidence.
+
+  Every donor share rests on 30 or more donors, which the script checks.
+- **The pub lean of the spending rule** (`scripts/alcohol_channel_weights.py`, `outputs/alcohol_channel_weights.json`). Measured on the LCFS 2024-25 donor at survey weights: how far allocating alcohol volume by spending moves it, against channel weights for published on-trade volume shares of 20 to 35 percent, overall and by equivalised income decile.
+- **The re-pin** (`scripts/repin_200bf6d.sh`). The commands that regenerated the vendored resources, the national and local references and membership, the local census, both compile-parity receipts and the coverage manifest. They ran on the national and local cuts of the pinned artifact that the runbook entry describes.
+
+## The gambling participation top-up
+
+- **Theta** (`scripts/theta_calibration.py`, `outputs/theta_calibration.json`). How a household's adults combine, fitted on the LCFS 2024-25 donor. The script reads the committed declaration and the vendored GSGB rows, and sets theta so that the step's adult rates, applied to the donors' own adults, give the diary's ratio of the two-adult to the one-adult household rate. The results are 0.897 for the National Lottery and 0.081 for betting and gaming; the declaration rounds them to 0.9 and 0.08. The ratio for three or more adults is the check, and it fits less well. Each household-size class rests on 30 or more donors with a stake, which the script checks.
+- **Stage-only runs with and without the top-up** (`scripts/stage_run_gsgb.py`, `outputs/stage_run_gsgb.json`). The branch head `348d1209b` ran on the same spine, on the `272c987` feed, twice in one process: as declared, and with `top_up_gambling_participation` removed. The difference is therefore the step's alone. The output records:
+  - every LCFS gate on the declared run (all nine pass) and the participation receipt;
+  - for the Lottery Duty base and one betting and gaming input, the share of households carrying it and each equivalised-income decile's share of households and of the base, in both runs;
+  - by region, the share carrying each key, beside GSGB's regional adult rates (tables B.2 and B.4). This is the out-of-sample check, since region is not a target.
+
+  Every stratum lands on its share. 62.7% of households carry Lottery Duty and 29.8% the other gambling duties, against 10.7% and 2.0% without the step. The added regular players' mean stake equals the seen households', and the occasional players' is about a tenth of it.
+
+  Two results are left for review:
+  - carrying rises with income, so the betting and gaming base moves towards the top decile;
+  - the regional pattern follows GSGB only weakly. London is the lowest region for the Lottery, as in GSGB, but the second lowest for betting and gaming.
+- **The propensity's features** (`scripts/stage_run_gsgb_variants.py`, `outputs/stage_run_gsgb_variants.json`). Vahid's round 4 found that the propensity read recreation with the stakes in it. The script runs the stage once as committed, then the top-up alone on the same inputs in three ways:
+  - recreation with the stakes in, as at `3a20860fb`. It reproduces `stage_run_gsgb.json` exactly;
+  - recreation net of the stakes, as committed;
+  - the walk ordered by the published chance alone.
+
+  Taking the stakes out flattens part of the income gradient. Betting and gaming now runs from 26.8% of households in the bottom equivalised-income decile to 34.8% in the top, against 24.5% to 36.2% with the stakes in. The Lottery runs from 51.5% to 67.3%, against 51.6% to 70.3%. Ordered by the chance alone, betting and gaming is flat (29.2% to 29.8%), but neither key's regional pattern then follows GSGB's. With the stakes out, the Lottery's correlation with GSGB's regional rates rises from 0.41 to 0.67, and betting and gaming's stays at 0.11. The stake scales hardly move (1.82 to 1.86 for the Lottery, 0.63 to 0.64 for betting and gaming), so the stake in the feature did not cause the raw draws' gap from the seen households' mean.
+- **The union and theta** (`scripts/union_theta_sensitivity.py`, `outputs/union_theta_sensitivity.json`). Vahid's round 4 asked how much the keys rest on two choices: activities independent within their population, and theta. GSGB's own tables check the first, by age band:
+  - B.13, the number of activities each four-week gambler took part in, rules out independence among players. 45% of them took part in one activity, against 26% under independence.
+  - A model in which each adult's activities share a gamma-distributed intensity comes within 2.5 points of B.13's one-activity share in every age band. It moves neither key's union by more than 9% in any age band.
+  - Where B.9 prints a union beside its parts, independence gets the two kinds of lottery draw right (36.7% against 36.5% of adults). It overstates one product from two sellers (scratchcards, online instant win games) by 13 to 15%.
+
+  On the frame's adults at prior weights, the share of households carrying each key is then computed at other thetas, with the model's unions, and at the bounds the published rows allow. Theta matters more than the union: from theta 0 to 1, betting and gaming runs from 28.5% to 41.2% of households (29.8% at the fitted 0.08), and the Lottery from 48.2% to 63.8% (62.7% at 0.9). The model's unions move neither key by more than 0.1 points.
+- **The re-pin to `272c987`** (`scripts/repin_272c987.sh`, with `scripts/quiet_footprint.sh`). The same steps as for `200bf6d`, on cuts of the artifact built from PolicyEngine/chronicle#391's head, whose tree the merge commit keeps. Each heavy step waits until no other process holds a 3 GB memory footprint (resident plus compressed and swapped), since `ps`'s resident size misses a swapped-out process.
+
+## What is not here
+
+- **Unit records.** Household dumps, the spine and the LCFS tabs stay out of the repository.
+- **Column digests.** The runs record the sha256 of every column they write. `scripts/sanitize_outputs.py` drops those digests, with any donor minimum or maximum, before an output is copied here; the comparison keeps only the verdicts.
+- **Small cells.** Every count in the receipts is a band ("30+") or withheld ("<10"), so no figure here rests on fewer than ten households.
+
+## Re-running
+
+The stage runs need the licensed LCFS 2024-25 tabs and the spine:
+
+```bash
+BASE_LABEL=base-74a2a378e HEAD_LABEL=head-restack scripts/run_stage_dev.sh
+```
+
+The reconciliation needs only the tree and its locked engine. The incidence script reads the split run, the reconciliation and the LCFS household tab:
+
+```bash
+python scripts/national_reconciliation.py outputs/national_reconciliation.json
+```
+
+```bash
+python scripts/excise_incidence.py outputs/stage_dev_run.head-main-rebase.json outputs/national_reconciliation.json <lcfs dvhh tab> outputs/excise_incidence.json
+```
+
+The top-up's runs need the LCFS tabs, the spine and the GSGB rows taken from the pinned artifact (`grep '"gambling_commission.gsgb' consumer_facts.jsonl`):
+
+```bash
+python scripts/theta_calibration.py <lcfs 2024-25 tab directory> outputs/theta_calibration.json
+```
+
+```bash
+python scripts/stage_run_gsgb.py <tree> <spine.h5> <gsgb rows jsonl> outputs/stage_run_gsgb.json
+```
+
+```bash
+python scripts/stage_run_gsgb_variants.py <tree> <spine.h5> <gsgb rows jsonl> outputs/stage_run_gsgb_variants.json
+```
+
+```bash
+python scripts/union_theta_sensitivity.py <gsgb rows jsonl> <spine.h5> outputs/union_theta_sensitivity.json
+```