Calibrate the QRF sign gates out of fold, starting with the LCFS chain

changelog.d/1359-lcfs-cross-fitted-gates.changed.md

@@ -0,0 +1,18 @@
+The LCFS consumption chain cross-fits its sign gates (`gate_calibration: cross_fitted` on its `fit_weighted_qrf_chain`). The in-sample gates had never seen the frame's households, and drew rare spending on them too rarely; now each spending is drawn at its donor rate.
+
+On the microcosm#1333 test spine, at prior weights, the share of households with each spending goes as follows (LCFS donors in brackets):
+- solid fuels: 0.3% → 2.2% (2.5%);
+- liquid fuels: 3.1% → 3.9% (4.5%);
+- education: 4.3% → 6.7% (5.8%);
+- tobacco: 8.2% → 11.2% (10.3%);
+- gambling stakes: 14.6% → 16.7% (17.0%);
+- betting and gaming: 2.0% → 2.6% (3.6%).
+
+The near-universal divisions' zeros are now drawn at their rate too. The twelve divisions' prior-weighted total falls 1.4%, and the seven divisions held at the shared frame-to-donor ratio fall with it.
+
+The excise inputs keep their national levels, but the multiples of spending change:
+- tobacco duty is about 1.8 times the tobacco spending of the households that carry it, down from 2.2;
+- the other gambling duties are 5.5 times betting and gaming stakes, down from 6.4;
+- Lottery Duty is 0.61 times National Lottery stakes, up from 0.56.
+
+Every LCFS stage-health gate passes.

changelog.d/1359-qrf-gate-calibration.added.md

@@ -0,0 +1,5 @@
+`RegimeGatedQRF` takes a `gate_calibration`, named by the new `GateCalibration` constants. The default, `in_sample`, is the sign gate as before, and its draws do not move. The gate is one boosted classifier, calibrated only on its own training rows. It memorises a rare class's training rows, so on every other row, every recipient included, it draws that class too rarely.
+
+`cross_fitted` fits a gate on each of five folds of the training rows, stratified by class. One weighted calibration (Platt scaling, for two classes) is fitted on the gates' out-of-fold scores and maps each of them, and a row's probability is the mean of the mapped gates. A rare class is then drawn at its rate on rows the gates were not fitted to.
+
+The forests and the draw uniforms are unchanged. A class with fewer rows than folds keeps the in-sample gate. A chain checkpoint names the calibration only when it is cross-fitted, so in-sample checkpoints are unchanged.

changelog.d/1359-uk-excise-quantities.added.md

@@ -0,0 +1,6 @@
+The UK LCFS stage now writes policyengine-uk's excise inputs, so alcohol, tobacco and gambling duty no longer score zero on Microcosm data (#1359). The chain draws five shares on the donor, appended after its twenty targets so every existing draw is unchanged: tobacco within P602's off-trade lines, on-trade alcohol within P611, gambling stakes within P609, and within the stakes the National Lottery's share and the betting and gaming share of the rest. The new `level_excise_quantities` step then shares FY2024-25 national figures out to households in proportion to that spending, at prior weights:
+- HMRC's alcohol clearances by product over the engine's strengths, with beer and cider draught shares from the draught-relief clearances;
+- HMRC's tobacco clearances;
+- each gambling duty's dutiable base: HMRC's receipts, or Machine Games Duty's liabilities, over the engine's rate. Lottery Duty's base follows the National Lottery stakes and every other base the betting and gaming stakes, so the society and Irish lotteries carry none. General Betting Duty's base is split by the Gambling Commission's gross gambling yield.
+
+No spending column moves, the VAT base included. `household_tax` rises by the three duties, about £24.8bn at FY2024-25 (£13.06bn alcohol, £8.17bn tobacco and £3.62bn gambling), so `household_net_income` falls by the same, and with it every net-income, decile and inequality output on the data; HBAI incomes do not move. The levels are national while spending and incidence stay at diary level, so a household's duty is one multiple of its own spending: about 2.2 times its tobacco spending, 0.56 times its National Lottery stakes and 6.4 times its betting and gaming stakes, against 0.41 times its alcohol spending. The stage notes give these and the other known biases: fewer households carry each duty than smoke or gamble, other tobacco includes e-cigarette refills, the draught shares are national, and alcohol volume follows spending, which leans it towards pub drinkers. The gambling inputs need policyengine-uk 2.151.0 or later. The new stage-health gate `uk_stage_lcfs_consumption_excise_quantities` recomputes every level from the vendored `excise_facts.json`.

changelog.d/1359-uk-feed-repin-200bf6d.changed.md

@@ -0,0 +1,8 @@
+The UK Chronicle feed is re-pinned to `200bf6d` (PolicyEngine/chronicle#384), which adds the excise facts for #1359 and replaces the June 2026 ONS Consumer Trends edition with September's.
+
+The September revisions move these UK build values:
+- **Household road fuel.** Calendar-2024 road fuel (COICOP 07.2.2) falls from £37.025bn to £35.460bn (−4.2%). This figure both sets the LCFS stage's petrol and diesel level and binds `ons.household_road_fuel_expenditure`, so household road-fuel spending and fuel duty fall with it.
+- **Measured diagnostics.** The calendar-2025 electricity row moves from £26.5bn to £28.8bn and the gas row from £15.3bn to £19.0bn.
+- **Report-only receipts.** The division capture and the uncarried liquid and solid fuels take the September values.
+
+No other compiled target moves on either surface.

docs/evidence/spec-engine/us-f0-coverage.json

@@ -1656,13 +1656,13 @@
           "compiler_ir.node_slices"
         ],
         "expected": {
-          "map_sha256": "0db599df419dd16779097361dc72afdefb8df2abf8b134078da9c52d31d8e017",
-          "protocol_sha256": "2b7bdc1d6b0da1bdaf54b4cac29f10f2d3c8bed2e177729c2c89e6b33a099cc5"
+          "map_sha256": "54223ee7d55283c8b2c6127ecece8ba6e76b8da0b72946b3cac88a1ba8fa7883",
+          "protocol_sha256": "72d918a0cda5ccc95dee3aaabaa4ad1cbabbf4f282abc5046b49b5f183dc0dc9"
         },
         "failures": [],
         "observed": {
-          "map_sha256": "0db599df419dd16779097361dc72afdefb8df2abf8b134078da9c52d31d8e017",
-          "protocol_sha256": "2b7bdc1d6b0da1bdaf54b4cac29f10f2d3c8bed2e177729c2c89e6b33a099cc5"
+          "map_sha256": "54223ee7d55283c8b2c6127ecece8ba6e76b8da0b72946b3cac88a1ba8fa7883",
+          "protocol_sha256": "72d918a0cda5ccc95dee3aaabaa4ad1cbabbf4f282abc5046b49b5f183dc0dc9"
         },
         "status": "covered"
       },
@@ -1677,7 +1677,7 @@
           "compiler_ir.seed_stream_map"
         ],
         "expected": {
-          "implementation_sha256": "2b7bdc1d6b0da1bdaf54b4cac29f10f2d3c8bed2e177729c2c89e6b33a099cc5",
+          "implementation_sha256": "72d918a0cda5ccc95dee3aaabaa4ad1cbabbf4f282abc5046b49b5f183dc0dc9",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -1698,7 +1698,7 @@
         },
         "failures": [],
         "observed": {
-          "implementation_sha256": "2b7bdc1d6b0da1bdaf54b4cac29f10f2d3c8bed2e177729c2c89e6b33a099cc5",
+          "implementation_sha256": "72d918a0cda5ccc95dee3aaabaa4ad1cbabbf4f282abc5046b49b5f183dc0dc9",
           "protocol": "legacy-v1",
           "streams": [
             "build_model",
@@ -2599,7 +2599,7 @@
       "country": "us",
       "schema_id": "country_spec",
       "schema_version": 1,
-      "spec_sha256": "19460fc168a6f9ed44e580d0e36777cce78bb33cb52b75630ee86203fa795684"
+      "spec_sha256": "08fbb11e05114797f0b8977fd0cfd1f7afb5196f51c9dc4ad4941b8f0aa2154e"
     }
   },
   "report_schema_version": 3,
@@ -2609,7 +2609,7 @@
     "country": "us",
     "schema_id": "country_spec",
     "schema_version": 1,
-    "spec_sha256": "19460fc168a6f9ed44e580d0e36777cce78bb33cb52b75630ee86203fa795684"
+    "spec_sha256": "08fbb11e05114797f0b8977fd0cfd1f7afb5196f51c9dc4ad4941b8f0aa2154e"
   },
   "status": "pass"
 }

docs/evidence/uk-excise-quantities-1359/README.md

@@ -0,0 +1,49 @@
+# Excise quantities: measurement scripts and outputs
+
+Evidence for microcosm#1359, behind `experiments/1359-excise-quantities-receipts.md`. `receipts.json` holds the digests of the files here and of the spine the stage runs used.
+
+## What was measured
+
+- **Stage-only runs** (`scripts/stage_dev_run.py`, `scripts/run_stage_dev.sh`). The `lcfs_consumption` stage was re-run on a finished spine: microcosm#1333's `spine-b145-treat.h5`, 78,440 households. Both runs used policyengine-uk 2.151.0, and every LCFS stage-health gate ran on each run's receipts. The two runs:
+  - `outputs/stage_dev_run.base-74a2a378e.json`: microcosm#1383's head `74a2a378e`, the branch base, on the `abf0921` feed;
+  - `outputs/stage_dev_run.head-restack.json`: the branch head on the `200bf6d` feed, with the excise step declared.
+
+  `outputs/stage_dev_run.comparison.json` (`scripts/compare_runs.py`) lists which of the base's 22 columns the head leaves byte-identical (13), and each other column's prior-weighted total in both runs. The other nine move through the re-pin, not the step:
+  - petrol and diesel, by exactly the Consumer Trends revision (−4.227%);
+  - the seven divisions #1383 holds at the shared frame-to-donor ratio, which the lower road fuel lowers (−0.149% each).
+
+  The head adds the 24 excise inputs.
+
+  A third run, `outputs/stage_dev_run.head-split.json`, is the head once the gambling stakes are split (Vahid's review): Lottery Duty's base follows the National Lottery stakes and every other gambling base the betting and gaming stakes. `outputs/stage_dev_run.split-comparison.json` sets it against the restacked head: 33 of the 46 columns are byte-identical, and only the 13 gambling inputs move. The run also records the share of households with each written column positive, at prior weights.
+- **The national reconciliation** (`scripts/national_reconciliation.py`, `outputs/national_reconciliation.json`). One household holds every national level the committed declaration resolves to, run through policyengine-uk 2.151.0. Every duty is linear in its inputs, so this is the frame's duty at prior weights. Each OBR row is read through its own binding's variables against the value the national compile-parity receipt records from the pinned feed. Public figures only.
+- **Who carries the duty** (`scripts/excise_incidence.py`, `outputs/excise_incidence.json`). From the split run, the reconciliation and the LCFS 2024-25 donor:
+  - each duty over the spending that shares it out, and the share of households that carry it;
+  - the share of donors and of frame households with each parent total and each part positive, and each part's rate within its parent, which shows where the frame loses incidence.
+
+  Every donor share rests on 30 or more donors, which the script checks.
+- **The pub lean of the spending rule** (`scripts/alcohol_channel_weights.py`, `outputs/alcohol_channel_weights.json`). Measured on the LCFS 2024-25 donor at survey weights: how far allocating alcohol volume by spending moves it, against channel weights for published on-trade volume shares of 20 to 35 percent, overall and by equivalised income decile.
+- **The re-pin** (`scripts/repin_200bf6d.sh`). The commands that regenerated the vendored resources, the national and local references and membership, the local census, both compile-parity receipts and the coverage manifest. They ran on the national and local cuts of the pinned artifact that the runbook entry describes.
+
+## What is not here
+
+- **Unit records.** Household dumps, the spine and the LCFS tabs stay out of the repository.
+- **Column digests.** The runs record the sha256 of every column they write. `scripts/sanitize_outputs.py` drops those digests, with any donor minimum or maximum, before an output is copied here; the comparison keeps only the verdicts.
+- **Small cells.** Every count in the receipts is a band ("30+") or withheld ("<10"), so no figure here rests on fewer than ten households.
+
+## Re-running
+
+The stage runs need the licensed LCFS 2024-25 tabs and the spine:
+
+```bash
+BASE_LABEL=base-74a2a378e HEAD_LABEL=head-restack scripts/run_stage_dev.sh
+```
+
+The reconciliation needs only the tree and its locked engine. The incidence script reads the split run, the reconciliation and the LCFS household tab:
+
+```bash
+python scripts/national_reconciliation.py outputs/national_reconciliation.json
+```
+
+```bash
+python scripts/excise_incidence.py outputs/stage_dev_run.head-split.json outputs/national_reconciliation.json <lcfs dvhh tab> outputs/excise_incidence.json
+```

docs/evidence/uk-excise-quantities-1359/outputs/alcohol_channel_weights.json

@@ -0,0 +1,206 @@
+{
+ "by_v": {
+  "0.2": {
+   "decile_share_channel": {
+    "1": 0.05256630738429888,
+    "10": 0.16251924953110025,
+    "2": 0.05326023979493307,
+    "3": 0.06982855194468635,
+    "4": 0.09022999346372193,
+    "5": 0.07847780743163882,
+    "6": 0.09008038971165136,
+    "7": 0.10645365490133937,
+    "8": 0.13757646695267162,
+    "9": 0.15900733888395843
+   },
+   "decile_spending_over_channel": {
+    "1": 0.8991829846472777,
+    "10": 1.1084215139898075,
+    "2": 0.9144419252310921,
+    "3": 0.8971043041743317,
+    "4": 0.9373556351590888,
+    "5": 0.9717899833896708,
+    "6": 1.0460821810658767,
+    "7": 1.0282597684462984,
+    "8": 0.9898997927959324,
+    "9": 1.0095420211358075
+   },
+   "pub_pound_overallocation": 2.148346081898008,
+   "share_of_national_volume_moved": 0.1633070937520641,
+   "shop_pound_allocation": 0.7129134795254979
+  },
+  "0.25": {
+   "decile_share_channel": {
+    "1": 0.05141256586374622,
+    "10": 0.166355328080157,
+    "2": 0.05226819487693822,
+    "3": 0.06826433329272477,
+    "4": 0.08899944095014692,
+    "5": 0.07799584030385238,
+    "6": 0.09098410264597774,
+    "7": 0.10710858717419604,
+    "8": 0.13727395561821207,
+    "9": 0.15933765119404866
+   },
+   "decile_spending_over_channel": {
+    "1": 0.9193614123630119,
+    "10": 1.0828618157090257,
+    "2": 0.9317979381345949,
+    "3": 0.9176606799222224,
+    "4": 0.9503160012090827,
+    "5": 0.9777950578305806,
+    "6": 1.0356918164867368,
+    "7": 1.0219723126502438,
+    "8": 0.992081240150249,
+    "9": 1.0074492065710414
+   },
+   "pub_pound_overallocation": 1.7186768655184064,
+   "share_of_national_volume_moved": 0.12775442014469934,
+   "shop_pound_allocation": 0.7604410448271978
+  },
+  "0.27": {
+   "decile_share_channel": {
+    "1": 0.05095106925552516,
+    "10": 0.16788975949977966,
+    "2": 0.05187137690974029,
+    "3": 0.06763864583194015,
+    "4": 0.08850721994471691,
+    "5": 0.0778030534527378,
+    "6": 0.09134558781970831,
+    "7": 0.1073705600833387,
+    "8": 0.1371529510844282,
+    "9": 0.15946977611808477
+   },
+   "decile_spending_over_channel": {
+    "1": 0.9276886600485714,
+    "10": 1.0729649810355817,
+    "2": 0.9389262271000718,
+    "3": 0.9261494480461296,
+    "4": 0.9556010558959604,
+    "5": 0.9802179194262256,
+    "6": 1.0315932360829085,
+    "7": 1.019478807357914,
+    "8": 0.9929565135365738,
+    "9": 1.006614508278182
+   },
+   "pub_pound_overallocation": 1.5913674680725984,
+   "share_of_national_volume_moved": 0.11353335070175342,
+   "shop_pound_allocation": 0.7812750460553403
+  },
+  "0.3": {
+   "decile_share_channel": {
+    "1": 0.05025882434319356,
+    "10": 0.17019140662921373,
+    "2": 0.05127614995894338,
+    "3": 0.06670011464076324,
+    "4": 0.08776888843657192,
+    "5": 0.07751387317606594,
+    "6": 0.09188781558030412,
+    "7": 0.10776351944705269,
+    "8": 0.1369714442837525,
+    "9": 0.15966796350413892
+   },
+   "decile_spending_over_channel": {
+    "1": 0.9404662720110231,
+    "10": 1.0584543378868112,
+    "2": 0.9498255281518766,
+    "3": 0.9391812119248586,
+    "4": 0.9636397855797082,
+    "5": 0.9838748091870372,
+    "6": 1.025505829534886,
+    "7": 1.0157612808191105,
+    "8": 0.9942723232728629,
+    "9": 1.0053650510058427
+   },
+   "pub_pound_overallocation": 1.4322307212653387,
+   "share_of_national_volume_moved": 0.0922017465373346,
+   "shop_pound_allocation": 0.8147582623148548
+  },
+  "0.35": {
+   "decile_share_channel": {
+    "1": 0.04910508282264091,
+    "10": 0.17402748517827046,
+    "2": 0.05028410504094855,
+    "3": 0.06513589598880169,
+    "4": 0.08653833592299691,
+    "5": 0.07703190604827953,
+    "6": 0.09279152851463054,
+    "7": 0.10841845171990935,
+    "8": 0.1366689329492929,
+    "9": 0.15999827581422915
+   },
+   "decile_spending_over_channel": {
+    "1": 0.9625628641421781,
+    "10": 1.0351228855213166,
+    "2": 0.9685644435092743,
+    "3": 0.9617353619363558,
+    "4": 0.9773424914115045,
+    "5": 0.9900306391568691,
+    "6": 1.0155182488018917,
+    "7": 1.0096252879712762,
+    "8": 0.9964731061489618,
+    "9": 1.003289500811369
+   },
+   "pub_pound_overallocation": 1.2276263325131476,
+   "share_of_national_volume_moved": 0.05664907292996989,
+   "shop_pound_allocation": 0.8774319748006129
+  }
+ },
+ "deciles": {
+  "1": {
+   "on_trade_share_of_its_alcohol_spending": 0.3100381371801737,
+   "share_of_volume_spending_weights": 0.0472667291657001,
+   "support": "30+"
+  },
+  "10": {
+   "on_trade_share_of_its_alcohol_spending": 0.534037558606296,
+   "share_of_volume_spending_weights": 0.18013983261774946,
+   "support": "30+"
+  },
+  "2": {
+   "on_trade_share_of_its_alcohol_spending": 0.32983874112325107,
+   "share_of_volume_spending_weights": 0.04870339621634823,
+   "support": "30+"
+  },
+  "3": {
+   "on_trade_share_of_its_alcohol_spending": 0.30728862922647476,
+   "share_of_volume_spending_weights": 0.06264349450383902,
+   "support": "30+"
+  },
+  "4": {
+   "on_trade_share_of_its_alcohol_spending": 0.35836160645506054,
+   "share_of_volume_spending_weights": 0.0845775928335875,
+   "support": "30+"
+  },
+  "5": {
+   "on_trade_share_of_its_alcohol_spending": 0.3986957944955206,
+   "share_of_volume_spending_weights": 0.07626394718045007,
+   "support": "30+"
+  },
+  "6": {
+   "on_trade_share_of_its_alcohol_spending": 0.47667220643505265,
+   "share_of_volume_spending_weights": 0.09423149054082841,
+   "support": "30+"
+  },
+  "7": {
+   "on_trade_share_of_its_alcohol_spending": 0.45899327098154097,
+   "share_of_volume_spending_weights": 0.10946201053911338,
+   "support": "30+"
+  },
+  "8": {
+   "on_trade_share_of_its_alcohol_spending": 0.4187824898318171,
+   "share_of_volume_spending_weights": 0.13618691613004608,
+   "support": "30+"
+  },
+  "9": {
+   "on_trade_share_of_its_alcohol_spending": 0.43975417772508696,
+   "share_of_volume_spending_weights": 0.16052459027233767,
+   "support": "30+"
+  }
+ },
+ "drinking_households_share": 0.6013231787655217,
+ "households": 4995,
+ "off_trade_only_share_of_drinkers": 0.4842431550469689,
+ "on_trade_only_share_of_drinkers": 0.17136932314633915,
+ "on_trade_share_of_alcohol_spending": 0.4296692163796016
+}