Require every reported number to come from a model run

[MaxGhenis]

Why

PolicyEngine is a microsimulation model; its outputs are the product. The uk-triple-lock analysis reported a 20,000-draw uncertainty range for State Pension costs that ran the model only about 30 times. Every other draw was priced as spend × (index_TL − index_alt) / index_central from one central run, and the few full runs were described as validating that shortcut. The draws also changed only the pension rate. Every other CPI- or earnings-linked parameter, and every income in the data, stayed on the central path. Nothing in the skills told an agent this is unacceptable, so this adds it where agents and reviewers look.

What changes

• policyengine skill: "Every reported number is a model run", a new top-level section. It lists what is banned (ratio scaling, interpolation or extrapolation between runs, elasticities or shares borrowed from one run, side models, and changing one parameter when a scenario should move the whole system). It also says what to do instead: run fewer scenarios fully and state how many runs stand behind each figure. The description gains scenarios, uncertainty, Monte Carlo so the skill loads for that work.

• policyengine-uk skill: "Macro scenarios", with a <!-- verify: slow --> example. A CPI or earnings path has to be a Scenario applied before the data load. Verified on policyengine 5.3.0 / policyengine-uk 2.90.2 with +1pp CPI and earnings in 2027–33, by reading the code and running it:

  2034	Baseline	Before data load	Same changes as reform= dict
  Employment income	£1,662.6bn	£1,779.4bn	£1,662.6bn
  Child benefit, eldest	£31.79/wk	£34.03/wk	£31.79/wk
  New State Pension	£305.94	£324.52	£305.94
  Income tax	£476.2bn	£510.0bn	£476.2bn

  The dict form is a silent no-op. It edits yoy_growth after process_parameters(), so nothing derived from it is rebuilt. The section also covers the period-key trap ("year:2030-01-01:1", not a bare year and not the start.stop form) and the ./data/ HDF5 lock when running processes in parallel.

• policyengine-tools: data files hold model outputs only; no scaling in the pipeline or the frontend.

• review-program rubric: a figure published as a PolicyEngine estimate that was derived from other runs' output, instead of a run of that scenario, is always CRITICAL, whatever validation accompanies it. Hand-calculated test values are not covered.

• Plugin release 4.0.3 (all bundles and the marketplace template).

Checks

• uv run pytest --ignore=tests/test_skill_examples.py: 178 passed
• scripts/build_claude_wrapper.py: builds
• The new slow example ran as written against the policyengine 5.3.0 certified bundle (dataset enhanced_frs_2024_25) and its assertions passed.
• skill-examples fails on 8 US examples unrelated to this PR: policyengine 6.0.0 (15 Sep, after main last passed on 13 Sep; CI now installs 6.1.2) makes pe.us.calculate_household raise SPMInputError: County selection has no county FIPS input. Tracked separately; this PR merges once that is fixed.

Review round 1 (independent Opus review)

All eight findings addressed:

• F1: the rule now bans figures derived from other runs' output, not arithmetic on the runs being reported.
• F2: the macro-plus-policy recipe now uses two macro simulations. Policies read by process_parameters() go into the macro scenario. It warns that + drops applied_before_data_load and that sim.baseline uses the default macro path.
• F3: documents the horizons (triple lock to 2034, indices to 2039).
• F4: says pe.uk Policy/Dynamic can't carry a macro path.
• F5: reform-comparator no longer allows labelled linear scaling.
• F6: the rubric covers published estimates, not hand-calculated test values.
• F7: gives the precise reason reform= is a no-op.
• F8: fixes the verify-marker wording and names the dataset.

Review round 2

The reviewer confirmed F1 and F3–F7 fixed. Two things were still open:

• F2. The recipe claimed that merging a policy into the macro scenario makes later years of an uprated parameter follow it. It doesn't: apply_parameter_changes reloads the raw YAML, and later years uprate from the old value. That clause is gone; the skill now says to set every reported year through the after-load route, and it documents the key formats and the merge override.
• F8. Fixed:
  • the rubric summary above;
  • the dataset line: the 5.3.0 bundle serves enhanced_frs_2024_25, and the skill's stale populace_uk_2023 default is corrected per policyengine.py#491;
  • the policyengine_uk exemption;
  • the tools-skill wording.

🤖 Generated with Claude Code