Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3b265a4
Re-pin the UK Chronicle feed to 00b4b14 (consumer_fact.v4, chronicle#…
juaristi22 Sep 22, 2026
12e3703
Add the calendar_year_window value operation for fiscal-year facts at…
juaristi22 Sep 22, 2026
d0d359e
UK: declared uprating appliers for the HMRC SPI band rows
juaristi22 Sep 22, 2026
b71a312
UK income targets bind at the calibration year (microcosm#280 lane)
juaristi22 Sep 22, 2026
744e28e
UK SPI Table 3.11 rows bind on the region tier, uprated (microcosm#28…
juaristi22 Sep 22, 2026
d7bac38
Move the UK Chronicle pin to 5324aa2 (chronicle#284, the v4 export fix)
juaristi22 Sep 22, 2026
497e526
UK OBR rows bind at the calendar-2025 window (microcosm#280 lane)
juaristi22 Sep 23, 2026
a230ea2
UK spine: reserved SPI income band donors for the top of the distribu…
juaristi22 Sep 23, 2026
f54c459
Route the reserved-band gate parameters through the stage-health binding
juaristi22 Sep 23, 2026
6091d42
Shard follow-ups for the reserved income band donors
juaristi22 Sep 23, 2026
9c17173
UK target-fit register: retire the SE 20-30k deferral, defer two v21b…
juaristi22 Sep 23, 2026
6e05342
UK uprating: vendor the engine index values so the target compile run…
juaristi22 Sep 24, 2026
1bd392f
Rebase onto main 7c5306f3 over #954: feed 5324aa2 over the union, fix…
juaristi22 Sep 24, 2026
7cbab37
UK targets: the Universal Credit total at the window of its cells, th…
juaristi22 Sep 24, 2026
bc7d7b2
UK spine: cut the reserved band pools on the total income a record re…
juaristi22 Sep 24, 2026
a03b5e3
Docs, evidence and regenerated surfaces for the #1006 review round
juaristi22 Sep 24, 2026
c6ac121
Cite the lane as PolicyEngine/chronicle#280 in the parity rationales …
juaristi22 Sep 25, 2026
3bd60b4
State the licensed limitations: the blocked spine build and calibrati…
juaristi22 Sep 25, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/uk-chronicle-feed-repin-00b4b14.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Re-pin the UK Chronicle feed to 00b4b14 (consumer_fact.v4; HMRC Income Tax liabilities July 2026, SPI 2023-24 Tables 3.3/3.4/3.5/3.8/3.11 and the 3.7 remainder, ESA caseload by payment type, property rental income 2026, DWP Spring 2026 expenditure tables, bus journeys revenue components; 287,024 rows). No compiled target value moved on either surface. The register version moves on v4 geography labels only (eleven region-tier specs). The pin then moves to 5324aa2 (Chronicle main after PR #284, the consumer-export validator fix): the same artifact bytes, so only the commit and feed label change.
1 change: 1 addition & 0 deletions changelog.d/uk-income-anchors-280.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
UK income targets at the calibration year (PolicyEngine/chronicle#280 lane, María's rulings of 2026-09-22): the HMRC SPI 2023-24 band rows declare an `uprating_index` and move to 2025 (amounts by the engine's own per-variable indices, the state pension by the rate the engine pays, counts by HMRC's projected taxpayer growth in the containing Table 2.5 band); HMRC's Income Tax liabilities Table 2.5 binds taxpayers, total income and tax by band as the calendar-2025 window of the 2024-25 and 2025-26 projections (new `calendar_year_window` operation, 3/12 and 9/12); the SPI savings-interest rows bind uprated and the ONS D.41 row stays a measured diagnostic on the exclusion register; the three ESA rows move to the Stat-Xplore payment-type cube as the mean of the four quarterly points inside calendar 2025, the paid caseload without credits-only claimants. the SPI Table 3.11 region rows (Income Tax payers, total income and Income Tax liabilities by ten regional bands) bind on the twelve-area region tier, uprated by HMRC's projected growth for the same measure in the Table 2.5 band(s) each regional band spans. 705 to 1,124 active national references; no local row moves. The 27 OBR rows that resolved to FY2025-26 bind at the same calendar-2025 window (income tax £331.4bn to £325.1bn). Review round on microcosm#1006: the Universal Credit total binds at the calendar-2025 window of the sum of its two welfare-cap cells (`calendar_year_window` now composes one series per declared selector member; £79.29bn to £76.05bn, the one compiled value that moves), the cars share of fuel duty declares its FY2024-25 hold as `measurement.period_basis` metadata, and the three new register adjudications cite the rulings recorded on microcosm#1006 (the `ons.savings_interest_income` exclusion expires 2026-10-22).
1 change: 1 addition & 0 deletions changelog.d/uk-spi-income-band-donors-280.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
UK spine stage `spi_income_band_donors` (PolicyEngine/chronicle#280 lane, María's ruling of 2026-09-23): reserved SPI-channel households for the HMRC Table 2.5 total-income bands from GBP 200,000 (200k-500k, 500k-1m, 1m-2m, 2m and over), 120 per band, each a whole FRS household copied at clone index 2 with one carrier adult drawn by the SPI 2022-23 tape's own band propensity given region, sex and age band, starting at the build tax year's published band taxpayers over 120 (mass added and receipted, as the CGT band donors are). The income stage's new `resample_band_donor_leaves` operation gives every carrier a band-conditional, FACT-weighted draw of all stage-1 leaves from the full prepared tape (regional pool where it holds at least 20 records), uprated as the forest draws are. New stage-health gate `uk_stage_spi_income_band_donors_support`. The synthetic income channel, the forest and every other stage are unchanged; the spine previously carried no record above GBP 1.51m of total income. Review round on microcosm#1006: the band pools are cut on the total income a tape record realises on the frame (its stage-1 leaves at the stage's uprating factors plus the published remainder held nominal), so a reserved row's amount sits in the band its weight is published for; the resample refuses any carrier outside its band and receipts `carriers_outside_band`.
1 change: 1 addition & 0 deletions changelog.d/uk-uprating-engine-pins-280.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The declared UK uprating appliers read the engine's index values from a vendored pin file (`uk/hmrc_uprating_engine_pins.json`: six parameters at every 1 January 2019–2026, written from policyengine-uk 2.100.0 by `tools/pin_uk_uprating_engine_values.py`), so the UK target compile runs and reproduces without policyengine-uk; a `requires_uk` test holds the pins in lockstep with the installed engine and a `--check` mode refuses drift.
41 changes: 41 additions & 0 deletions docs/evidence/uk-income-280/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Evidence for the income-anchor lane (PolicyEngine/chronicle#280, microcosm#1006)

Aggregate-only extracts of the licensed v21c national run behind microcosm#1006: the first
national calibration on the income-anchor target surface (HMRC Income Tax liabilities and the
OBR rows at the calendar-2025 window, the SPI 2023-24 bands uprated by declared indices, the
Table 3.11 rows on the region tier) on spine-u, the spine with the reserved SPI income band
donors. The licensed artifacts (the spine and calibrated H5 files, the full sidecars, the
FRS/SPI/WAS/LCFS/ETB inputs) live outside the tree under `data/ukds/acceptance/` and
`runs/uk-623-first-calibrated/spine-assessment-v21c/`; the head-to-head page with the
downstream legs is the private evaluation site's `national-v21c-2026-09-23` report.

- `v21c-receipts.json`, written by `scripts/extract_v21c_receipts.py` from the run tree:
the run identity (code f6d33a26 on the pre-rebase head, spine-u 897a0f3e, release
candidate false), the calibration summary (53,806 records all nonzero, loss 0.3072 to
0.0091, 97.6 percent of the 1,062 targets within 10 percent, Kish ESS 9,193), the six
terminal gate verdicts with the two signed target-fit deferrals as applied, the compiled
target, estimate and relative error of the 119 income-anchor rows (OBR income tax, Universal
Credit and fuel duty, the HMRC liabilities rows, the SPI amount bands, the region tier's
200,000-and-over rows and the deferred South East row), the pass-2 score against the
incumbent snapshot in the run tree (910 wins to 32 on 942 common targets, 120 rows pruned as
incumbent-unresolvable, losses 0.0105 against 0.2923), and the band-donor receipts (120
donors per band at the published band weights, carriers by region, the propensity table's
cell count, and the resample's per-band pool sizes, composite counts, weighted taxpayers,
regional matches and realised band means).

Redaction: per-record values are withheld (each band's realised minimum and maximum total
income, the reserved households themselves); counts, masses, means and target errors are kept.

The measurements are on the pre-rebase head (f6d33a26), and they do not hold at the current
head. A licensed spine build from the rebased branch is blocked by the student-loan PLAN_5
realisation gate (deviation 1.1027 against the limit of 1.0; raised to 2.0 for assessment builds
only, locally), and the national calibration at 8d36ecc4 blocks at the terminal gate
`uk_cgt_projection_entrants`, which #979 introduced after v21c's tree: 146,920 weighted
sub-exempt gainers cross the frozen exempt amount by 2030 against the bound of 73,000, most of
the mass on about fifty of #970's CGT band-donor rows whose weight floats with the target set.
So this head cannot produce a certified cut, the six green gate verdicts in `v21c-receipts.json`
are v21c's, and the fence is settled on the #970/#979 side (microcosm#1006, comments 5812328880
and 5814299594; the doc's limitations section). The realised-income pool cut from the review
round is likewise unmeasured until that rebuild. The receipt's text fields are the run's own
compiled metadata and still spell the lane as the run did (`microcosm#280`); the tree now cites
PolicyEngine/chronicle#280.
182 changes: 182 additions & 0 deletions docs/evidence/uk-income-280/scripts/extract_v21c_receipts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
"""Aggregate-only receipts of the v21c national run for docs/evidence/uk-income-280.

Usage (from the repository root, with the licensed run tree present):

python docs/evidence/uk-income-280/scripts/extract_v21c_receipts.py \
--run-dir runs/uk-623-first-calibrated/spine-assessment-v21c \
--out docs/evidence/uk-income-280/v21c-receipts.json

Kept: calibration summary, terminal gate verdicts and the deferral register as
applied, the target errors of the income-anchor rows (OBR, HMRC liabilities,
SPI bands, the region tier's top band), the pass-2 score against the
incumbent, and the band-donor stage receipts (counts, weights, pool sizes,
band means). Withheld: every per-record value (per-band realised minimum and
maximum total income, weights of individual households).
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any

ROW_PREFIXES = (
"obr.income_tax",
"obr.universal_credit",
"obr.fuel_duties_cars",
"hmrc.itl.",
"hmrc/employment_income_income_band",
"hmrc/state_pension_income_band",
"hmrc/self_employment_income_income_band",
"hmrc/savings_interest_income_income_band",
"hmrc/dividend_income_income_band",
"hmrc/private_pension_income_income_band",
"hmrc.spi_region.total_income_by_region_200000_plus",
"hmrc.spi_region.income_tax_by_region_12570_15000",
)


def _load(path: Path) -> Any:
return json.loads(path.read_text(encoding="utf-8"))


def _target_rows(diagnostics: dict[str, Any]) -> list[dict[str, Any]]:
rows = []
for row in diagnostics.get("targets") or diagnostics.get("target_rows") or []:
name = str(row.get("name") or row.get("target") or "")
if not name.startswith(ROW_PREFIXES):
continue
rows.append(
{
"name": name,
"family": row.get("family"),
"target": row.get("target")
if isinstance(row.get("target"), (int, float))
else row.get("compiled_target"),
"estimate": row.get("final_estimate") or row.get("estimate"),
"relative_error": row.get("final_relative_error")
if row.get("final_relative_error") is not None
else row.get("relative_error"),
}
)
return rows


def _band_donor_receipts(record: dict[str, Any]) -> dict[str, Any]:
evidence = (record.get("spine_provenance") or {}).get("stage_evidence") or {}
out: dict[str, Any] = {}
donors = evidence.get("spi_income_band_donors")
if isinstance(donors, dict):
out["spi_income_band_donors"] = {
key: value
for key, value in donors.items()
if key not in {"households", "carriers_by_household"}
}
income = evidence.get("hmrc_spi_income_spine") or {}
resample = income.get("band_donor_resample") if isinstance(income, dict) else None
if isinstance(resample, dict):
bands = []
for band in resample.get("bands") or []:
bands.append(
{
key: value
for key, value in band.items()
if key
not in {"realized_min_total_income", "realized_max_total_income"}
}
)
out["band_donor_resample"] = {
**{k: v for k, v in resample.items() if k != "bands"},
"bands": bands,
}
return out


def _verdict(entry: dict[str, Any]) -> Any:
for key in ("passed", "verdict", "status", "outcome"):
if key in entry:
return entry[key]
return None


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--run-dir", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
run = args.run_dir
diagnostics = _load(run / "calibration_diagnostics.json")
gates = _load(run / "microcosm_uk_2024_25.terminal_gates.json")
score = _load(run / "score_vs_enhanced_frs.json")["score_vs_enhanced_frs"]
record = _load(run / "build_record.json")
manifest = _load(run / "rowwise_candidate_manifest.json")
target_fit = (gates.get("gates") or {}).get("uk_target_fit") or {}
payload = {
"run": {
"build_id": record.get("build_id"),
"code": ((manifest.get("identity") or {}).get("code") or {}).get(
"git_commit"
),
"release_id": (record.get("run_config") or {}).get("release_id"),
"release_candidate": manifest.get("release_candidate"),
"calibration_year": (record.get("run_config") or {}).get(
"calibration_year"
),
},
"calibration": {
key: diagnostics.get(key)
for key in (
"n_records",
"n_nonzero",
"initial_loss",
"final_loss",
"fraction_within_10pct",
"fraction_within_25pct",
"effective_sample_size",
"epochs",
"learning_rate",
"loss_weight_rule",
)
if key in diagnostics
},
"terminal_gates": {
"blocked_at_phase": gates.get("blocked_at_phase"),
"verdicts": {
gate_id: _verdict(entry)
for gate_id, entry in (gates.get("gates") or {}).items()
},
"target_fit_deferrals": (target_fit.get("details") or {}).get(
"reviewed_exclusions"
),
},
"income_anchor_rows": _target_rows(diagnostics),
"pass_2_vs_incumbent": {
key: score.get(key)
for key in (
"candidate_target_wins",
"incumbent_target_wins",
"candidate_full_loss",
"incumbent_full_loss",
"target_wins_by_family",
)
}
| {
"n_scored": (score.get("incumbent_unresolvable_pruned") or {}).get(
"n_scored"
),
"n_pruned": (score.get("incumbent_unresolvable_pruned") or {}).get(
"n_pruned"
),
},
"band_donors": _band_donor_receipts(record),
}
args.out.write_text(
json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
)
print(f"wrote {args.out}: {len(payload['income_anchor_rows'])} income-anchor rows")
return 0


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading