Fall back to the committed OBR workbooks when obr.uk answers 200 with a non-workbook page

[MaxGhenis]

What

_download_workbook in policyengine_uk_data/targets/sources/obr.py falls back to the committed EFO workbooks in storage/obr_efo/ when the download fails. obr.uk can also answer HTTP 200 with an HTML page titled "No Access - Office for Budget Responsibility" instead of the xlsx. The UK hub saw this on the receipts URL from Max's machine on 2026-10-04. On a 200, openpyxl.load_workbook then raised zipfile.BadZipFile. That is not a requests.RequestException, so it escaped before the fallback ran. get_targets() logged Failed to download/parse OBR receipts: File is not a zip file, and nine targets disappeared:

• the six receipts targets: obr/income_tax, obr/ni, obr/vat, obr/fuel_duties, obr/capital_gains_tax, obr/sdlt;
• the three NICs targets: obr/ni_employee, obr/ni_employer, obr/ni_self_employed.

Fix. A 200 body that openpyxl cannot read is now a permanent download failure:

• it is not retried (one request per URL);
• control reaches the existing fallback;
• the warning names the URL, the status, the Content-Type and the parse error.

The except clause wraps only the one load_workbook call on the response body, and it catches Exception. A first version caught a named tuple (BadZipFile, InvalidFileException, KeyError, OSError, ValueError). Review showed that a zip with malformed XML inside raises xml.etree.ElementTree.ParseError, a SyntaxError subclass, which escaped that tuple. So could zlib.error, NotImplementedError, RuntimeError and TypeError. An openpyxl break on every workbook still surfaces, because the fallback goes through the same load_workbook outside any try. The parse error is chained as the __cause__ of the reported error. The docstring now says the fallback covers both 403s and 200 pages that are not workbooks. get_targets()'s broad catch-and-log is unchanged.

The fallback warning now reads:

OBR download <url> failed (200 for url: <url>, but the body (text/html; charset=UTF-8) could not be read as an xlsx workbook (BadZipFile: File is not a zip file)); using committed workbook fallback

Impact, from real calibrations (2026-10-04)

I ran four seeded calibrations of one saved input, changing only the code (main or this branch) and what obr.uk returns.

Setup

• Code: main b45c373 (base) and this branch 7605e63 (head). They differ only in obr.py, its tests and a changelog fragment; uv.lock and pyproject.toml are identical. Both ran in one environment built from that lock, with PYTHONPATH selecting the code.
• Input: one calibration input (calibration_input_2025.h5, sha256 02b1aa09ec86…), built from main through every step of create_datasets.py up to calibration, including calibration-year materialisation.
  • It uses 1 output-area clone, the same as the release build (PE_UK_DATA_OA_CLONES: "1" in .github/workflows/push.yaml; 10 is only the default in create_datasets.py). Each run takes about 8 minutes.
  • It reuses the cached imputation models in storage/.
• Calibration: calibrate_local_areas for constituencies exactly as create_datasets.py calls it: 512 epochs, 650 areas, torch.manual_seed(0), 8 threads.
• OBR responses: requests.get is wrapped for the two EFO URLs only; every other target source downloads normally.
  • blocked: 200 with the "No Access" HTML page;
  • receipts blocked: only the receipts URL gets that page, which is the split the UK hub saw on 2026-10-04;
  • served: 200 with the committed workbook bytes (today's live downloads are byte-identical to these).

Results. Weighted 2025 totals from the calibrated dataset (£bn):

Run	Code	obr.uk	National targets	Nine receipts/NICs targets	Income tax	Employee NI	Employer NI	VAT	UC	HB	Household net income
A	main	blocked	609	missing	287.38	47.39	144.08	392.93	76.04	1.22	1,666.33
A2	main	receipts blocked	628	missing	287.14	47.44	143.74	388.68	74.39	12.28	1,693.89
B	this branch	blocked	637	present	290.72	48.45	146.21	316.82	74.45	12.27	1,692.73
D	main	served	637	present	290.72	48.45	146.21	316.82	74.45	12.27	1,692.73

• Released dataset: no change. B and D have identical household weights (maximum absolute difference 0.0; the weight arrays and the full calibration logs are byte-identical). With a working download, main and this branch take the same path to the same targets. With the block page, this branch reproduces that result through the committed workbooks.
• What the fix restores, A2 → B (the case seen on 2026-10-04):
  • the 9 receipts and NICs targets (628 → 637 national targets);
  • income tax +£3.6bn, employee NI +£1.0bn and employer NI +£2.5bn;
  • VAT −£71.9bn: without its target, the model's VAT total drifts further from OBR.
• What the fix restores, A → B (both workbooks blocked): 28 OBR targets (609 → 637). These are the nine above plus the expenditure targets: council tax, HB, PIP, ESA, Pension Credit, UC in and outside the cap, State Pension, Child Benefit and others. Without them, HB falls from £12.27bn to £1.22bn and household net income falls by £26.4bn.
• Released datasets: the 2026-09-25 main build (Actions run 36131888928, push of b45c373) logged no OBR download failure, retry or fallback. Its _parse_nics "no row matched" warnings appear, which happens only after the receipts workbook loads. So that build fetched the workbooks directly and had these targets, matching run D.

Two pre-existing fits on main are outside this PR's scope; the same values appear in B and D:

• the obr/ni target (total NICs, £200.08bn) maps to the ni_employee variable, whose estimate is £48.45bn;
• obr/vat ends at 1.76× its target.

They are being raised separately.

The calibration inputs and outputs are FRS-derived, so they stay on the hub's machine (~/reviews/uk-hub/jobs/obr-fallback-r2/), along with the harness (impact.py) and run logs. Only the aggregates above leave it.

Unit-level check, with the receipts request stubbed and expenditure live: get_targets() returns 25 targets on main and 34 on this branch.

Invariants

• For each of the ten download-failure kinds below, get_targets() returns the nine OBR receipts and NICs targets, provided the committed workbook exists and is readable. Enforced by test_every_failed_download_keeps_receipts_and_nics_targets, with requests.get monkeypatched (no network). Out of scope: a 200 whose body is a valid xlsx that lacks the expected OBR sheets or rows. It loads, skips the fallback, and the per-table parsers then log and drop those targets. The ten kinds:
  • 200 with an HTML page;
  • 200 with an empty body;
  • 200 with a truncated zip;
  • 200 with a zip that has no xlsx manifest;
  • 200 with a zip that has no workbook part;
  • 200 with a zip containing malformed XML;
  • 403;
  • 404;
  • 500 until retries run out;
  • ConnectionError.
• Per URL, _download_workbook returns the committed workbook for every failure kind, at both EFO URLs in sources.yaml. Enforced by test_every_failed_download_uses_committed_workbook, which uses a spy on _fallback_workbook and checks that the returned object is the fallback's.
• Retry bound. A non-workbook 200, a 403 or a 404 costs exactly one request. 500 and ConnectionError cost _DOWNLOAD_MAX_ATTEMPTS (4). Enforced by the same parametrized test.
• Positive control. A 200 whose body is the committed efo_receipts.xlsx is parsed from the response, with one request and the fallback never called: test_workbook_response_is_parsed_without_fallback.
• No silent swallow. A URL with no committed workbook still raises: HTTPError for a 403, and ValueError chained from BadZipFile for an HTML 200 (test_unknown_url_with_failed_download_still_raises). The warning text is pinned by test_non_workbook_200_warning_names_url_status_and_parse_error.

The two older tests, test_403_uses_fallback_without_retrying and test_connection_failure_uses_fallback, are folded into the response-kind table, which covers both cases at both URLs. Hypothesis is not a dependency, so the property is enforced by exhaustive parametrization over the response kinds.

Tests

• pytest policyengine_uk_data/tests/test_obr_*.py -q: 54 passed, 5 skipped. The skips are the enhanced-FRS-gated signal tests in test_obr_nic_signal.py. Of these, test_obr_efo_fallback.py accounts for 36 passed.
• Mutation check, run on an earlier revision of this suite (33 cases; the head's suite has 36). The tests were run against main's obr.py, and exactly the non-workbook-200 cases failed (15 failed, 18 passed), with the hub's error: File is not a zip file. The malformed-XML row fails against the first version's exception tuple.
• ruff format --check and ruff check are clean on both changed files.

Review

There were two rounds of independent review by an Opus 5.5 agent, then GPT-6.1 Sol reviews at 7605e63. The first asked for real-run impact and a narrower invariant, both addressed above, and found no code defect. The second approved with two wording nits (the clone count and the mutation-check revision), both fixed in this description.

The Opus rounds:

• Round 1 requested changes: the tuple let errors escape, two behaviours were untested, and some tests duplicated others.
• Round 2 approved, with three nits: the comment's scope, the changelog's framing of "nine", and the cause chaining not being tested. All three are fixed in 7605e63.

axiom: n/a: data pipeline infrastructure (OBR target download), no policy change.

Part of the uk-data batched release (d833).

🤖 Generated with Claude Code

[MaxGhenis]

Queued in release PR #544 for the 10/8 uk-data batch. It lands only on Max's go (d833).