UK bus fares: impute local-bus journeys from the NTS microdata and price them at the published yield (#930)

[juaristi22]

Closes #930. Follow-up to #927 (#890): the structural fix María ruled on 2026-09-15 for the rake-then-bind bus fares. Plan of record and rulings in the receipts (experiments/930-nts-bus-journeys-receipts.md).

The licensed NTS tabs are pinned (UKDS SN 5340, 19th edition, End User Licence) and the stage's codebook is verified against the deposited lookup tables and the data extract user guide (three corrections: the frequency question is OrdBus2Freq_B01ID, Age_B01ID's 21 bands map to lower ages before the declared edges, and the trip weight is W5 divided by W2). The licensed twins (spine-t against a main control, then each change against its predecessor), the stage receipts, the E6 bounds and the calibration rounds are in the receipts (Parts E–K).

What changes

• Chronicle re-pin to 7846605 (PolicyEngine/chronicle#274 → chronicle PR Carry the full geography ladder from census block: county, place, state leg districts, metro/CBSA as spine columns #275): BUS01 journeys and concessionary journeys by area, the nine BUS05i revenue components, NTS0303/NTS0601 trip rates, NTS0205 car availability, the DfI concession journeys. Three new vendored resources, the NTS0705a selection pinned to its income-quintile breakdown; no committed target moves.
• nts_bus_travel, a person-grain stage between was_wealth and lcfs_consumption: one regime-gated QRF on the NTS frequency-of-use band, trips assigned from the diary's band means by series (bus in London, other local bus) and residence group, the statutory concessionary eligibility per area (England outside London 66+, London under 18 or 60+, Scotland under 22 or 60+, Wales 60+, Northern Ireland 60+, under-5s free everywhere). Two stage gates; NTS0205 car availability as a receipt line; a published-shares fallback if the extract lacks the frequency column.
• bus_fare_spending priced, not raked: the Bind DfT, HMRC, DESNZ, Ofgem and ORR facts for bus fares, road fuel, domestic energy and rail support (#790 ruling: bind facts and calibrate) #890 incidence draw and fare rake leave the lcfs declaration; price_bus_journeys sets the column as the household's non-eligible persons' trips × boardings per resident trip (BUS01 over NTS0705a × ONS population) × yield per fare-paying boarding (BUS05ai receipts over journeys less concessionary), per area; Scotland and Northern Ireland from their own receipts and journeys with England's trip rate as the declared proxy; Wales keeps the clipped raw draw. The chain still conditions on the raw draw, so every other LCFS column is byte-identical. A bus_pricing gate recomputes every factor from the vendored rows.
• Support side: the ETB bus_subsidy_spending rake retires too (María's ruling, 2026-09-23). price_bus_support sets each household's support from its persons' boardings at the published support per boarding (concessionary reimbursement per concessionary boarding for eligible persons plus the rest of the net support per boarding for every boarding) for London, England outside London and Scotland; Wales (no published boardings) and Northern Ireland (bus and rail jointly in the NITHC accounts) keep the clipped raw draw. The rake_to_vendored_facts kind and its module leave with their last consumer; a bus_support_pricing gate recomputes every area's factors from the vendored rows.
• The single-fare share (C13, María's ask of 2026-09-23): the stage also reads the NTS stage and ticket tables and gives every person local_bus_single_fare_share, the share of their fare-paying boardings paid at the point of use rather than on a season ticket or pass, the boardings a per-journey fare cap binds on; the fare paid per boarding by area and year (2022 pre-cap, 2023–24 capped), the boarding-class shares and the NTS boardings per trip are receipted. The parameter side is policyengine-uk#1871 (with Vahid's #1869).
• Weight-stretch instrument (Bind DfT, HMRC, DESNZ, Ofgem and ORR facts for bus fares, road fuel, domestic energy and rail support (#790 ruling: bind facts and calibrate) #890's unread acceptance line): the calibration writes its compiled system and both weight vectors before the terminal battery; tools/diagnose_uk_target_support.py reads the mass on households stretched beyond 3x and 5x per target.
• Vahid's third pass on UK #890 PR-S: uprate the LCFS and ETB donors, condition on vehicles, draw NTS bus use, rake bus and energy to vendored facts, price energy at the FY2024-25 cap #927: all nine items closed (stale energy-level sentence, conditional schema keys, declared fallback regions, per-sweep IPF residuals with tolerance governance, the walk documented with an unequal-weights test, the evidence-absent seam test, receipts attributions, the 04.5 sign-out recorded).

Licensed measurement (Parts E–K of the receipts)

State of this tree first. The national calibration on the final tree (spine-t4, Part K) solves with every bus row on target, but its terminal battery blocks on a row this PR does not touch: María's signed v20 deferral of the HMRC self-employment 20–30k band (approved 2026-09-18, expiring 2026-10-16) is stale here because the row solves to +24.6 %, inside the 25 % fence, and the target-fit gate refuses stale deferrals by design. No H5 is written from this tree until that deferral comes off, on her ruling; the readings below come from the pre-battery sidecars. The "all six terminal gates pass" line further down is spine-t's, the pre-rebase build on the register without that deferral.

• spine-t (this branch, the three tabs) passes all 24 stage gates; the control spine-u from main passes its 20. The frame reproduces NTS0313 (user share 0.5047 vs 0.5049; 60-and-over 0.5010 vs 0.5004) and sits inside the 15 % trip-rate fence (12.4 vs 13.1 London-series trips per person, 31.1 vs 28.1 other local bus).
• Design-weight fares against the bound receipts, without any rake: England outside London 1.12 of BUS05ai, London 0.90 (inside the 25 % fence); Scotland 1.43 and Northern Ireland 1.76 (outside it, because their persons are predicted on a declared proxy region that overstates their boardings 1.5× and 1.9× against BUS01; a ruling, recorded). The framing, as Vahid put it: the level enters from the receipts, the composition is free. Frame over receipts is the frame's trips over the published rate times population, times the frame's fare-paying share over the publisher's; the first is fenced at 15 % by the stage gate, the second is reported, and the receipts targets stay bound and act on both.
• Twin diff: nine differences, seven expected (the six new columns and bus_fare_spending), two the column_order structural surface adjudicated as in UK spine: relationship-to-head frame column, so the ONS household-composition split is measurable #791; Wales rows byte-equal; weights and every other column byte-equal.
• Calibration on spine-t4 (the final tree: support pricing, the single-fare share, Northern Ireland at 60 and under-5s free; Part K): loss 0.01045 on 643 rows, 96.7 % within 10 %, ESS 9,036, every bus row on target (England fares from −14.0 % at design weights, Northern Ireland from +49.1 %); the England fare row's mass on households stretched beyond 3× is 50.2 % against the frame-wide 48.0 %, the support row's 47.8 %. The terminal battery blocks on the stale self-employment deferral described above; the readings are from the pre-battery sidecars.
• Support priced from journeys (spine-t2, Part H): England outside London 0.95 of the published net support at design weights, London 1.16, Scotland 1.62 (the same proxy-region reading as the fares); Wales and Northern Ireland on the raw draw. Twin diff spine-t → spine-t2 clean: bus_subsidy_spending everywhere, rail support on the 679 Northern Ireland rows the retired joint cell used to scale, nothing else.
• National calibration on spine-t (the pre-rebase build; the Promotion for the constant-id national line: pointer and certified-loader extension #823 national role, this branch's Chronicle pin): all six terminal gates pass; loss 0.01036 against the v20 main twin's 0.01041, 96.6 % of 638 rows within 10 % (96.4 %), ESS 9,231 (9,305); every bus row lands on target, and the England fare row's mass on households stretched beyond 3× is 47.6 % against the frame-wide 46.6 %, the Bind DfT, HMRC, DESNZ, Ofgem and ORR facts for bus fares, road fuel, domestic energy and rail support (#790 ruling: bind facts and calibrate) #890 acceptance line measured for the first time.

Verification

Rebased over #939, #967, #971 and #979 (main 43ca60f6): the feed pin 78466057 is the descendant of #939's c5e5bf8, so the branch keeps its pin and regenerates every surface on it over the merged register; the national target references are byte-identical to main's; the H2 fixture carries 32 stages (#725's asset-type stage, #970's incidence anchor and nts_bus_travel), 34 declared, every stage recording the household-mass receipt #967 requires. The #979 rebase conflicted only on the regenerated surfaces (coverage manifest, contract digests, fixture) and on four test files' additions; both sides' additions are kept, the surfaces regenerated and the digests re-pinned over the union. #970's incidence anchor runs after the three bus stages, so the stage receipts in Parts E–K are unchanged by it; only the terminal weights and the calibration would move, and the rebased tree is not rebuilt here.

Unit suites for the stage, pricing, support pricing, gates, schema, graph, spine tool, source stages, vendoring, feed, contract pins and the H2 parity fixture green locally; tools/ci_test_groups.py --verify ok; ruff clean on the repo.

Rulings recorded

Chronicle facts first; per fare-paying boarding with a declared boardings-per-trip translation; statutory age eligibility at zero fare (Northern Ireland at 60 and under-5s everywhere since Vahid's second pass); Wales unpriced; the 04.5 sign-out stands; the support rake retires (2026-09-23). Open at review: pooled 2022–2024 donor vintage; vehicle shares as a receipt line; the devolved proxy region for Scotland and Northern Ireland.

Out of scope

The engine-side bus_usage quantity (policyengine-uk issue drafted), the dense solve, the QRF drift issue, rail support value-side treatment, local-grain journeys.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review at 78c1073b (Claude Code, high effort; worktree checkout of the seven commits on ce76b358; without the engine, the NTS or LCFS tapes or the feed: 1,047 passed, 15 skipped, 0 failed across 31 suites — the new stage, pricing, support and gate tests plus graph, stages, parity, terminal gates, energy pricing, LCFS, ETB, vendoring, feed, references, country spec, spec bundles and microcosm-fit; ruff check clean; the coverage manifest, both census tools, the spec-engine coverage check and ci_test_groups --verify current; CI 24 of 24 green; merge-tree clean against main 18c6d39a and against #965.)

The structure is what #930 asked for and the pieces I could recompute hold. The feed pin is chronicle #275's merge commit, no committed target moves, the three vendored resources and the NTS0705a quintile selection are declared with real consumers. The stage is registered in lockstep with a regime-gated QRF on identity-keyed uniforms, band means by series and residence group, and a declared published-shares fallback. Every England pricing factor recomputes from the vendored rows: London boardings per resident trip 2.369 and yield £1.018 (fare per trip £2.41, concessionary share 0.273); outside London 1.125 and £1.555 (£1.75, 0.281), matching the receipts, and the NTS0303 rates equal the NTS0705a all-income rates exactly. The bus_pricing gate recomputes every factor at 1e-9, the fact_rake gate fences the support rake and fails closed on a skipped cell, the stretch instrument writes the compiled system and weights before the battery as evidence only, and all nine of my #927 third-pass items are closed where you said, from the Ofgem sentence to the per-sweep IPF residuals to the 04.5 confirmation in Part T. The rebase fix removed two concatenated stale-plus-new digest strings and the pin test proves the result equals a clean regeneration.

Should-fix

1. The Northern Ireland eligibility rule is wrong in law. sources.yaml:1312 sets min_age: 65, the basis text says "Northern Ireland SmartPass 65+", the receipts justify it as "the 65+ SmartPass series, so half-fare 60–64 boardings are not deducted", and the test pins 60 → not eligible, 65 → eligible. nidirect's concessionary travel page says the 60+ SmartPass gives free travel on all Translink scheduled services within Northern Ireland from age 60; the Senior 65+ pass adds all-Ireland travel and nothing else. There is no age-based half-fare tier. So 60–64 year olds in Northern Ireland are fare-priced when they ride free, and the receipts' premise is false. The rule should be 60, with the test and the receipts moving with it; the DfI concession series' composition (a travel-status split, not a 65+ series) is worth checking too while you are there.

2. The body says "draft until the licensed NTS tabs land" but the PR is not draft, and it promises spine-t, the twin diff, the E6 bounds and the calibration round "in this PR". Either mark it draft or split the licensed measurement out; otherwise the review target keeps moving, and the changelog fragments describe a finished feature.

3. Nothing tests the zero placeholder pins, and one path skips them. "Refused at runtime" is the generic size-then-sha mismatch in read_pinned_tab, which a real tab does trip on size_bytes: 0, but no test feeds a 64-zero pin and the schema accepts it. --synthetic-fixture-dir skips the tab arguments entirely and is stopped from release only by the battery's "a synthetic smoke build cannot be a release candidate" rule. One refusal test on the placeholder.

4. Trivial. ruff format --check flags test_uk_stage_health.py:655.

Questions for María

• Is the rake-then-bind loop broken? Mechanically yes, in level no. The fare rake is gone and price_bus_journeys sets the column. But dft.bus_fare_receipts.england, scotgov.bus.passenger_revenue and dfi_ni.bus.passenger_receipts remain bound targets, byte-identical to main, and the same receipts R now price the column: at design weights frame over R is the frame's trips over T×P times the frame's fare-paying share over 1 − C/B, which the pricing module's own algebra string says. The first ratio is fenced at ±15% by the stage gate; the second is unfenced. So the England, Scotland and NI fits are near-tautological by construction, differing only by those two composition ratios. London is the exception, since non-London residents' London boardings land in their home region and London residents' other-local-bus trips are priced at the outside-London rate. I would describe this as "level injected from the receipts, composition free" rather than "loop broken", and let María rule whether that is the intended end state or whether the receipts targets should now be diagnostics.
• The support side is a pure tautology, acknowledged. The ETB rake still sets bus_subsidy_spending to the BUS05bi cells and the same cells are bound; London plus outside-London sum to the England target exactly at design weights. The fence and the per-journey diagnostic are the right preparation for the ruling, but the body should say "kept, tautological".

Nits

Receipts Part D still says 32 declared / 30 on the fixture with the pre-rebase oracle where Part A says 33 / 31. Scotland's under-22 rule has no floor, so under-fives are eligible there alone while the basis text says the yield absorbs them; immaterial. dft_bus_journeys.json lists lcfs_consumption as a consumer when that module only re-exports the constant. Discretionary local extensions of the English scheme are not mentioned.

Could not verify

The licensed Part E measurement and sidecar sizes, the DfI series composition, and vendor_uk_ledger_facts --check without the feed.

---

Nothing blocks on the code that ran. 1 is a factual error about the law; 2 is the state of the PR; the tautology question is María's and this PR is the right place to settle it.

[vahid-ahmadi]

Second pass at 99d1770e (16 commits on main 35e10f02, which is also the merge base). Local, without the tapes or the feed: ruff check/format clean on the 42 changed files; coverage-manifest --check current; ci_test_groups.py --verify ok; spec-engine coverage 42,162/42,162; duplicate-key scan over every uk/*.json and the H2 fixture clean; pytest over the 24 stage/pricing/support/gate/schema/graph/parity/vendoring/feed/contract suites 708 passed / 12 skipped / 0 failed. CI: 5 lanes green, 18 still queued or running at post time.

The new work is solid. C10's real SN 5340 pins are committed and asserted non-zero; C12 retires the ETB rake cleanly (no dangling rake_to_vendored_facts reference, the bus_support_pricing gate recomputes at 1e-9, the priced column carries no terminal bound); C13's output-order fix was a genuine column-order bug caught by the parity identity; the rebase declares nts_bus_travel's mass receipt with its coverage reason and the fixture stands at 31 stages. The support side is no longer tautological: with the rake gone the design-weight ratios are 0.95 / 1.16 / 1.62 against the still-bound targets, and the fares framing ("composition check, never a level the stage set") is recorded in sources.yaml:1860 and the changelog.

Requesting changes on one item from round one that has not moved, plus the stale text.

1. Northern Ireland eligibility is still 65. sources.yaml:1384 min_age: 65, basis text :1354, receipts :111 and :149 ("the 65+ SmartPass series, so half-fare 60–64 boardings are not deducted"), the test pin, and the body's stage bullet are all unchanged, and neither rulings list mentions it. nidirect: the 60+ SmartPass gives free travel on all Translink scheduled services within Northern Ireland from age 60; the 65+ pass adds all-Ireland travel and nothing else. There is no half-fare tier. Since C13, bus_pass_eligible also gates the fare-cap bite, so NI 60–64s are now both fare-priced and counted as cap-eligible. Either set 60 (with test, basis, receipts and body moving) or record a ruling that says why 65 stands.
2. Stale text. Body "Verification": "Licensed measurement pending (Part E of the receipts)" and "Rebased over #939 (main ce76b358)" while the tree is on #971 and Parts E–J exist; heading "Parts E–G" but bullets cite H and J. Receipts Part D still reads as current: :103 "32 declared / 30 on the fixture" (Part A says 33/31), :117 zero placeholder pins, :125 "the bus_subsidy_spending rake stays", and :128 cites uk_runtime/bus_support_per_journey.py, which is not in the tree. Mark Part D superseded or edit it.
3. The headline calibration line is spine-t's, not this tree's. Part J says the battery on spine-t3 blocks at uk_target_fit on the stale self-employment deferral (+24.2 %, deferral approved 2026-09-18, expires 2026-10-16), and no H5 is written. The body discloses this two paragraphs later, but "all six terminal gates pass" reads as the tree's state on a fast pass. Lead with the block and the ruling it needs.

Smaller. Scotland under-22 and London under-18 rules still have no floor. dft_bus_journeys.json still lists lcfs_consumption as a consumer. Schema size_bytes is an unbounded number (no runtime refusal test for a zero pin; acceptable now that the real pins are asserted). The C13 order fix is covered only by the parity identity, not a unit assertion on column order. test_acceptance_h_parity.py:283 comment says 30 spine stages against an assertion of 31.

Merge picture. Conflicts with #979 (e01c6846) in eight files: the coverage manifest, contract.py and its test, the H2 fixture, and four test files. Whichever lands second re-pins; the fixture needs regenerating over the union.

Ready to approve once 1 and 2 are settled.

[juaristi22]

Thanks for the second pass. Addressed at f9aa757d, rebuilt and re-measured as spine-t4 (Part K of the receipts).

1. Northern Ireland eligibility. You are right, and the round-one item should not have slipped. The rule is 60 (sources.yaml, the basis text, the test pins, the changelog). While there I made the child rule uniform: under-5s are declared free in every area (the London and Scotland under-age rules already covered them; England outside London, Wales and Northern Ireland now say so too instead of absorbing them in the yield). The receipts' description of the DfI series was wrong as well: dfi_ni.translink_bus.full_fare_concession_passenger_journeys is the full-fare-concession travel status, the count of free-travel boardings across every concession, and the only status the feed carries; not a 65+ series and there is no half-fare tier to deduct. Corrected in Parts D and K. Effect on the twin (spine-t3 → spine-t4): bus_pass_eligible moves on Northern Ireland's 60–64s and on under-5s outside London and Scotland, bus_fare_spending and the priced bus_subsidy_spending move on their households, nothing else. Northern Ireland's eligible persons go from 888 to 1,417 of 3,945 and its design-weight fares from 1.76 to 1.57 of the Translink receipts; England outside London moves from 1.12 to 1.06 on the fares and from 0.95 to 1.00 on the priced support, since the under-5s' boardings now earn the reimbursement instead of pricing at the yield; London and Scotland are unchanged.

2. Stale text. Part D is marked superseded with the four sentences you listed corrected in place (the stage count, the placeholder pins, the rake, the module name); the body's licensed section is now "Parts E–K", the verification paragraph says rebased over #939, #967 and #971 on 35e10f02, and "licensed measurement pending" is gone.

3. The headline. The body's licensed section now opens with the state of this tree: the calibration on spine-t4 solves with every bus row on target, its terminal battery blocks on the stale self-employment deferral (María's ruling), no H5 is written, and the "six terminal gates pass" line is labelled as spine-t's, the pre-rebase build.

Smaller. Scotland's and London's under-age rules keep no floor because under-5s are now declared free everywhere, so the floor would only re-price them; dft_bus_journeys.json no longer names the lcfs module (its constant is a re-export for the allowed-resource set, which the comment now says); a zero placeholder pin has a refusal test (test_read_pinned_tab_refuses_a_zero_placeholder_pin); the end-to-end test asserts the person column order against the declared outputs; the parity comment says 31. The schema's size_bytes stays unbounded, as you allowed.

Merge picture. Noted in the body: whichever of #979 and this lands second re-pins the digests and regenerates the fixture over the union.

On your round-one framing question, "level injected from the receipts, composition free" is the honest description of the fares and the body now uses it; the support side stopped being a tautology on 2026-09-23 when María ruled the rake retired.

[vahid-ahmadi]

Third pass at f9aa757d (two commits since 99d1770e; main 35e10f02 is still the base). Local, without the tapes or the feed: ruff check/format clean on the 40 changed files; ci_test_groups.py --verify ok; coverage-manifest --check current; merge-tree against main clean; pytest over the stage, pricing, stage-health, spine, source-stages, country-spec, contract-pin, data-contract and H2 parity suites: all passed (summary line not flushed; no F markers). CI: 2 lanes green, 11 queued or running at post time (GitHub's queue is slow today), none failed.

Everything from round two is closed.

1. Northern Ireland at 60, with the basis text now stating the 60+ SmartPass rule and the 65+ pass's all-Ireland scope correctly (sources.yaml:1354, 1381), the test pinning 60, and the changelog saying 60+. Making under-5s free in every area (max_age_exclusive: 5 on England outside London, Wales and Northern Ireland) is the right uniform rule, and the DfI series is now described as the full-fare-concession travel status rather than a 65+ series. Part K carries the twin effect: Northern Ireland eligible 888 → 1,417 of 3,945, its fares 1.76 → 1.57 of the Translink receipts, England outside London 1.12 → 1.06 on fares and 0.95 → 1.00 on priced support, London and Scotland unchanged. That is what the eligibility change should do and nothing else moved.
2. Stale text. Part D is headed superseded with the four sentences corrected in place; the body's licensed section is "Parts E–K", the verification paragraph names the three rebases on 35e10f02, and "pending" is gone. The dead module name survives only as history ("since renamed bus_support_pricing.py"), which is fine.
3. Headline. The licensed section now opens with the state of this tree: spine-t4 solves with every bus row on target and its battery blocks on the stale self-employment deferral, no H5 written; the "six terminal gates pass" line is labelled spine-t.

Smaller items landed: dft_bus_journeys.json no longer names the lcfs module, test_read_pinned_tab_refuses_a_zero_placeholder_pin exists, the end-to-end test asserts person column order against the declared outputs, the parity comment says 31. Keeping the Scotland and London under-age rules floorless is right now that under-5s are declared free everywhere.

One nit, not blocking: changelog.d/930-nts-bus-travel-stage.added.md still says "the artifact pins are placeholders refused at runtime until the licensed tabs land" and "(30 stages)"; the pins are real and the fixture has 31.

Merge picture is as the body says: eight conflicts with #979 (e01c6846) on the coverage manifest, contract digests, the H2 fixture and four test files; whichever lands second re-pins and regenerates the fixture over the union. The battery block on the stale deferral is María's ruling and sits outside this PR.

Approving. Merge on green CI, after settling the order with #979.