Repository navigation
UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135
Description
Lanes: ordinary maintainer tooling under db/; if the suite wiring touches policyengine_chronicle/** consumer surfaces, that PR runs under ledger-contract-maintainer (.github/chronicle-agents.yml)
Related: the academic_year period type landed via #131/#136; the wave packages this depended on are all merged (#137/#139/#141/#155/#159/#163, plus the sex-split facts in #173)
Revised 2026-08-13 for the #166 ruling (selection contracts live in the consumer; Chronicle is facts-only): PR 2 no longer embeds target profiles — the UK consumer artifact is a facts-only feed, and everything profile-shaped now lives with the contracts in microcosm (microcosm#707/#708) or retires under #172. PRs 1 and 3 are unaffected; PR 3 is now unblocked.
Three small, independent enablers for the UK target migration.
PR 1 — de-hardcode db/pe_source_inventory.py
DEFAULT_PE_UK_ROOT = Path("/Users/maxghenis/PolicyEngine/policyengine-uk-data") (and the US twin) key the UK ingestion inventory to one person's checkout. Replace with env vars / CLI arguments with no default, and fail with a clear message when unset. The inventory's UK rows (ons 13, hmrc 6, dwp 5, obr 3, voa 2, nrs 2, nomis 2, slc 1, nts 1) were the working checklist for waves 1–3; the waves are done, but the inventory remains the supersession cross-check for PR 3 and for any backfill lane.
PR 2 — UK consumer-artifact suite (facts-only)
Add the UK suite wiring so chronicle build-suite / build-consumer-artifact can produce a hash-pinned facts-only UK bundle covering the wave 1–3 packages: consumer_facts.jsonl + manifest hashes, no embedded profiles. This is the feed microcosm#622 consumes (ledger_facts_sha256-pinned) to compile the UK contract (uk/uk_national_targets.json, microcosm#707) into a target registry — per the #166 ruling the artifact carries facts and provenance, and nothing selection-shaped. Per-profile embedding and coverage.json are being retired from the artifact under #172; do not build UK wiring against them.
Known behavior to design around (observed on the US side, June 2026): a single---year bundle build drops year-pinned sources; the working pattern is per-suite build at each pinned year + concatenated consumer_facts.jsonl. Either fix the year handling properly here or codify the per-suite+concat pattern for the UK suite — pick one and document it.
PR 3 — retire the legacy hardcoded UK ETLs (unblocked)
db/etl_obr.py and db/etl_ons.py are hardcoded March-2024 dicts feeding the legacy three-table schema. docs/agent-source-package-harness.md already declares the old ETL loaders "compatibility and migration inputs, not the preferred agent" lane. The #133 OBR/ONS packages are merged, so this is ready: delete these two modules (or reduce them to explicit tombstones pointing at the packages), so no future agent mistakes them for the ingestion path. Verify the wave-2 packages supersede every value they carry before deletion.
Acceptance
- Inventory runs against arbitrary checkout paths; CI proves the no-default failure mode.
- One documented UK suite command produces a hash-pinned, facts-only UK consumer artifact (no profile surfaces referenced).
- Legacy UK ETLs gone or tombstoned, with supersession verified in the PR description.
Activity
Add OBR EFO March 2026 source packages: receipts and expenditure (Pol…
Add OBR EFO March 2026 source packages: receipts and expenditure (Pol…
Add OBR EFO March 2026 source packages: receipts and expenditure (Pol…
juaristi22 commentedon Aug 20, 2026
Both PRs are up, implementing the revised scope in two diffs split by review nature:
db/etl_obr.py/db/etl_ons.pyretirement under the audit doctrine (map → port → register): 9 of 11 OBR macro series ported via two new EFO March-2026 packages (obr-efo-economy-march-2026,obr-efo-aggregates-march-2026), household totals + average household size ported ontoons-families-households-2025, and the residual gaps registered first-class in the checklist audit table (real-GDP-growth derivation, PSND-in-GBP, FY2024-25 aggregates rows,etl_hmrc.pypending its own audit). Bundle surface re-measured (145,295 facts / 122 packages). Publish-raw for the two new workbook artifacts is pending credentials.build-bundle --suite uk(curated 66-alias tuple + drift test;--yeardocumented inert for the UK suite), facts-only consumer artifacts ("profiles": {}— the post-Adjudicate: do target profiles and measurement bindings belong in Chronicle? The boundary records disagree #166 artifact shape, prerequisite Retire the target-profile surface: Chronicle is facts-only (#166 ruling) #172 inherits), and the inventory de-hardcode (env/CLI roots, jurisdiction-scoped, loud failures, prune guard). End-to-end evidence in the PR: 107,515 UK facts → facts-only artifact → loaded hash-pinned by microcosm'sload_ledger_consumer_artifact.Sequencing: merge #181 first, then #186 (stacked).
Retire the legacy UK ETLs onto publisher-backed facts (PolicyEngine#135)
UK consumer feed tooling: suite, facts-only artifacts, inventory roots (
Retire the legacy UK ETLs onto publisher-backed facts (PolicyEngine#135)
UK consumer feed tooling: suite, facts-only artifacts, inventory roots (
UK consumer feed tooling: suite, facts-only artifacts, inventory roots (
Retire the legacy UK ETLs onto publisher-backed facts (PolicyEngine#135)
UK consumer feed tooling: suite, facts-only artifacts, inventory roots (
Merge pull request #181 from juaristi22/uk-135-etl-retirement
Merge pull request #186 from juaristi22/uk-135-feed-tooling