Skip to content

Register the ONS / NRS / NISRA small-area geography lookups and census counts behind microcosm's UK atomic-area supports (raw-only, no facts) #269

Closed

Description

@juaristi22

Why. microcosm is rebuilding its UK atomic-area supports (E&W 2021 Output Areas, Scotland 2022 Output Areas, NI 2021 Data Zones with their LSOA/MSOA, local authority, ward, constituency, region and ITL layers) from the latest published lookups, for the geography-first reordering tracked in PolicyEngine/microcosm#931. The supports are derived build artifacts (the CONGRESSIONAL_DISTRICT_VINTAGE_CROSSWALK.md rule: regenerable, sidecar provenance, never a Ledger fact), so they stay in microcosm. Their publisher inputs are exactly what Chronicle's governance says it may hold: "register raw publisher artifacts and checksums" (docs/chronicle-governance.md), with the bytes in git plus ledger-raw R2 and the source URL, sha256, size, vintage and retrieval date in the package manifest. The precedent is statbel-nis-2025-commune-crosswalk (db/data/statbel/nis_2025_commune_crosswalk/manifest.yaml): a manifest-only registration with no source_package.yaml. docs/pe-uk-source-checklist.md line 97 already records that "Microcosm's sha-pinned OA ladder keeps its own path until consumption switches"; this issue registers the path's inputs without changing who consumes them.

Pairing rule. Each file is fetched once into microcosm's cache (~/.cache/populace-uk-geography), registered in Chronicle from those exact bytes with chronicle fetch-artifact, and consumed by microcosm's support tool through a --source-manifest pointing at the same files, so both repos pin one sha256 per file by construction. microcosm's provenance JSON carries each file's chronicle_source_id, chronicle_package_id and chronicle_year; a later cross-repo check diffs the two registers.

Ask: sixteen raw-only packages (source_id / package_id / year / source_url / vintage). Sha256s are as fetched by microcosm (2026-08-19, 2026-09-09 and 2026-09-15) and pinned in its uk/uk_atomic_area_supports.provenance.json; registration from the same bytes should reproduce them.

England and Wales (source_id ons):

  1. ons-oa21-lsoa-msoa-lad22-lookup / 2021 — Output Area (2021) to LSOA to MSOA to LAD (December 2021) Exact Fit Lookup, ONS Open Geography Portal item b9ca90c10aaa4b8d9791e9859a38ca67; vintage 2021_census; sha256 1b7e715936374c586efbe669d82e3f3d6d093dc5295c2d99ce6ea2349f85153a (18,193,283 bytes, cached copy ew_oa21_lsoa_msoa_lad22_hierarchy.csv).
  2. ons-oa21-parncp-lad-rgn-ctry-dec2024-lookup / 2024 — Output Area (2021) to Parish/NCP to LAD to Region to Country (December 2024) Lookup V2, item 7507c0292db546ed83e4ba60f1115b1d; carries the April 2023 LAD code set as LAD24CD and RGN24CD; vintages 2023_april_lad, 2024_rgn; sha256 e01842a57c26036641fa5d6c2769dd5ac67af03a64d6790d4f35883f04eb2694 (27,249,403 bytes, cached copy ew_oa21_parncp_lad24_rgn24_dec2024.csv).
  3. ons-oa21-pcon-jul2024-lookup / 2024 — Output Area (2021) to Westminster Parliamentary Constituency (July 2024) Best Fit Lookup, item 5968b5b2c0f14dd29ba277beaae6dec3 (fields now labelled PCON25CD, boundaries unchanged); vintage 2024_pcon; sha256 e4ec577416bde73d5718ef133af4adeaae9b95a41867aeb006b5fc0260c790c0 (12,879,604 bytes; unchanged since the ladder build despite the field rename).
  4. ons-oa21-wd24-lad24-lookup / 2024 — Output Area (2021) to Ward (2024) to LAD (May 2024) Best Fit Lookup V2, item b21c632804094c149ccdfea1ae59c8c8; vintage 2024_wd; sha256 f2d2df158c6e3e29da09a96d456c718a065e1229590d27f6f5349e45689d7b67 (12,628,147 bytes, cached copy ew_oa21_ward24_lad24_bestfit.csv).
  5. ons-lad-dec2024-itl-jan2025-lookup / 2025 — Local Authority District (December 2024) to LAU1 to ITL3 to ITL2 to ITL1 (January 2025) Lookup in the UK, item 15bdb6b0ff1c4a64b34a64e2e39f8caf; vintage 2025_itl; covers E/W/S12/N09 codes (361 LADs; North Ayrshire appears on two ITL3 rows); sha256 2db528a486fdefebfb03a3fb89d706d037cd10f06692c98cdef60a9b7796b284 (45,097 bytes, cached copy uk_lad24_itl25_lookup.csv).
  6. ons-census2021-ts001-oa / 2021 — Census 2021 TS001 usual resident population by Output Area (Nomis bulk zip census2021-ts001.zip); sha256 b724c1208b7628c7c558156fcc619446486353a12d0b6e0bf388add225653dc6 (1,870,128 bytes).
  7. ons-census2021-ts041-oa / 2021 — Census 2021 TS041 number of households by Output Area (Nomis bulk zip census2021-ts041.zip); sha256 e4f11fc0ea22393fe3be444027878e67b51d2a44687e122c37439d19a548cb9f (1,511,641 bytes).

Scotland (source_id nrs):

  1. nrs-oa22-dz22-iz22-lookup / 2022 — oa22_dz22_iz22.zip (OA 2022 to Data Zone 2022 to Intermediate Zone 2022); vintage 2022_census; sha256 3565794561306479b6c618aaa3915f1f29fdce31ee798fdcdab45f880a7ba4aa (237,841 bytes; the server copy was re-uploaded on 3 Nov 2025 and the cached bytes still match).
  2. nrs-oa22-lau25-itl25-lookup / 2025 — oa22_lau25_itl25.zip (OA 2022 to LAU 2025 to ITL 2025, bridged to Council Area 2019); vintages 2019_council_area, 2025_itl; sha256 4d94e959dfcc83ce6bcd0c09cfc073ad597fb89812eab1d8ab664f4ccbc6ba77 (369,739 bytes).
  3. nrs-oa22-ukpc24-lookup / 2024 — oa22_ukpc24.zip (OA 2022 to UK Parliamentary Constituency 2024); vintage 2024_pcon; sha256 b63829d28cdf93606d17ffd12999ec12f54a8bc3065cc40fbf9e975629e4689a (181,009 bytes).
  4. nrs-census2022-oa22-usual-resident-population / 2022 — outputarea2022_usualresidentpopulation.csv; sha256 f7af756710c56f335d9332c08f95a68a5775aec4f25f17b21ab979257ed40148 (678,338 bytes).
  5. nrs-census2022-index / 2022 — census_2022_index.zip (OA to higher areas incl. electoral ward 2022; Postcode_To_OA household counts, cell-key perturbed); vintages 2022_census, 2022_ew; sha256 0e4096a321cba2318dc5d2e35c197bf815ac333aebdf4cb0ddce82112b57a261 (2,922,015 bytes).

Northern Ireland (source_id nisra):

  1. nisra-dz2021-geojson / 2021 — geography-dz2021-geojson.zip (Data Zone 2021 boundaries with SDZ2021, LGD2014, DEA2014); sha256 3acb357ebdecae64dbc2fb645e62c68f529a00f7ae9c19684bd3b7a10e62676b (26,823,947 bytes).
  2. nisra-census2021-dz21-people / 2021 — NISRA table builder table.csv?d=PEOPLE&v=DZ21; sha256 5fea5eb724034b4d8d749c67114d903383074dca090e8203a8e819a0d6dd445b (108,660 bytes; dynamic URL; NISRA flags MS-A01 as [UPDATED]; the R2 copy is the durable bytes).
  3. nisra-census2021-dz21-households / 2021 — NISRA table builder table.csv?d=HOUSEHOLD&v=DZ21; sha256 fa97a7f61036f9a3ec40ea6e3812249d87d0010375f0ff7df0b5d9f77864563d (108,641 bytes; same caveat, MS-E01).
  4. nisra-dz-sdz-lookups-v3 / 2025 — geography-data-zone-and-super-data-zone-lookups-v3.xlsx (sheet DZ2021_Admin_geog_lookup, PARLCON2024); vintage 2024_pcon; sha256 8e2e1f6daaddd2f5b6887ccab1bf9d0bf179f6d0d2ba60ef8e0ebb495a3e1999 (603,543 bytes; already the pin in Bind UK census households from Chronicle and map NI Data Zones with NISRA's published lookup (#887) microcosm#900 / #887).

Retired inputs, not to register: OA21→LAD (April 2023) item 83982ff4…, LAD22→Region 2022 item 78b348cd…, OA21→Ward 2022 item 7207b517…, LAD (April 2023)→ITL (January 2021) item 02b49429… (superseded by 2, 4 and 5 above).

What this issue does not ask. No facts, no source_package.yaml, no consumer-artifact change, no consumer channel for factless artifacts (microcosm keeps reading its own provenance register), and no change to which repo owns the derived supports.

Microcosm side: the supports built from exactly these bytes, the per-file register and the pairing are in PolicyEngine/microcosm#931's first PR (tools/build_uk_atomic_area_supports.py, uk/uk_atomic_area_supports.provenance.json).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions