Skip to content

Ingest the full raw FRS table set and assemble the UK spine as a Frame #678

Closed

Description

@juaristi22

Workstream E2 of the UK-migration master epic #665; child of #145. The new build entry point: every existing UK stage transforms a loaded spine; this one creates it.

First task: raw-vintage audit (blocking)

Enumerate and sha+size-pin the FRS 2023-24 tabs, WAS round 8, LCFS, ETB, and SPI vintages in the private HF repo. The parity bar is frozen at the incumbent's enhanced_frs_2023_24.h5, so the port builds at the 2023 vintage first. Any missing vintage is a hard stop — escalate to #665 before proceeding.

Scope

  • Port the incumbent's 14-table FRS read (adult, child → derived person, benunit, househol, pension, oddjob, accounts, job, benefits, maint, penprov, chldcare, extchild, mortgage) as declarative artifacts in uk/source_stages.json, each tab sha+size-verified before open (the existing put2223uk.tab pattern).
  • Entity/ID construction (sernum → household_id; benunit = hh×100+n; person = hh×1000+n) including the household-ID sort invariant (policyengine-uk-data#436 — the raw household table is not sernum-ordered from 2024-25 onward; positional reads scramble the grossing weight).
  • gross4 → typed design weights; demographics, region, tenure, raw council-tax columns; benefit-code mapping to *_reported columns; income mapping using WEEKS_IN_YEAR uniformly (the incumbent uses bare *52 at three call sites — fix, and record the deltas as signed differences).
  • New uk_runtime/frs_spine.py + a driver mode producing a Frame with design-kind weights and provenance bound to the raw-tab fingerprints.

Acceptance

  • Spine passes validate_uk_national_frame; per-column nonzero shares vs the raw-only columns of efrs_parity_reference.json within declared tolerances or signed.
  • Two runs byte-identical via checkpoint digests; every artifact pinned in the manifest.
  • Every build attempt lands a Logbook row (Record every UK build attempt in Logbook #666 must be merged first).

UK-specificity note (required in the PR)

FRS table semantics and the ID scheme are genuinely UK; the read/verify/assemble operations should be generic kinds. The PR lists any UK-specific code beyond the declarations and why.

Depends on: E1 (shared manifest machinery), #666.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions