Skip to content

UK source stages on the shared manifest: source_stages.json, operation handlers, and the StagePlan migration #677

Closed

Description

@juaristi22

Workstream E1 of the UK-migration master epic #665; child of #145. Lands alone, before every other WS-E child.

Why

The UK country package declares hmrc_income_source_stages.json — a bespoke manifest whose 10 operation kinds are absent from ALLOWED_SOURCE_OPERATION_KINDS and which only uk_runtime/hmrc_source_contract.py validates — so load_country_spec("uk").sources is None and country_stage_plan refuses UK. UK stages run as UKNationalStage (no produces/consumes/donor declarations) instead of the shared plan.Stage/StagePlan. Every WS-E stage needs the shared machinery: the closed operation vocabulary, produces/consumes enforcement, StageRecord.nonzero_share (the per-stage parity evidence carrier), and the donor graph.

Scope

  • New build/uk/source_stages.json — a real SourceManifest, initially carrying the two existing stages, declarations copied from hmrc_income_source_stages.json. Absorb by copy, not move: the old file stays byte-identical (the coverage manifest and the frozen June evidence sidecars pin it); a lockstep test fails if the copies diverge; the original retires at the FRS 2024-25 refresh.
  • Add the 10 existing HMRC operation kinds to ALLOWED_SOURCE_OPERATION_KINDS in build/source_manifest.py.
  • New uk_runtime/source_runtime.py::uk_source_operation_handlers() mirroring us_runtime/source_runtime.py — handlers delegate to the existing executable modules, so dispatch moves but behavior does not.
  • Migrate build_uk_national_dataset from Sequence[UKNationalStage] to plan.Stage/StagePlan via country_stage_plan(load_country_spec("uk"), implementations). Stage names unchanged so checkpoint identities remain valid; UKNationalStage kept as a deprecated shim for one release.
  • Thin, country-parameterized rules-engine adapter: frame/adapters/policyengine_uk.py implementing the shared frame/rules.py protocol (variable metadata + materialization only for now), plus a shared materialize_rules_engine_predictors operation kind. The engine-country-must-match-dataset-country check lives in the shared layer; each stage declares its materialized predictors in the manifest so the dependency stays attested.
  • Known landmine: FORBIDDEN_SOURCE_DEPENDENCIES substring-matches policyengine-uk-data inside the HMRC manifest's reviewed_source citation strings. Rephrase the citations in the copy (keep the DOI and UKDS study number); do not weaken the rejection.

Acceptance

  • load_country_spec("uk").sources is non-None; country_stage_plan assembles the two-stage plan.
  • A staging rebuild from the same inputs is payload-identical to a pre-migration build (tools/compare_uk_h5_payload.py exit 0); all 13 gate verdicts unchanged.
  • The lockstep test pins the copied declarations to the frozen original.

UK-specificity note (required in the PR)

Expected UK-specific surface: the adapter module (country parameterization is shared), the UK handler map, and the copied stage declarations. Everything else (op-kind allowlist entries, the materialize_rules_engine_predictors op, StagePlan use) is country-agnostic. The PR lists anything beyond this and why.

Coordination

#666 (Logbook adoption) touches the same driver — coordinate merge order.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions