Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 24 additions & 6 deletions docs/uk_replay/RELEASE_REPLAY.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,15 @@ Observed on 2.125.1 on 11 October (parameter tree only):

`pipeline/diff_uk_event_runs.py --base <variant or root> --head <variant or root>`
compares two complete namespaces, engine-free, after re-verifying each side's
staging and comparison receipts. Keys:
staging and comparison receipts against that side's receipt-bound registry
snapshot. The current event registry does not replace a historical snapshot.
Every inventoried artifact is checked by the staging and comparison validators,
including artifacts that supply no numeric source counterpart. Each run
manifest's execution identity, registry and artifact inventory, supported and
unsupported years, and measure-year grids must agree with the verified staging
rows and tally. Inertness comes from the staging manifest named and verified by
comparison provenance. Inconsistent receipts are refused before a diff is
written. Keys:

* source row: `(event, source_row_id)`;
* measure-year total: `(event, measure_key, year)`;
Expand All @@ -289,9 +297,9 @@ bin transitions.
|---|---|
| `numerical` | Both sides numeric and `abs(head − base) > max(£1,000,000, 0.01 × max(abs(head), abs(base)))`. Strictly greater. |
| `structural` | Sign reversal or a zero crossing (zero means `abs(x) ≤ £0.50`, accounting roundoff); artifact schema or orientation changed; non-finite value. Regardless of size. |
| `coverage` | A counterpart, artifact pair or source row present on one side only, in a year both sides support. |
| `mapping` | A row's `head_variables`, an artifact's `head_channels` or `head_definitions`, or the registry SHA-256 differs. |
| `support` | A year supported on one side only. |
| `coverage` | A counterpart, artifact pair or source row present on one side only, in a year both sides support. The shared-support requirement also applies to rows outside household scope or without an expressible counterpart. |
| `mapping` | A row's `head_variables`, an artifact's `head_channels` or `head_definitions`, or the registry SHA-256 differs. Registry changes flag the event's affected records with both digests. |
| `support` | A year supported on one side only. Source rows that cannot carry a counterpart do not receive this flag. |
| `inertness` | `inert_construction` on one side only, or identical worlds in an in-force year on one side only. |
| `construction` | Executed `baseline_world`/`reform_world` reform dicts differ, or a premise check's result differs. A changed current-law value of a touched parameter is recorded as context on the row, not flagged: it is what a model release is expected to change. |

Expand All @@ -300,14 +308,24 @@ recorded once per diff; they explain moves and are not flags on rows.

Outputs: `DIFF.json` (canonical, sorted), `DIFF.md`, and one issue packet per
flagged `(event, measure, year)` under `packets/`. A packet holds the source
cells (table, column, row number, source-row id, OBR value), both execution
cells (table, column, row number, source-row id, OBR value) and measure details
from each registry with base/head attribution, both execution
identities, registry and artifact digests, the executed reform dicts, the
affected heads with base/head values, focused rerun commands for each
variant, the statutory sources the registry already cites, and empty
variant using that side's verified registry path and variant selection, the
statutory sources each registry already cites with side attribution, and empty
`minimal_case` and `statutory_evidence` sections for the investigating lane.
Removed measures retain their base title, source cells and citations.
Packets are evidence for the coordinating session, which files upstream
issues; this pipeline files none.

Diff invariants are checked with deterministic cases and Hypothesis properties
in `tests/test_uk_event_diff.py`: identical sides raise no flags; swapping sides
preserves each record's flag set and mirrors its values; every source row in
either side appears once at source-row level; every flagged record appears in
exactly one packet, including support-only records; numerical moves flag only
strictly above the stated threshold; and coverage and support do not overlap.

## Qualification requests and receipts

See [QUALIFICATION_INTERFACE.md](QUALIFICATION_INTERFACE.md) for the request
Expand Down
14 changes: 11 additions & 3 deletions pipeline/compare_uk_event.py
Original file line number Diff line number Diff line change
Expand Up @@ -1321,8 +1321,14 @@ def render_summary(
return "\n".join(lines)


def load_verified_comparison(path: Path, *, artifact_root: Path = ROOT) -> list[dict]:
"""Bind summary-only reads to current source, stage, axes and artifact bytes."""
def load_verified_comparison(
path: Path, *, artifact_root: Path = ROOT, registry_path: Path | None = None
) -> list[dict]:
"""Verify receipts against the current registry or an explicit snapshot.

Historical readers may select the receipt-bound registry snapshot. The
default preserves summaries' binding to the current event registry.
"""
provenance_path = path.parent / "COMPARISON_PROVENANCE.json"
if not provenance_path.exists():
raise EventComparisonError("summary requires COMPARISON_PROVENANCE.json")
Expand Down Expand Up @@ -1354,7 +1360,9 @@ def load_verified_comparison(path: Path, *, artifact_root: Path = ROOT) -> list[
raise EventComparisonError("comparison replay-grid receipt differs")
if registry.get("event_slug") != path.parent.name:
raise EventComparisonError("summary registry event differs")
canonical_registry = (
if registry_path is not None and registry_path.resolve() != paths["registry"]:
raise EventComparisonError("comparison registry snapshot path differs")
canonical_registry = registry_path or (
artifact_root / "data" / "uk" / "events" / f"{path.parent.name}_measures.json"
)
if (
Expand Down
Loading
Loading