Skip to content

Receive qualification requests, queue them durably and replay in CI - #163

Draft
MaxGhenis wants to merge 10 commits into
w6/version-scopedfrom
w6/replay-receiver
Draft

MaxGhenis wants to merge 10 commits into
w6/version-scopedfrom
w6/replay-receiver

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Stacked on #162. It also carries #161, merged in, and #160's latest commit, until those land. WIP: W6 of the Autumn Budget 2026 CI plan.

What this adds

Receiver. .github/workflows/uk-replay.yml receives W4's uk-replay-qualification-request dispatches (interface: docs/uk_replay/QUALIFICATION_INTERFACE.md) and manual requests.

  • Persistent queue. The enqueue job (no concurrency group) writes each request to the replay-state branch before anything else (pipeline/uk_replay_queue.py). GitHub's cancellation of older pending runs therefore cannot lose a request. Enqueueing is idempotent.
  • Drain. One drain job at a time (concurrency group uk-replay-drain, two simulation workers at most) runs .github/scripts/uk_replay_drain.sh for the oldest request, through pipeline/uk_replay_qualify.py run-request:
    1. Schema check.
    2. Download the dataset with HUGGING_FACE_TOKEN and check its size and SHA-256.
    3. Download the model and core wheels without dependencies and check them against the request's SHA-256.
    4. Install those wheels, the wrapper (PyPI wheel or public commit) and the receipt tools. A certified pair gets its exact freeze; a new candidate resolves within the newest certified freeze.
    5. Probe the installed identity, the release bundle under packaged certification, the certification and the data years.
    6. Run the corpus, then diff against the newest other certified variant.
    7. Seal the receipt.
  • Rejection codes: schema_invalid, data_access_failed, artifact_hash_mismatch, release_bundle_mismatch, incompatible_pair, no_supported_years.
  • Status. A failure before the identity is known, or a crash, yields a failed receipt, so no request loops. The receipt is committed first. The terminal STATUS.json then names its digest and commit.
  • Catch-up. A schedule every three hours drains leftovers. A request still running after seven hours is retried.

Replay in CI (ci.yml):

  • Engine-free checks:
    • every committed namespace is re-staged and re-compared;
    • every committed diff is re-rendered with --check;
    • every seal is verified byte for byte, under the existing no-drift gate.
  • replay engine job, per certified variant:
    • installs the variant's exact freeze and verifies the installed environment;
    • runs every event's construction guards;
    • regenerates the committed premises and re-verified diagnostics byte for byte;
    • with the secret, replays one real measure-year and compares bytes.

Verified end to end, outside Actions

Using the actual subcommands against a released_bundle request for policyengine.py 6.2.6 built from its PyPI wheel hashes:

  • prepare downloaded and verified the three wheels and installed the certified freeze exactly.
  • probe recognised the environment as the committed certified variant (no new record).
  • corpus recomputed all five events in about 18 minutes with no failures. Artifact numbers were byte-identical to the earlier run except the identity digest.
  • The sealed namespace and diff land in the follow-up results PR.

Needs before it can run in Actions

  • HUGGING_FACE_TOKEN repo secret, requested from the coordinating session.
  • W4's GitHub App installed on scorecard for repository_dispatch (W4 asked the coordinating session).
  • The replay-state branch is created by the first enqueue (orphan).

Invariants and tests (tests/test_uk_replay_qualify.py, 18 tests)

  • Enqueue is idempotent. Invalid requests are recorded under their JCS digest for a rejection receipt.
  • Terminal states are final and name their receipt.
  • Pending is oldest-first, recovers stale runs, and never returns terminal requests (Hypothesis).
  • A golden candidate matches the committed certified pair. Exact (-r) versus constrained (-c) installs work as specified.
  • Wheel digests are enforced. Freezes normalise to exact pins. Variant ids get a digest suffix only on a collision.
  • Every pinned distribution is checked.
  • Rejected and pre-identity failed receipts validate. A complete receipt needs its execution.
  • Seals bind every file and the diff, and detect edits and additions. Sealed namespaces refuse writes.

axiom: n/a: replay CI and qualification plumbing in scorecard, no policy encoding.

🤖 Generated with Claude Code

MaxGhenis and others added 5 commits October 11, 2026 11:51
pipeline/diff_uk_event_runs.py compares two complete replay namespaces after
re-verifying each side's staging, comparison and artifact receipts. It
flags, at source-row, measure-year and PE-head level: numerical moves above
max(£1m, 1% of the larger magnitude); sign reversals and zero crossings;
coverage, mapping, support, inertness and construction changes regardless
of size. A year one side cannot support is a support flag, never coverage,
and only on rows that could carry a counterpart. Ratio bins are reported as
transitions. Flags are investigation prompts; nothing passes or fails.

Each flagged (event, measure, year) gets a packet with the OBR source
cells, both execution identities, artifact digests, executed reform
dictionaries, focused rerun commands and the sources the registry cites.
--check re-renders and compares bytes for CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rebuilt environment for the same three wheels resolved other pydantic, pydantic-core and
filelock versions and left out pypdf, and a raw RECORD digest also bound the
installer (uv_cache.json, console-script shebangs that name the venv). Now:

- identity.requirements_sha256 binds the exact freeze; verify_installed
  checks every pinned distribution at its version (extras allowed);
- record_sha256 excludes installer bookkeeping, bytecode and generated
  scripts outside site-packages, so pip and uv installs of one wheel agree;
- a freeze with local-wheel or commit installs is normalised to exact pins;
- a variant id takes an identity-digest suffix only when another identity
  already holds its version slug;
- compute_uk_event --variants-file selects from another variants document
  (a qualification run adds its candidate there).

Schemas: requested_by.workflow_run_id is optional (a catch-up run started
outside Actions has none); a failed receipt may have no execution block
(a step failed before the candidate's identity was known). Both relax v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.github/workflows/uk-replay.yml receives W4's uk-replay-qualification-request
dispatches (and manual requests). The enqueue job writes each request to
the replay-state branch first (pipeline/uk_replay_queue.py), so GitHub
cancelling an older pending run never loses one; the drain job runs one
request at a time, two simulation workers at most, and a schedule every
three hours drains what is left. A request still running seven hours after
its last update is retried.

pipeline/uk_replay_qualify.py run-request takes one request end to end:
schema check; the dataset from Hugging Face, checked by size and digest;
the model and core wheels downloaded and checked against the request, then
installed with the wrapper and, for a certified pair, that pair's exact
freeze (a new candidate resolves within the newest certified freeze); a
probe of the installed identity, the release bundle under packaged
certification, the certification and the data years; the corpus; the diff
against the newest other certified variant; and a sealed receipt
(NAMESPACE_MANIFEST.json, REPLAY_RECEIPT.json). Every outcome, including a
rejection or a crash, produces a receipt. The terminal STATUS.json names
the receipt digest and the commit that added it.

ci.yml now re-stages and re-compares every committed namespace, re-renders
every committed diff (--check) and verifies every seal; and a replay engine
job per certified variant installs its exact freeze, verifies the installed
environment, runs the construction guards and regenerates the committed
premises and diagnostics, and with the HUGGING_FACE_TOKEN secret replays
one real measure-year and compares bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rebuilt environment for the same three wheels resolved other pydantic, pydantic-core and
filelock versions and left out pypdf, and a raw RECORD digest also bound the
installer (uv_cache.json, console-script shebangs that name the venv). Now:

- identity.requirements_sha256 binds the exact freeze; verify_installed
  checks every pinned distribution at its version (extras allowed);
- record_sha256 excludes installer bookkeeping, bytecode and generated
  scripts outside site-packages, so pip and uv installs of one wheel agree;
- a freeze with local-wheel or commit installs is normalised to exact pins;
- a variant id takes an identity-digest suffix only when another identity
  already holds its version slug;
- compute_uk_event --variants-file selects from another variants document
  (a qualification run adds its candidate there).

Schemas: requested_by.workflow_run_id is optional (a catch-up run started
outside Actions has none); a failed receipt may have no execution block
(a step failed before the candidate's identity was known). Both relax v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e983891)
@vercel

vercel Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policyengine-scorecard Ready Ready Preview Oct 11, 2026 8:20pm UTC

Request Review

MaxGhenis and others added 2 commits October 11, 2026 16:06
…ruction

On the first real premise diff (2.89.2 to 2.125.1) every reversal's reform
world, which is current law, moved with current law: the employer-NIC
secondary threshold (96 to 96.153846), the Class 2 small-profits
threshold, float noise in the blended CGT rates. Those were raised as
construction flags, which release qualification proposes to fail on. A
baseline or reform world value change is now a construction flag only
where one side's world overrides that side's current law; otherwise the
move is recorded in current_law_changed context. Premise-check changes
(the CGT annualisation premise) still flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 11, 2026 16:10
The variants file is the index (no INDEX.json); a sealed namespace holds
NAMESPACE_MANIFEST.json and REQUEST.json; reruns of a sealed variant use an
explicit output directory rather than a rerun-<n> suffix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 6af674da Deployed Oct 11, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant