Repository navigation
Conversation
Contract for certified replay variants, exact validation, unsupported years, per-release namespaces, version-scoped diagnoses, construction premises, the release diff and the W4 qualification interface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
JSON Schemas for the W4 qualification request, the replay receipt and the replay-state status file; RFC 8785 (JCS) request_id and receipt_sha256; golden fixtures with pinned bytes and digests, including the JCS number and non-ASCII probes W4 asked for. Adds rfc8785 and jsonschema to the CI test environment. Records the amendments W4 accepted on 11 October and the current certified variant's exact requirements freeze. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
data/uk/replay_variants.json records two certified replay variants: the legacy populace-2023 world (policyengine 5.0.2, policyengine-uk 2.89.2) and policyengine.py 6.2.6's pair (policyengine-uk 2.125.1, uk-data 1.58.0). pipeline/uk_replay_variants.py checks a record's consistency, the installed wheels (versions, every installed file against RECORD, the packaged manifest), the release bundle under packaged certification, and the dataset bytes, and refuses any other combination. compute_uk_event.py takes --variant. A non-legacy variant runs the registry years its population supports, records the rest as unsupported, writes to results/uk/replays/<variant_id>/ and refuses sealed namespaces. Every managed simulation now refuses a year outside its dataset's years: on the 2024 enhanced FRS a 2023 calculation returns defaults without raising. Staging keeps unsupported-year rows as unsupported_on_variant. The legacy variant's outputs are byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NAMESPACE_MANIFEST.json lists the SHA-256 of every file in a sealed result namespace; the receipt names its digest. Additive and optional, so the schema stays v1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 11, 2026
A rebuilt environment for the same three wheels resolved other pydantic, pydantic-core and filelock versions and left out pypdf, and a raw RECORD digest also bound the installer (uv_cache.json, console-script shebangs that name the venv). Now: - identity.requirements_sha256 binds the exact freeze; verify_installed checks every pinned distribution at its version (extras allowed); - record_sha256 excludes installer bookkeeping, bytecode and generated scripts outside site-packages, so pip and uv installs of one wheel agree; - a freeze with local-wheel or commit installs is normalised to exact pins; - a variant id takes an identity-digest suffix only when another identity already holds its version slug; - compute_uk_event --variants-file selects from another variants document (a qualification run adds its candidate there). Schemas: requested_by.workflow_run_id is optional (a catch-up run started outside Actions has none); a failed receipt may have no execution block (a step failed before the candidate's identity was known). Both relax v1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e983891)
3 tasks
The variants file is the index (no INDEX.json); a sealed namespace holds NAMESPACE_MANIFEST.json and REQUEST.json; reruns of a sealed variant use an explicit output directory rather than a rerun-<n> suffix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 11, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Independent review of the variant runner found trust gaps; each is now closed and tested. - Staging took a manifest's variant_id and unsupported years on trust: a self-consistent manifest could stage a 2023 artifact from a population with no 2023 inputs, and dropping variant_id switched the artifact check off. Staging now loads the record from the variants document (--variants-file), requires its identity digest, data years and the supported/unsupported partition that record gives, and requires artifact variant fields to match the manifest (none for legacy). - --resume gathered receipts for every registry year; it now reads only supported years and refuses a receipt naming an unsupported one. - Staging, comparison and the summary now refuse a sealed namespace, as compute already did. - record_sha256 binds RECORD-listed files, but policyengine-core loads every file in a parameter directory and every module in a variable directory (parameter_node.py, tax_benefit_system.py). The installed check now also refuses files in the three packages' directories that no installed RECORD lists; engine bytecode is compiled from source into an empty sys.pycache_prefix with writing off (workers inherit it), and each imported package must be its verified distribution's own files. - Removing tokens forced the packaged certification only for a private data repository: policyengine.py 6.x fetches the release manifest with requests, which ignores HF_HUB_OFFLINE. The isolated environment now also points the HTTP(S) proxy variables at a closed local port and drops NO_PROXY, so the fetch fails for a public repository too. - Only certified variants run; a qualification run passes --allow-uncertified; the default must be certified. - The legacy variant's installed environment is now checked before its preflight, and --dataset-file is refused there instead of ignored. - A variant simulation whose dataset does not state its years fails instead of skipping the year check (legacy unchanged). - Variant rows name their own population instead of "a single certified 2023 population"; legacy text is unchanged. - Record fields of the wrong type raise VariantError; a symlinked runtime data directory is refused. Verified: the five legacy events restage and recompare byte-identically; both pinned venvs (.venv-candidate 6.2.6, .venv-replay 5.0.2) reproduce their committed records with no unlisted files; a real focused 6.2.6 replay (AB2024 CGT, 2025) through every new check is byte-identical to the committed artifact (sha256 8316a941...). Full suite: 1521 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- rejected: variant_id, variant_identity_sha256 and execution must be null and integrity not_run (nothing ran). - complete: variant_identity_sha256 must be a digest and results.manifest_sha256 must be present (every complete receipt is sealed; results.commit stays optional because the commit binding lives in STATUS.json). - execution.python is anchored; a request's data.path refuses ".." segments. - The golden receipt gains a placeholder results.manifest_sha256; its JCS bytes and receipt_sha256 are regenerated (new digest 5cbc44c1...). The golden request and its request_id are unchanged. - expected_release_bundle's description and agreed point 4 now name the full isolation (tokens and NO_PROXY removed, proxies at a closed port), pending W4's confirmation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of dc040de: - The Modal image allowlist lacked pipeline/uk_replay_variants.py and data/uk/replay_variants.json, which compute_uk_event has imported and read since this branch began, so every remote run would have failed at import. Both are added; a test walks the worker's pipeline imports and the files a legacy run reads and requires each in the allowlist. - The unlisted-file scan could be bypassed. policyengine-core's loaders recurse into every subdirectory, __pycache__ included (tax_benefit_system.py: glob "*/"), and follow symlinked ones (parameter_node.py: os.path.isdir). The scan now skips only *.pyc inside a __pycache__, reports every symlink, and counts a file as listed only in its own package's RECORD, by unresolved path. A *.pth startup file beside the packages must belong to a pinned distribution or be the virtual environment's own shim (site.py runs their import lines). - The symlink guard in isolate_bytecode resolved the path first and never fired; it now checks before anything resolves or removes it. - The legacy variant now runs under the same certification isolation and the same exact release-bundle comparison as any other variant; --guards compiles the engine from source and checks import origins too. - Staging also checks the manifest's dataset digest and data bundle against the record. - installed_distributions keeps the first copy of a name on sys.path, as Python's import does. Found by running this on Modal: its containers keep their own client packages behind the pinned environment, and counting the later copy refused a correct environment. - Legacy preservation is now tested: staging, comparison and the summary for all five events regenerate from committed artifacts and match the committed bytes; no legacy artifact, manifest or receipt carries a variant field. - Docs: the checks' real order; what the identity leaves out and why; what the installed check does not bind; the seal manifest and the CI job are marked as arriving with the receiver change; interface point 4 is marked confirmed by W4 (policyengine.py#574). Verified: a focused legacy replay on Modal (AB2024 CGT, 2025) through every check is byte-identical to the committed artifact (sha256 f8dcfdd9...); the 6.2.6 --guards run and focused replay pass locally with an artifact byte-identical to the committed one (8316a941...); both pinned venvs reproduce their records. Full suite: 1536 passed before the last fix, then the changed files' 169 tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Oct 12, 2026
After #160, staging checks a manifest's variant against its record in a variants document, and the diff re-stages each run to revalidate it. The diff now takes --variants-file (default data/uk/replay_variants.json) and passes it to staging, so a qualification run can diff a candidate whose record lives only in its work file. The diff tests build synthetic variants (several share an id with different data years), so they stand in a record derived from each manifest; staging's variant checks are tested in test_uk_replay_variants.py, and a new test checks the file reaches staging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP (W6 of the Autumn Budget 2026 CI plan). Draft from the first commit so the work is visible to the other workstreams and hubs.
What this PR will do
Make the fiscal-event replay runner execute on any certified replay variant (a policyengine.py bundle's exact wrapper/model/core/data identity), validated exactly, instead of the hard pin to policyengine-uk 2.89.2 on populace-uk-2023.
Design contract:
docs/uk_replay/RELEASE_REPLAY.md. Proposed W4 interface:docs/uk_replay/QUALIFICATION_INTERFACE.md.Plan (this PR):
data/uk/replay_variants.json+pipeline/uk_replay_variants.py: variant records, engine-free consistency checks, installed-environment checks (versions, dist-info RECORD hashes, packaged manifest hash), release-bundle equality under packaged certification, dataset materialisation per wrapper generation.compute_uk_event.py/modal_uk_event.py/uk_engine_registry.py/ staging take--variant; unsupported years reported (unsupported_on_variant), never computed.results/uk/replays/.request_idfixture for W4.Follow-up PRs: release diff (
diff_uk_event_runs.py) and issue packets; version-scoped diagnoses; construction-premise audit; receiver/queue/catch-up workflow and CI integration; the five-event rerun on 6.2.6/2.125.1/1.58.0 with its diff report.axiom: n/a: replay infrastructure in scorecard, no policy encoding.
🤖 Generated with Claude Code