Per-line release pointers, the UK 2024-25 national default and its promotion command (#823)

[juaristi22]

Part of #823 (publication). Stacked on #965: the base is that PR's branch, so CI runs here only once #965 merges and this PR is retargeted to main. Three commits, one per increment of the reviewed plan; the shape follows Max's ruling on #823 (comment 5730335820) and the five proposals posted on 2026-09-18 (comment 5731044925), which this PR assumes and can be revised against his answers.

P1 — per-line release pointers (microcosm-data)

latest.json stays the frozen repository-global pointer on the June 2023 release. Each promotable UK line gets its own schema-1 pointer, latest-<line>.json, whose payload adds the discriminator line and the immutable cut tag as revision, so a line can follow a per-cut tag with no schema bump.

• release.py: line_pointer_path, line_pointer_payload, latest_line_release beside the latest.json and latest-evidence.json families; LatestPointer gains revision (defaults to the release id) and line; latest_release refuses line and revision fields; prepare_release(line=…) refuses the evidence tier, --no-latest and --tag-only, refuses a line that is not the release id's, reserves every latest*.json root path, and narrows the Promotion for the constant-id national line: pointer and certified-loader extension #823 refusal to "latest.json may not name a per-cut tag; a line pointer may"; _publish_atomic writes the chosen pointer last; the Slack alert names the promoted line and revision.
• contract.py: line_for_release_id (microcosm-uk-2024-25-national → national, microcosm-uk-2024-25-local-k<N> → local-k<N>, everything else None) and the shared per-cut tag grammar keyed on the line.
• publish_cli.py --promote-line LINE (requires --tag-name; refuses --no-latest, --tag-only, --evidence).

P2 — registry default and loader

• registry.py: DatasetSpec.pointer_path (default latest.json); the 2025 national line is registered as variant national on microcosm_uk_2024_25.h5 through latest-national.json (superseding the original "default variant at merge" wording, see review round 1 below), while ("uk", 2023) keeps latest.json
• loader.py: the pointer is read through the spec's path (latest_release for latest.json, latest_line_release for latest-<line>.json, anything else refused); the release manifest is fetched at the pointer's revision and every artifact must be pinned to it; artifact selection follows the manifest role (national_default → default_datasets.national; non_default_local_area only through a line pointer, with empty default_datasets and exactly one microdata artifact); available() lists default-variant keys only.

P3 — the national assembler's promotion command and the docs

• tools/assemble_uk_release_dir.py: _DATASET_KEY = "microcosm_uk_2024_25", pinned to the registry's ("uk", 2025, "compact") filename by a lockstep test; a candidate not named microcosm_uk_2024_25.h5 (the name --release-role national writes on Build the UK national line through the rowwise driver's release role (#823) #965) is refused by name; default_datasets, the build manifest's dataset filename, the calibration NPZ and the release manifest artifact key follow the constant; the JSON summary keeps publish_command (inspect, --no-latest --tag-name <cut>) and adds promote_command (--promote-line national --tag-name <cut>).
• Runbook 806 §4 shows both commands and §6 becomes "Promote the reviewed cut" (only latest-national.json moves; latest.json never; load("uk", 2025, variant="national") then follows the pointer and the default flips in a follow-up; the alert fires); README "Releasing & alerts" and CLAUDE.md name line promotion.

Verification

• microcosm-data suite (735 tests) and the build-side assembler, certification, contract-pin, dense assembler, pre-flight, posture and scorer suites green on this stacked tree; ruff check ., tools/ci_test_groups.py --verify.
• P1's fake-Hub publish test seeds the June latest.json bytes and asserts that promoting a national cut changes only latest-national.json, writes the pointer last, tags the cut, and that latest_line_release resolves to the cut tag.
• P1 and P2 were implemented by Codex from the reviewed plan and reviewed here; P3 likewise under a narrowed scope (no clone step: the driver on Build the UK national line through the rowwise driver's release role (#823) #965 already writes the pinned name, so the assembler refuses instead).

Not in this PR

• The dashboard change (calibration-diagnostics: pointer_path on the country registration, loadPointerReleaseId returning the pointer's revision) is on its own branch there.
• B0 (the score receipt written in the certifier's shape) is on its own branch.
• The pe.uk consumer switch is deferred; the exact-K local-area line (identity and pointer reserved here) is Complete UK exact-K release parity and make the runtime country-neutral #898.

🤖 Generated with Claude Code

cc @MaxGhenis

Review round 1 (2026-09-23): Vahid

Base retargeted to main and the three commits rebased over it (clean); round commit 7053b917.

• Inspect → promote works, and so does a retry. _publish_atomic recognises an existing cut tag before touching the staging branch: when the tagged release_manifest.json is byte-identical to the local one the immutable revision already exists, so no staging branch, no immutable commit, no create_tag, and only the main commit carrying the line pointer is written; a tag that describes another release refuses. Tested on the FakeHub: the documented two-step sequence (one tag, one extra create_commit, latest.json byte-identical), the retry after an injected pointer-commit failure, the foreign-tag refusal both ways. Runbook §4 now says which failures still need the staging branch deleted (only a failure before the tag exists).
• The default no longer moves at merge. resolve("uk") is a registry lookup, so a 2025 default entry would have sent every default load to a pointer that does not exist until the first promotion. The 2025 national entry is registered as variant national (Max's Promotion for the constant-id national line: pointer and certified-loader extension #823 ruling keys the line as a variant): load("uk", 2025, variant="national") follows latest-national.json once it exists, resolve("uk") and available() stay on the June 2023 line, and the flip to the default variant is a one-line follow-up PR after the first promotion, with its test. Runbook §6 and the README carry the sequencing. (This supersedes "resolve('uk') lands on 2025 at merge" in the sections above.)
• Should-fixes. The live pointer test is parametrized over the national variant, skipping until the pointer exists and asserting the cut-tag revision shape once it does; the loader states why every artifact must pin the pointer's revision (stricter than the contract's annual-revision allowance, by design) and the latest.json path is tested for it; dataset_role_for_line binds national → national_default and local-k<N> → non_default_local_area, refused by the publisher on promotion and by the loader on resolution.
• Nits. line_pointer_path("evidence") refuses at the grammar; the assembler binds certification.candidate.filename to the line's dataset filename again; the pointer wording above is corrected.

Verification at 7053b917: test_release, test_loader, test_registry, test_publish_guard, test_contract, the national and dense assemblers, certification, contract pins and preflight suites green locally (556 + 221 tests across the two runs, one live skip); ruff check . and tools/ci_test_groups.py --verify clean.

Post-approval notes (72d6094c): the tag-reuse path swallows only the Hub's "does not exist" answers (a transient failure looking up the tag or its manifest now propagates instead of re-entering the create path or sending the operator to a fresh cut tag; tested with injected failures on both calls); the live pointer test skips "not yet promoted" only on an EntryNotFoundError naming the pointer path, after the credentials branch; the 823-uk-2025-registry fragment and the data README now describe the national variant and the default-variant-only available().

[vahid-ahmadi]

Review at f998e774 (worktree checkout of the three commits on cc23ec06). Local: ruff check/format clean on the 14 changed files; ci_test_groups.py --verify ok; microcosm-data 733 passed / 2 skipped; build-side assembler, certification, contract-pin, dense assembler, preflight, posture and scorer suites 166 passed. Merge-tree against main d9a248d0 is clean (main has not touched registry.py, loader.py or the assembler since), and the suites also pass on the materialised merge, so this only needs its base retargeted to main; no rebase.

The pointer mechanics follow Max's ruling where it was explicit: line pointer path/payload/reader beside the two existing ones (release.py:163-204, 1008-1050), LatestPointer.revision defaulting to the release id so latest.json stays byte-identical, latest_release refusing line/revision (:992-997), the writer-side reservation of every latest*.json root path (:378-388), the narrowed #823 refusal (:419-427), the loader fetching the manifest at the pointer revision, and the CLI refusals. Only _publish_atomic writes pointers, so latest.json is immutable under promotion. line_for_release_id and its test, the Slack line, _DATASET_KEY with its lockstep test and the filename refusal all check out.

Blocking

1. The documented inspect → promote sequence cannot run. Runbook §4 publishes with --no-latest --tag-name <cut>, §6 promotes the same cut with --promote-line national --tag-name <cut>. But publish_release always re-runs the staging commit and _create_release_tag (release.py:733-739, no exist_ok), so the second call fails on the existing tag and no pointer is written; I reproduced it against your own FakeHub (second publish → tag exists, latest-national.json absent, the staging branch left behind). The runbook's 409 paragraph then says delete the branch and retry the same cut, which hits the same error. The only bypass, --no-create-tag, is refused at :405-411. test_national_cut_promotes_only_its_line_pointer promotes a fresh cut in one shot, so the two-step path has no test. The same root cause covers the partial-failure case: the tag lands before the pointer commit, so any retry after a failure between them dies on create_tag. One fix covers all three: accept an existing tag that already points at the revision being published, or give --promote-line a mode that skips the immutable commit and writes only the pointer. Add the two-step test.
2. The consumer default moves at merge, not at promotion, and that departs from Max's ruling. resolve("uk") is a pure registry lookup (loader.py:129-166, latest_year picks the highest default-variant year), so the moment this merges it returns ("uk", 2025) with pointer_path="latest-national.json" (registry.py:114-126). That file does not exist until the first promotion, and the loader raises rather than falling back, so every default load("uk")/download("uk") fails between merge and promotion, and available() advertises 2025 before anything is published; only explicit ("uk", 2023) callers keep working. Max's item 5 keyed the national line as a variant precisely so resolve("uk") is undisturbed; your proposal 2 (default variant) is the one he never answered. Either get his yes and sequence promotion before merge, or register the 2025 default only after the pointer exists. The body's "pe.uk consumer switch is deferred" is true of policyengine.py's own pins but not of this registry default.

Should fix

3. test_live_latest_pointer_resolves_to_a_coherent_certified_release (test_loader.py:804-809) is parametrized ("us", 2024), ("uk", 2023) only; nothing live exercises the new default or latest-national.json.
4. The loader now requires every manifest artifact pinned to the pointer revision (loader.py:319-331), stricter than the contract's annual_revision allowance (contract.py:1317). No live impact today (the live US and June UK manifests are uniformly pinned), but a future mixed-revision default would be refused; state the intent in a comment or a test.
5. prepare_release(line=…) skips the non-default-role refusal (release.py:313) and relies on line_for_release_id plus the manifest role. I could not find a contract rule that microcosm-uk-2024-25-national must declare national_default; without one a non_default_local_area manifest could ride the national pointer through the loader's line-pointer branch (:293-298).

Nits. _LINE_RE (release.py:87) admits evidence, so line_pointer_path("evidence") collides with latest-evidence.json; guarded downstream, but refuse it at the grammar. The assembler dropped the certification.candidate.filename check (sha binding survives at :166). "Pointer written last" means last CommitOperationAdd in one atomic commit, which is the right property; the body slightly oversells it. Commit prose says P1/P2 were implemented by Codex, but the trailers name only the Claude co-author.

Requesting changes for 1 and 2; retarget to main so CI runs.

[juaristi22]

Thanks — retargeted to main and rebased over it (clean, as your merge-tree said); the round is 7053b917.

Blocking 1, inspect → promote. Reproduced on the FakeHub as you did. _publish_atomic now recognises an existing tag before touching the staging branch: if the tag's release_manifest.json is byte-identical to the local one, the immutable revision already exists, so no staging branch, no immutable commit, no create_tag, and only the main commit carrying the pointer is written; a tag that describes another release (different manifest, or none) refuses. That one recognition covers the documented two-step sequence, a re-publish of the same cut, and a retry after a failure between tag creation and the pointer commit. Tests: the two-step sequence (second call emits exactly one create_commit, one tag, latest.json byte-identical), the retry after an injected pointer-commit failure, and the foreign-tag refusal both ways. Runbook §4's 409 paragraph now says which failures still need the branch deleted (only a failure before the tag exists).

Blocking 2, the default at merge. Agreed: resolve("uk") is a registry lookup and would have chased an absent pointer between merge and promotion. The 2025 national entry is now registered as variant national (your item 5 reading of Max's ruling): load("uk", 2025, variant="national") follows latest-national.json once it exists, resolve("uk") and available() stay on 2023, and the default flip is a one-line follow-up after the first promotion, with its test. The runbook §6 and the README say so; María's "must flip" ruling is honoured at promotion time rather than at merge.

3. The live test is parametrized over the national variant too, skipping (not passing) until the pointer exists on the Hub, and asserting the cut-tag revision shape once it does.
4. The intent is stated in the loader (one revision per pointer; stricter than the contract's annual-revision allowance by design) and tested on the latest.json path as well as the line path.
5. dataset_role_for_line in the contract: national → national_default, local-k<N> → non_default_local_area; prepare_release(line=…) refuses a mismatched manifest and the loader refuses a manifest that reached a line pointer with the wrong role (both tested).

Nits. line_pointer_path("evidence") refuses at the grammar; the assembler binds certification.candidate.filename to the line's dataset filename again; the body now says the pointer is the last operation of the one atomic main commit; the Codex prose in the P1/P2 commit messages stands as written since the commits are already reviewed (the trailers name the co-author that opened the PR).

🤖 Generated with Claude Code

[vahid-ahmadi]

Second pass at 7053b917 (retargeted to main, rebased; merge-tree against 7c5306f3 clean; CI 24/24 green). Local: microcosm-data suite 741 tests, 0 failures, plus the same suites on the materialised merge, 0 failures; ruff clean; ci_test_groups.py --verify ok.

Both blockers closed.

1. Inspect → promote. _publish_atomic now reads the revision an existing tag points at (release.py:969) and refuses unless that tag's release_manifest.json is byte-identical to the local one (:989); on a match it skips the staging branch, the immutable commit and create_tag and writes only the pointer commit. I re-ran my two-step repro against your FakeHub: the second publish now succeeds and latest-national.json resolves to the cut tag. test_inspect_then_promote_reuses_the_immutable_tag (test_release.py:2279) and test_retry_after_a_failed_pointer_commit_reuses_the_tag (:2332) cover the sequence and the retry; the foreign-tag refusal is tested both ways. Runbook §4 now says only a failure before the tag exists can leave the staging branch behind.
2. Default at merge. The 2025 national entry is registered as variant national (registry.py:125); test_resolve_uk_default_stays_on_2023_until_the_national_line_is_promoted (test_loader.py:348), test_resolve_uk_national_variant_follows_the_line_pointer (:361) and test_resolve_uk_2023_still_uses_latest_json (:387) pin it, and test_registry.py:152 names the follow-up flip. Runbook §6 (:207) and the README say the flip is a one-line PR after the first promotion. That is Max's item 5 as written.

Items 3–5 and the nits all landed: the live pointer test is parametrized over the national variant, the loader states the one-revision-per-pointer intent and tests both paths, dataset_role_for_line (contract.py:634) binds national → national_default and local-k<N> → non_default_local_area with refusals in both publisher and loader, line_pointer_path("evidence") is refused at the grammar (release.py:170), and the assembler binds the candidate filename again.

One small question, not blocking: for the national parametrization of the live pointer test, make sure an absent latest-national.json (an EntryNotFoundError) is what skips, and that an auth failure on the private repo does not also skip silently; _is_offline_error lists LocalEntryNotFoundError, and I could not tell from the code which exception the remote absence raises.

Approving.

[vahid-ahmadi]

Approval stands. Three small things I'd fold in before merge, since the changelog will publish as written:

• changelog.d/823-uk-2025-registry.changed.md still says "Changed the UK 2025 registry default to the 2024–25 national dataset", contradicting its sibling 823-national-variant-until-promoted.changed.md and the code; the body's P2 paragraph says the same. Delete or rewrite it. available() narrowing to default variants only (the dense entry no longer listed) is also not in any fragment, and packages/microcosm-data/README.md:25 still says it lists published pairs.
• _require_tag_describes_release (release.py:997-1010) catches any exception from the manifest download and tells the operator to publish under a fresh cut tag; a transient network error would orphan a good cut. Distinguish EntryNotFoundError from the rest. Similarly _tag_revision returns None on any repo_info error, so a transient failure with an existing tag re-enters the create path and dies on create_tag with the staging branch left behind, which the rewritten §4 says cannot happen.
• The live pointer test's _is_missing_pointer_error is path-agnostic and runs before the credentials branch, so a post-promotion manifest 404 or an anonymous RepositoryNotFoundError (whose text contains "404") both skip as "not yet promoted". Narrow it to spec.pointer_path in str(exc).

[juaristi22]

Thanks — all three are in 72d6094c. _tag_revision and _require_tag_describes_release now swallow only the Hub's "does not exist" answers (RevisionNotFoundError, EntryNotFoundError, and the fake's KeyError/FileNotFoundError); a transient failure on either call propagates, so a good cut is never orphaned and the create path is never re-entered with the tag in place (tested with injected failures on both). The live test's skip now requires an EntryNotFoundError naming spec.pointer_path and runs after the credentials branch, so an anonymous RepositoryNotFoundError or a post-promotion manifest 404 fails rather than skipping; on your question, the remote absence of latest-national.json raises exactly that EntryNotFoundError (a 404 on the file URL), while LocalEntryNotFoundError is the offline-cache case the offline branch already handles. The 823-uk-2025-registry fragment now describes the national variant registration and the default-variant-only available(), and the data README says the same; the PR body's P2 paragraph is corrected.

🤖 Generated with Claude Code

[vahid-ahmadi]

Checked 3bd0ed2f: _tag_revision and _require_tag_describes_release now re-raise anything that is not an absent-revision / absent-entry answer (release.py:975-981, 1025-1029), _is_missing_pointer_error is keyed on the pointer path (test_loader.py:819-825), and 823-uk-2025-registry.changed.md describes the variant registration. Approval stands; nothing further from me.