Repository navigation
Promotion for the constant-id national line: pointer and certified-loader extension #823
Description
Activity
Status check on 2026-09-18 (main at 8c44daa): still live, and it now gates more than the national line.
Unchanged since filing.
latest_pointer_payloadstill carries{schema_version, release_id, updated_at, paths}with nothing naming a cut;loader.pystill fetches the manifest atrevision=pointer.release_idand refuses a default artifact whoserevision != release_id;publish_releasestill refuses to movelatest.jsonfor any tag that is not the release id; the staging branch is stillrelease-staging/<release_id>, so the ride-along strand recurs. No change has touched these contracts inmicrocosm-datasince 30 August.Now also the dense line. #762's release path publishes
microcosm-uk-2024-25-densethe same way: constant release id, per-cut tag,--no-latestmandatory and enforced (docs/uk-dense-release-assembly-runbook-762.md§5). A dense cut is reachable by its tag and by the registry key("uk", 2025, "dense")only, so promotion of the full build's dense line waits on the same pointer and loader extension. The exact-count line (#898) has no permitted release identifier yet; this issue is a precondition it meets later, not what blocks it today.Not affected. The staged-dataset lane from #946 never reads or writes
latest.json, the pointer or the certified loader: a run's bundle lives understaged/<run_id>/and is fetched by run id and commit sha. Inspection of any run, releasable or not, works without this issue.The repo-pointer adjudication is more pressing than when written.
policyengine/populace-uk-privatenow holds the June 2023 release that certified consumers follow through the repo-globallatest.json, the dense line's per-cut tags, and (from #946) the staged bundles. Promoting any newer line re-points what "current" means for the whole repo, so the ruling this issue names (one shared pointer, per-line pointers, or separate repos) decides the shape of the pointer fix and should come first.Ruling on the repo-pointer question this issue names, so the pointer fix has a shape to build to.
Pointer semantics: per-line pointers, in the one repo.
latest.jsoninpolicyengine/populace-uk-privatestays exactly where it is, naming the June 2023 release, and is not touched by the national or dense lines. Each promotable line gets its own pointer file.Pointer shape: the line pointer names the cut. It carries the cut's immutable tag, and the certified loader fetches the manifest at that tag. The loader does not read manifests from
main.Why, from the code on
main:- The loader selects the pointer by repo alone (
loader.py,latest_release(spec.hf_repo)) and then requires the manifest's default artifact to equal the registry filename. One repo-global pointer can therefore serve one registry entry. The live pointer namespopulace-uk-2023-dd68c73-4aa4b14-20260619T023711Z; the moment it moves,load("uk", 2023)raises on every install, old and new. That fails this issue's own acceptance line about 2023 consumers. - Reading the manifest from
mainwould make certified consumers follow the newest unreviewed cut. The national release id is constant, and every--no-latestpublish that is not--tag-onlywritesreleases/<id>/release_manifest.jsontomain, so each inspect cut overwrites the same path. The sha pins do not help: they pin the artifact to whatever that manifest declares. - Bumping
LATEST_POINTER_SCHEMA_VERSIONis a fleet-wide event, not a UK one. The reader checks exact equality and the writer is shared, so the next US publish would emit a pointer every older install refuses. Five readers followlatest.jsontoday: microcosm's loader, bottom-50-tax-analysis, the calibration dashboard'slatest-artifact.ts, microcosm.institute, and calibration-diagnostics' Python evaluation harness, which also pinsschema_versionto 1. None of them ever sees a new pointer file, so nothing upgrades in lockstep. - The pattern already exists and is tested in this module: the evidence tier's
latest-evidence.json(Evidence-tier publishing: ship the best available artifact at all times, with its failures recorded in the manifest (dense first) #506) is a second pointer file with its own payload, its own reader, and a guard against cross-reading.
What the implementation needs, so it is not rediscovered in review:
- A line pointer path, payload and reader beside the existing two, and
DatasetSpec.pointer_pathdefaulting tolatest.json. Forlatest.jsonpayloads the revision defaults to the release id, so that path stays byte-identical. - The loader fetches the manifest at
pointer.revisionand asserts the default artifact's revision against it. Still a hard assertion, against the cut tag. - Reserve the new path in the writer guard as well as the reader: the reserved-pointer-path set in
prepare_releaseis a literal of the two existing paths, and without the third a release manifest could declare a root artifact at the line-pointer path and move the pointer as an ordinary upload. latest_releaserefuses arevisionkey outright, and the line reader requires both its discriminator and arevision. Otherwise a hand-editedlatest.jsoncarrying a cut tag would send every certified consumer to an unpromoted cut.- Key the national line
("uk", 2024, "national"): the artifact ismicrocosm_uk_2024.h5and the release id ismicrocosm-uk-2024-25-national. Adding a variant does not disturbresolve("uk"). - Narrow
publish_release's refusal from "no pointer may name a per-cut tag" to "latest.jsonmay not; a line pointer may", keeping the fail-closed default.
The dense line is outside this ruling: its role forbids it from moving any pointer, and
("uk", 2025, "dense")does not resolve through the certified path today. Separate repos were the alternative I rejected: they multiply tokens and access control and buy nothing per-line pointers do not.What would change this: a certified consumer that cannot take a line key without a breaking change. policyengine.py is not one; its certification path pins an explicit repo, revision and path.
- The loader selects the pointer by repo alone (
Thanks Max, building to that shape. Five things to settle before I write the implementation plan, each with the answer I would propose so you can just say yes or correct it.
1. Registry key and filenames for the 2024-25 lines. The raw data is published by fiscal year, so the dataset name is
microcosm-uk-2024-25and the national artifactmicrocosm_uk_2024_25.h5, matching the constant idmicrocosm-uk-2024-25-national.DatasetSpec.yearstill needs one integer: you keyed 2024, the existing dense entry is keyed 2025 (("uk", 2025, "dense")), and the H5's own time period is 2025, the calibration year. Proposal: key every 2024-25 line on 2025 (soload("uk", 2025)reads as the 2025 policy year, as the dense entry already does) and re-key and rename the dense entry to match (microcosm_uk_2024_25_dense.h5), so the two lines never disagree.2. The national line as the default variant, not a
nationalvariant. We are moving to 2025 as the modelled year by default, soresolve("uk")should land on the new line.latest_yearpicks the highest registered year for the default variant, so a("uk", 2025)default entry with its own pointer file does that on its own, while("uk", 2023)keepslatest.jsonuntouched exactly as you ruled. Proposal: amend item 5 to("uk", 2025)default withpointer_pathlatest-national.json(or whatever name you prefer), the 2023 entry unchanged and retired from "default" only by the year ordering.3. A line pointer for the local-area line, and narrowing the role's prohibition. You wrote the dense line out of the ruling because its role may move no pointer. The line we need people to consume is the exact-count one: the full-pool dense build is an intermediate we L0-penalise to exact-K files calibrated to all national and subnational targets, and those are the local-area dataset. Proposal: a
non_default_local_arealine may move its own line pointer and neverlatest.json; the certified loader, for that role, selects the singlemicrodataartifact the local-area contract already guarantees (itsdefault_datasetsis empty by contract); the dense full-pool cut stays tag-only or staged, without a pointer.4. The local-area line's identity. This is #898's naming decision and better taken now than rediscovered there. The contract already reserves
populace-uk-<year>-<tier>-k<N>and binds the count fields to the id. Proposal: one constant line id per approved count (microcosm-uk-2024-25-local-k55000, per-cut tags as on the national line), each with its own pointer, and the registry variantlocalnaming the approved default count; the dense line's constant id and assembler stay for inspection only. The alternative is a count-free line id with the count in the manifest; I prefer the count in the id because a pointer move can then never silently change the file size a consumer gets.5. The dashboard's UK default. With
latest.jsonfrozen on June 2023,?country=ukkeeps showing 2023 as current unless the dashboard reads the national line pointer. Proposal: the UK registration follows the line pointer (a small change inlatest-artifact.ts::loadPointerReleaseIdbeside theproduction_release_idpath), or gains a pinned production release id for UK once a cut is reviewed; your call which.Everything else in your ruling I am building to as written: per-line pointers in the one repo, the pointer naming the cut, no schema bump, the writer-side reservation,
latest_releaserefusing arevisionkey, and the narrowedpublish_releaserefusal. The build-side consolidation (the national dataset built by the rowwise driver without cloning, on national targets, and the standalone national command retired) is orthogonal and will come to you as a PR.- added 4 commits that reference this issue
on Sep 21, 2026 Evaluation rule on the UK national line (microcosm#967, 2026-09-22): rule 1 (#578) is decided on the surface both artifacts can materialize. A target whose measure the incumbent cannot compute is pruned from BOTH arms only under a signed, in-force entry of the reviewed incumbent-unresolvable register (
uk/incumbent_unresolvable_measures.json); an unlisted measure refuses the evaluation. The receipt names every pruned row, its measure and the entry it stands on, and the release-cut certifier re-checks the pruned measures against the committed register. At v21 the six measures the enhanced FRS never carried remove 120 of 644 targets (UC administrative family 93, two-child limit 15, ONS household type 10, CGT asset type 2); the six entries are signed on the register as of 2026-09-22 (a8aa8684on #967), in force to 2026-12-22.Update (2026-09-22, after the rebase over #965): the signing commit on the branch is
2f97e087; the register ispackages/microcosm-build/src/microcosm/build/uk/incumbent_unresolvable_measures.json.- added 6 commits that reference this issue
on Sep 22, 2026 Yes to all five, with one caveat on 4.
- Key every 2024-25 line on 2025 (the calibration and policy year) and rename the dense entry to
microcosm_uk_2024_25_dense.h5, soload("uk", 2025)means the same thing on every line. - The national line becomes the
("uk", 2025)default with its ownlatest-national.jsonpointer. 2023 stays untouched, and the default moves only because 2025 is the higher year, solatest.jsonnever moves. - The local-area line may move its own line pointer, never
latest.json. Its certified loader selects the single microdata artifact, and the full-pool dense cut stays tag-only or staged. People consume the exact-K file; the dense pool is an intermediate. That matches the principle that only the L0 budget should differ across sizes. - Put the count in the line id (
microcosm-uk-2024-25-local-k55000), so a pointer move can never silently change the file size a consumer gets. This approves the naming scheme, not 55k as the default count. Your 9/11 report has most constituencies under the effective-sample-size floor at 55k, so the default count is still a separate call. - The dashboard's UK registration follows the national line pointer rather than a pinned production id, so there's one source of truth. Please coordinate the
latest-artifact.tschange with Anthony's diagnostics unification (Unify calibration diagnostics schema and publication #1007).
Go ahead and land #966 on this basis.
- Key every 2024-25 line on 2025 (the calibration and policy year) and rename the dense entry to
- added a commit that references this issue
on Sep 24, 2026
#806 / #812 delivered the UK national line's inspect lane: assemble a certified cut, publish with
--no-latestunder an immutable per-cut tag (microcosm-uk-2024-25-national-<YYYYMMDDTHHMMSSZ>-<uuid8>), and adjudicate it on the dashboard via the release list /?release=. Promotion — movinglatest.json— is structurally refused there (publish_releaserejects a pointer move for any tag that is not the release id itself), because the current pointer and loader contracts cannot express it. This issue is the design and implementation that lifts that refusal.Why promotion cannot work today
The 2026-08-27 ruling makes the national release id constant across cuts; run identity lives in the per-cut tags and the Logbook chain. Two contracts still assume id == tag:
latest_pointer_payloadcarries only{schema_version, release_id, updated_at, paths}(release.py). A promoted pointer could say the current release ismicrocosm-uk-2024-25-nationalbut not which cut that means.loader.pyfetchesrelease_manifest.jsonatrevision=pointer.release_id— an immutable tag named exactly the release id, which the national line never mints (and must not: minted once at cut 1 it would pin cut 1 forever) — and separately hard-asserts the default artifact'srevision == release_id, which per-cut revisions fail by design.The dashboard itself would survive a pointer move (it reads
releases/<id>/paths off main, which always hold the current cut); every certified consumer would break.Design decisions this issue owns
tag/revisionfield naming the current cut (bumpsLATEST_POINTER_SCHEMA_VERSION; the reader refuses unknown versions, so microcosm-data consumers, the dashboard, and the pe certification path upgrade in lockstep) — versus teaching the loader to read the manifest from main and trust its per-cut revision + sha pins (no schema bump, different immutability trust model).latest.jsonis repo-global, andpolicyengine/populace-uk-privatecurrently serves the June 2023 release to certified consumers. Promoting the national line re-points what "current" means for the whole repo: one shared pointer, per-line pointers, or separate repos is an adjudication, not a refactor.(uk, 2023) → populace_uk_2023.h5), so pe can load what the pointer names.publish_release's per-cut-tag pointer refusal with whatever the new contract permits, keeping the fail-closed default for tags the pointer cannot express.Riding along
release-staging/microcosm-uk-2024-25-nationalstrands when tag creation 409s mid-publish and blocks the retry until deleted by hand; a per-attempt staging branch name removes the recurrence.Acceptance
--no-latestpublishes still work identically, and promotion of a tag the pointer cannot express still refuses.Out of scope
uk_target_fitadjudication line (UK WAS stage: split private pension wealth out of corporate_wealth (uk-data#452 M2) #750 et al.) — a shippable certification is a precondition for having anything to promote.