From deebbcd210a94576b4f43cbbbe185b2623b68ab8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:24:40 +0100 Subject: [PATCH 01/11] Declare the UK rowwise driver's release role and move the national doctrine to the ruled posture (#823) One driver builds both UK dataset lines. `--release-role {national,dense}` is now required on tools/build_uk_rowwise_candidate.py: a frozen posture record (uk_runtime/rowwise_posture.py) supplies each role's doctrine object, clone count, solve length, learning rate, seed, target-weighting rule and its admissible overrides, gate scope and posture, Logbook pipeline and attempt-id prefix, staging operation id, release id and output filenames. The driver fills the role's defaults into unset solve arguments, records which were given explicitly, and refuses the other role's flags by name before any value is range-checked; the manifest (schema 4) and the dry-run plans record `release_role` and the manifest `release_id`; the size-checkpoint identity carries the role so a pre-role checkpoint refuses to resume. The national role is declared and validated here and raises NotImplementedError after validation: its build flow (delegation to the calibration seam) is the next increment. Doctrine: UK_NATIONAL_SOLVE_EPOCHS is 1500 and UK_NATIONAL_TARGET_WEIGHT_RULE is "family_equal" (ruling 2026-09-20): the certified-cut posture every campaign national run had overridden to is the doctrine, so a doctrine run records no overrides; "uniform" stays a receipted per-run override. The pinned tests move with the constants. Dense line: the pre-flight and the assembler accept only a manifest whose release_role is "dense" (a pre-role candidate is rebuilt, never grandfathered); the published artifact is microcosm_uk_2024_25_dense.h5, mirrored in the contract as _UK_DENSE_DATASET_FILENAME with a lockstep pin; the driver's joint-surface outputs are microcosm_uk_2024_25_local.h5 and microcosm_uk_2024_25_local.local_gates.json (the `_local` family stays because the size evaluator and the pre-flight glob it). Staging: the driver's default upload cadence is 300 s (the Hub allows about 128 commits per hour per repository; the v20 run lost uploads at 30 s), a refused stage event is reported and dropped instead of aborting the build (the #946 follow-up), and add_staging_arguments takes a per-command default cadence. Co-Authored-By: Claude Fable 5.1 --- CLAUDE.md | 2 +- changelog.d/uk-dense-filename-823.changed.md | 1 + ...ional-doctrine-family-equal-823.changed.md | 1 + .../uk-rowwise-release-role-823.added.md | 1 + docs/uk-dataset-size-plan-355.md | 2 +- docs/uk-dense-release-assembly-runbook-762.md | 26 +- docs/uk-national-calibration-runbook-623.md | 15 +- .../src/microcosm/build/staging_cli.py | 12 +- .../build/uk_runtime/national_doctrine.py | 20 +- .../build/uk_runtime/rowwise_posture.py | 299 +++++++++++++ .../tests/test_gate_battery_contract_pins.py | 18 + .../microcosm-build/tests/test_staging_v2.py | 21 + .../tests/test_uk_calibration_seam_driver.py | 2 +- .../tests/test_uk_dense_release_assembler.py | 65 ++- .../tests/test_uk_local_release_preflight.py | 27 +- .../tests/test_uk_national_calibration.py | 23 +- .../tests/test_uk_rowwise_candidate.py | 236 ++++++++++- .../tests/test_uk_rowwise_posture.py | 185 +++++++++ .../src/microcosm/data/contract.py | 6 +- .../tests/test_uk_dense_release_contract.py | 8 +- tools/assemble_uk_dense_release_dir.py | 12 +- tools/build_uk_rowwise_candidate.py | 393 +++++++++++++++--- tools/preflight_uk_local_release_candidate.py | 8 + 23 files changed, 1244 insertions(+), 139 deletions(-) create mode 100644 changelog.d/uk-dense-filename-823.changed.md create mode 100644 changelog.d/uk-national-doctrine-family-equal-823.changed.md create mode 100644 changelog.d/uk-rowwise-release-role-823.added.md create mode 100644 packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py create mode 100644 packages/microcosm-build/tests/test_uk_rowwise_posture.py diff --git a/CLAUDE.md b/CLAUDE.md index 4197ba491..fd2a40ce9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -171,7 +171,7 @@ Update this guide in the same PR whenever the workspace layout, test commands, or release flow change. If you find it contradicting the repo, trust the repo and fix this file. -UK size experiments use `tools/build_uk_rowwise_candidate.py --dataset-households` +UK size experiments use `tools/build_uk_rowwise_candidate.py --release-role dense --dataset-households` with the same pool inputs as the dense candidate. The flag changes exported support, not clone K. Sizes remain candidate-only until their matched comparison and promotion scorecard are adjudicated; see diff --git a/changelog.d/uk-dense-filename-823.changed.md b/changelog.d/uk-dense-filename-823.changed.md new file mode 100644 index 000000000..a42c31b70 --- /dev/null +++ b/changelog.d/uk-dense-filename-823.changed.md @@ -0,0 +1 @@ +The UK dense line's published microdata is `microcosm_uk_2024_25_dense.h5` (the FRS release vintage, matching the `microcosm-uk-2024-25-dense` release id) and the driver's joint-surface outputs are `microcosm_uk_2024_25_local.h5` and `microcosm_uk_2024_25_local.local_gates.json`; the contract mirrors the published filename with a lockstep pin. diff --git a/changelog.d/uk-national-doctrine-family-equal-823.changed.md b/changelog.d/uk-national-doctrine-family-equal-823.changed.md new file mode 100644 index 000000000..765214941 --- /dev/null +++ b/changelog.d/uk-national-doctrine-family-equal-823.changed.md @@ -0,0 +1 @@ +The UK national calibration doctrine is now the certified-cut posture every campaign run had overridden to: `UK_NATIONAL_SOLVE_EPOCHS` is 1500 and `UK_NATIONAL_TARGET_WEIGHT_RULE` is `family_equal` (ruling 2026-09-20), so a doctrine national run records no overrides; `uniform` remains a receipted per-run override. diff --git a/changelog.d/uk-rowwise-release-role-823.added.md b/changelog.d/uk-rowwise-release-role-823.added.md new file mode 100644 index 000000000..c36d97c92 --- /dev/null +++ b/changelog.d/uk-rowwise-release-role-823.added.md @@ -0,0 +1 @@ +`tools/build_uk_rowwise_candidate.py` now requires `--release-role {national,dense}`: a frozen posture record (`uk_runtime/rowwise_posture.py`) supplies each role's solve defaults, gate scope, Logbook pipeline, staging ids, release id and output filenames, the manifest (schema 4) records `release_role` and `release_id`, and each role refuses the other's flags. The dense pre-flight and assembler accept only a `dense` manifest; the national role is declared and validated here and gains its build flow in the next increment. The driver's default telemetry upload cadence is now 300 s, and a refused stage event is reported and dropped instead of aborting the build. diff --git a/docs/uk-dataset-size-plan-355.md b/docs/uk-dataset-size-plan-355.md index 0b9c932d6..df667b8f8 100644 --- a/docs/uk-dataset-size-plan-355.md +++ b/docs/uk-dataset-size-plan-355.md @@ -76,7 +76,7 @@ Use the inputs and environment from the existing Pass the same pinned source arguments to the existing driver and add: ```bash -uv run python tools/build_uk_rowwise_candidate.py \ +uv run python tools/build_uk_rowwise_candidate.py --release-role dense \ --input-h5 "$UK_SPINE_H5" --input-sha256 "$UK_SPINE_SHA256" \ --ladder "$UK_LADDER_NPZ" --ladder-sha256 "$UK_LADDER_SHA256" \ --ledger-facts "$UK_LEDGER_FACTS" \ diff --git a/docs/uk-dense-release-assembly-runbook-762.md b/docs/uk-dense-release-assembly-runbook-762.md index f8bedd93f..ef36300e4 100644 --- a/docs/uk-dense-release-assembly-runbook-762.md +++ b/docs/uk-dense-release-assembly-runbook-762.md @@ -36,16 +36,23 @@ digest mismatch, or doctrine constants that are not the ruled ones. ## 2. Run the release candidate ```bash -uv run --no-sync python tools/build_uk_rowwise_candidate.py --release-candidate \ - --input-h5 --input-sha256 \ +uv run --no-sync python tools/build_uk_rowwise_candidate.py --release-role dense \ + --release-candidate --input-h5 --input-sha256 \ --ladder build/uk/uk_oa_ladder_2021.npz --ladder-sha256 \ --ledger-facts --ledger-facts-sha256 \ --ledger-manifest-sha256 --seed 42 \ --logbook-prev-row-digest --out ``` -`--release-candidate` pins the doctrine (bound 10, `grain_equal`, K=15, 1500 -epochs), resolves the engine in a single block, and runs the rotated holdout. +`--release-role dense` is required (microcosm#823): the role supplies the +solve defaults (K=15, seed 42, 1,500 epochs, learning rate 0.15, +`grain_equal`), names the outputs `microcosm_uk_2024_25_local.h5` and +`microcosm_uk_2024_25_local.local_gates.json`, records itself in the +manifest, and refuses the national role's flags. `--release-candidate` pins +the doctrine (bound 10, `grain_equal`, K=15, 1500 epochs), resolves the +engine in a single block, and runs the rotated holdout. Best-effort staging +telemetry uploads every 300 s by default on this driver (the Hub allows about +128 commits per hour per repository). Expect about 3.5 hours and 10 GB at K=15. ## 3. Pre-flight the finished run, then score it @@ -66,7 +73,7 @@ Measure the full pinned incumbent surface before assembly: ```bash uv run --no-sync python tools/evaluate_uk_incumbent_surface.py \ - --candidate-h5 /microcosm_uk_2025_local.h5 \ + --candidate-h5 /microcosm_uk_2024_25_local.h5 \ --candidate-manifest /rowwise_candidate_manifest.json \ --ledger-facts --ledger-facts-sha256 \ --ledger-manifest-sha256 --engine-blocks 1 \ @@ -104,14 +111,17 @@ uv run --no-sync python tools/assemble_uk_dense_release_dir.py \ --out-dir releases ``` -Assembly verifies the hash join (every manifest output against its bytes, the -spine against its pin, the gate report against the Logbook build id), requires +Assembly requires `release_role: "dense"` in the candidate manifest (as does +the pre-flight; a candidate built before the role existed is rebuilt, never +grandfathered), verifies the hash join (every manifest output against its +bytes, the spine against its pin, the gate report against the Logbook build +id), requires the manifest's `staging_delivery` receipt (the run's version 2 staging telemetry evidence, copied into `build_manifest.json` as `staging` so publication can apply its undelivered-staging refusal, as on the national lane), re-runs the candidate pre-flight, mints the cut tag `microcosm-uk-2024-25-dense--` from the run's attempt -id, clones the H5 beside itself as `microcosm_uk_2025_dense.h5`, stages +id, clones the H5 beside itself as `microcosm_uk_2024_25_dense.h5`, stages `build_manifest.json`, `release_manifest.json`, `calibration_diagnostics.json`, `gate_summary.json`, `uk_source_coverage.json`, the signed `uk_local_gates.json`, `score_vs_incumbent.json`, `incumbent_surface_evaluation.json`, the original diff --git a/docs/uk-national-calibration-runbook-623.md b/docs/uk-national-calibration-runbook-623.md index d7907f1c7..81c940e16 100644 --- a/docs/uk-national-calibration-runbook-623.md +++ b/docs/uk-national-calibration-runbook-623.md @@ -45,12 +45,15 @@ diagnostics file, hashes its actual bytes, and only then constructs and signs the terminal gate evidence. There is no `--calibration-diagnostics-sha256` to supply, and no way for the receipt to claim an identity the file does not have. -Solve parameters are per-run overrides, not defaults. The campaign settings are -`--epochs 1500 --target-weight-rule family_equal`; each override is validated -through the doctrine dataclass and echoed as an explicit deviation in the -manifest, diagnostics and build record. `--release-candidate` refuses every -override flag, and refuses `--measure-exclusions`, so a release candidate is -always solved under declared doctrine against the committed target surface. +Solve parameters are per-run overrides of the declared doctrine. Since the +2026-09-20 ruling (microcosm#823) the doctrine is the campaign posture itself: +1,500 epochs, `family_equal`, learning rate 0.02, seed 0, so a doctrine run +passes no solve flags and records no overrides; `--target-weight-rule uniform` +or another `--epochs` is validated through the doctrine dataclass and echoed +as an explicit deviation in the manifest, diagnostics and build record. +`--release-candidate` refuses every override flag, and refuses +`--measure-exclusions`, so a release candidate is always solved under declared +doctrine against the committed target surface. Signing the terminal gate report needs `MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY` in the environment. diff --git a/packages/microcosm-build/src/microcosm/build/staging_cli.py b/packages/microcosm-build/src/microcosm/build/staging_cli.py index b083a331b..633d0160b 100644 --- a/packages/microcosm-build/src/microcosm/build/staging_cli.py +++ b/packages/microcosm-build/src/microcosm/build/staging_cli.py @@ -21,8 +21,16 @@ def add_staging_arguments( parser: argparse.ArgumentParser, *, repository: StagingRepositoryConfig, + default_upload_interval_seconds: float = 30.0, ) -> None: - """Add staging controls using defaults supplied by a country module.""" + """Add staging controls using defaults supplied by a country module. + + ``default_upload_interval_seconds`` lets a long-running command choose a + slower best-effort upload cadence: the Hub allows about 128 commits per + hour per repository and every telemetry cycle is up to eight single-file + commits, so a multi-hour solve at the 30-second default exhausts the + budget and loses uploads (the UK rowwise driver runs at 300). + """ parser.add_argument( "--staging-dir", @@ -48,7 +56,7 @@ def add_staging_arguments( parser.add_argument( "--staging-upload-interval-seconds", type=float, - default=30.0, + default=float(default_upload_interval_seconds), help="Minimum interval between best-effort progress uploads.", ) mode = parser.add_mutually_exclusive_group() diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py index a17ac1928..db667c216 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py @@ -31,14 +31,14 @@ "UKNationalSolveDoctrine", ] -UK_NATIONAL_SOLVE_EPOCHS = 256 +UK_NATIONAL_SOLVE_EPOCHS = 1500 UK_NATIONAL_LEARNING_RATE = 0.02 UK_NATIONAL_MAX_WEIGHT_RATIO = 10.0 UK_NATIONAL_SEED = 0 UK_NATIONAL_TARGET_LOSS_CAP = 10.0 UK_NATIONAL_L0_LAMBDA = 0.0 UK_NATIONAL_MASS_RULE = "free" -UK_NATIONAL_TARGET_WEIGHT_RULE = "uniform" +UK_NATIONAL_TARGET_WEIGHT_RULE = "family_equal" _ALLOWED_SCALE_RULES = ("default_target_loss_scales",) @@ -53,10 +53,18 @@ # them annihilated 24 population and OBR targets that were never past cap # themselves (objective gain from annihilating a segment: +4.48 uniform vs # +0.68 family_equal at cap 10, negative at cap 2). María's ruling -# (2026-08-24): the default stays "uniform" — family_equal is reachable only -# as an explicit, receipted per-run override while the weighting doctrine is -# still being measured (its own run-9 receipts show a 9-member family -# carrying 5.5x the per-reference weight of a 50-member one). +# (2026-08-24) kept "uniform" as the default while the weighting doctrine was +# being measured, with family_equal reachable only as an explicit, receipted +# per-run override (its own run-9 receipts show a 9-member family carrying +# 5.5x the per-reference weight of a 50-member one). +# +# Ruling 2026-09-20 (microcosm#823 consolidation): every campaign national +# run since v18 has overridden to family_equal at 1,500 epochs — the v20 +# certified-cut posture (spine-s, loss 0.0104, 6/6 seam gates) carried the +# receipt {epochs 256 → 1500, target_weight_rule uniform → family_equal} — +# so those two values are now the doctrine and a certified national cut +# records no overrides. "uniform" stays vocabulary: a receipted per-run +# override, never silently reachable. _ALLOWED_TARGET_WEIGHT_RULES = ("uniform", "family_equal") _ALLOWED_MASS_RULES = ("free",) _OVERRIDABLE_FIELDS = frozenset( diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py new file mode 100644 index 000000000..60a5002df --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py @@ -0,0 +1,299 @@ +"""Release-role postures of the UK rowwise candidate driver (microcosm#823). + +One driver builds both UK dataset lines. The role a run declares fixes every +solve parameter that used to be a hard-wired constant or an argparse default: +the doctrine object, the clone count, the solve length and learning rate, the +seed, the target-weighting rule and its admissible overrides, the gate scope +and posture, the Logbook pipeline and attempt-id prefix, the staging operation +id, the release id and the output filenames. The driver checks the declared +role against the parameters it was given and refuses the other role's flags, +so a national cut can never be a dense run in disguise and vice versa. + +``national`` is the certified national line: no cloning, national targets +only, the calibration-seam doctrine (:mod:`national_doctrine`) and its six +in-process gates; the seam-shaped build record it emits is what +``tools/certify_uk_release_cut.py`` certifies. ``dense`` is the K-clone joint +national + local surface under the local doctrine (:mod:`local_doctrine`) +and the six local gates; its dense pool is the intermediate the exact-K +local-area cuts are drawn from. + +Every value here is a reviewed constant. Changing one is a doctrine change: +it must edit this module (and the pinned test), which forces review. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +import numpy as np + +from microcosm.build.uk_runtime.calibration_run import ( + UK_CALIBRATION_GATE_SCOPE, + UK_LOCAL_GATE_SCOPE, +) +from microcosm.build.uk_runtime.local_doctrine import ( + UK_LOCAL_CLONE_COUNT, + UK_LOCAL_SOLVE_DOCTRINE, + UK_LOCAL_SOLVE_EPOCHS, + UKLocalSolveDoctrine, +) +from microcosm.build.uk_runtime.national_doctrine import ( + UK_NATIONAL_SOLVE_DOCTRINE, + UKNationalSolveDoctrine, +) +from microcosm.build.uk_runtime.release_identity import ( + UK_DENSE_RELEASE_ID, + UK_NATIONAL_RELEASE_ID, +) + +__all__ = [ + "UK_ROWWISE_DENSE_POSTURE", + "UK_ROWWISE_NATIONAL_POSTURE", + "UK_ROWWISE_POSTURES", + "UK_ROWWISE_RELEASE_ROLES", + "UKRowwisePosture", + "uk_rowwise_posture", +] + +#: The dense role's Adam learning rate (microcosm#762 adjudication; the +#: argparse default the joint runs have always used). +UK_DENSE_LEARNING_RATE = 0.15 + +#: The dense role's clone-assignment and solve seed (the driver's default +#: since #762; every dense receipt reads ``s42``). +UK_DENSE_SEED = 42 + +#: The constituency vintage the dense role requires from the ladder. +UK_DENSE_CONSTITUENCY_VINTAGE = "2024_pcon" + + +@dataclass(frozen=True) +class UKRowwisePosture: + """The declared, reviewed parameters of one UK rowwise release role.""" + + role: str + doctrine: UKNationalSolveDoctrine | UKLocalSolveDoctrine + clone_count: int | None + epochs: int + learning_rate: float + seed: int + target_weight_rule: str + allowed_target_weight_rules: tuple[str, ...] + expected_constituency_vintage: str | None + gate_scope: tuple[str, ...] + gate_posture: str + gate_policy_suffix: str + pipeline: str + build_id_prefix: str + staging_operation_id: str + release_id: str + dataset_filename_template: str + gate_report_filename_template: str + evidence_shape: str + ladder_required: bool + local_rows: bool + holdout: bool + + def __post_init__(self) -> None: + if self.role not in UK_ROWWISE_RELEASE_ROLES: + raise ValueError( + f"posture role must be one of {UK_ROWWISE_RELEASE_ROLES}, " + f"got {self.role!r}." + ) + if self.clone_count is not None and ( + isinstance(self.clone_count, bool) + or not isinstance(self.clone_count, int) + or self.clone_count <= 0 + ): + raise ValueError( + f"posture clone_count must be None or a positive integer, " + f"got {self.clone_count!r}." + ) + if (self.clone_count is None) != (not self.ladder_required): + raise ValueError( + "a posture clones exactly when it requires the ladder; got " + f"clone_count={self.clone_count!r}, " + f"ladder_required={self.ladder_required!r}." + ) + if self.local_rows != self.ladder_required: + raise ValueError("local target rows exist exactly on the ladder role.") + if isinstance(self.epochs, bool) or not isinstance(self.epochs, int): + raise ValueError(f"posture epochs must be an integer, got {self.epochs!r}.") + if self.epochs <= 0: + raise ValueError(f"posture epochs must be positive, got {self.epochs!r}.") + if ( + isinstance(self.learning_rate, bool) + or not isinstance(self.learning_rate, int | float) + or not np.isfinite(self.learning_rate) + or self.learning_rate <= 0 + ): + raise ValueError( + "posture learning_rate must be a positive finite number, got " + f"{self.learning_rate!r}." + ) + if ( + isinstance(self.seed, bool) + or not isinstance(self.seed, int) + or self.seed < 0 + ): + raise ValueError( + f"posture seed must be a non-negative integer, got {self.seed!r}." + ) + if self.target_weight_rule not in self.allowed_target_weight_rules: + raise ValueError( + f"posture target_weight_rule {self.target_weight_rule!r} is not " + f"among its allowed rules {self.allowed_target_weight_rules}." + ) + if self.doctrine.target_weight_rule != self.target_weight_rule: + raise ValueError( + "the posture's target_weight_rule must be its doctrine's: " + f"{self.target_weight_rule!r} vs " + f"{self.doctrine.target_weight_rule!r}." + ) + if (self.expected_constituency_vintage is None) != (not self.ladder_required): + raise ValueError( + "a constituency vintage is expected exactly on the ladder role." + ) + if not self.gate_scope: + raise ValueError("posture gate_scope must name at least one gate.") + for name in ( + "gate_posture", + "gate_policy_suffix", + "pipeline", + "build_id_prefix", + "staging_operation_id", + "release_id", + "evidence_shape", + ): + if not str(getattr(self, name)).strip(): + raise ValueError(f"posture {name} must be non-empty.") + for name in ("dataset_filename_template", "gate_report_filename_template"): + if "{vintage}" not in getattr(self, name): + raise ValueError(f"posture {name} must carry the {{vintage}} token.") + + def dataset_filename(self, vintage: str) -> str: + """The role's dataset filename for one FRS release vintage (``2024_25``).""" + + return self.dataset_filename_template.format(vintage=_vintage(vintage)) + + def gate_report_filename(self, vintage: str) -> str: + """The role's terminal gate-report filename for one FRS release vintage.""" + + return self.gate_report_filename_template.format(vintage=_vintage(vintage)) + + def doctrine_bounds(self) -> dict[str, Any]: + """The doctrine block the run records and a release pre-flight checks. + + The dense block is exactly the shape the dense pre-flight has pinned + since microcosm#762 (six keys); the national block adds the seam + doctrine's remaining reviewed fields and records ``clone_count`` as + ``None`` so a national manifest can never read as a clone run. + """ + + bounds: dict[str, Any] = { + "target_loss_cap": float(self.doctrine.target_loss_cap), + "max_weight_ratio": ( + None + if self.doctrine.max_weight_ratio is None + else float(self.doctrine.max_weight_ratio) + ), + "scale_rule": str(self.doctrine.scale_rule), + "target_weight_rule": str(self.doctrine.target_weight_rule), + "solve_epochs": int(self.epochs), + "clone_count": None if self.clone_count is None else int(self.clone_count), + } + if isinstance(self.doctrine, UKNationalSolveDoctrine): + bounds.update( + { + "learning_rate": float(self.doctrine.learning_rate), + "seed": int(self.doctrine.seed), + "mass_rule": str(self.doctrine.mass_rule), + "l0_lambda": float(self.doctrine.l0_lambda), + } + ) + return bounds + + +def _vintage(vintage: str) -> str: + text = str(vintage) + if ( + len(text) != 7 + or not text[:4].isdigit() + or text[4] != "_" + or not text[5:].isdigit() + ): + raise ValueError(f"an FRS release vintage reads YYYY_YY, got {vintage!r}.") + return text + + +UK_ROWWISE_RELEASE_ROLES = ("national", "dense") + +UK_ROWWISE_NATIONAL_POSTURE = UKRowwisePosture( + role="national", + doctrine=UK_NATIONAL_SOLVE_DOCTRINE, + clone_count=None, + epochs=UK_NATIONAL_SOLVE_DOCTRINE.epochs, + learning_rate=UK_NATIONAL_SOLVE_DOCTRINE.learning_rate, + seed=UK_NATIONAL_SOLVE_DOCTRINE.seed, + target_weight_rule=UK_NATIONAL_SOLVE_DOCTRINE.target_weight_rule, + allowed_target_weight_rules=("family_equal", "uniform"), + expected_constituency_vintage=None, + gate_scope=tuple(UK_CALIBRATION_GATE_SCOPE), + gate_posture="calibration_seam", + gate_policy_suffix="calibration_seam_scope", + pipeline="uk-frs-calibration", + build_id_prefix="uk-frs-calibration-attempt-", + staging_operation_id="uk_national_calibration", + release_id=UK_NATIONAL_RELEASE_ID, + dataset_filename_template="microcosm_uk_{vintage}.h5", + gate_report_filename_template="microcosm_uk_{vintage}.terminal_gates.json", + evidence_shape="calibration_seam_build_record", + ladder_required=False, + local_rows=False, + holdout=False, +) + +UK_ROWWISE_DENSE_POSTURE = UKRowwisePosture( + role="dense", + doctrine=UK_LOCAL_SOLVE_DOCTRINE, + clone_count=UK_LOCAL_CLONE_COUNT, + epochs=UK_LOCAL_SOLVE_EPOCHS, + learning_rate=UK_DENSE_LEARNING_RATE, + seed=UK_DENSE_SEED, + target_weight_rule=UK_LOCAL_SOLVE_DOCTRINE.target_weight_rule, + allowed_target_weight_rules=("grain_equal", "uniform"), + expected_constituency_vintage=UK_DENSE_CONSTITUENCY_VINTAGE, + gate_scope=tuple(UK_LOCAL_GATE_SCOPE), + gate_posture="local_candidate", + gate_policy_suffix="local_candidate", + pipeline="uk-local-candidate", + build_id_prefix="uk-local-candidate-", + staging_operation_id="uk_rowwise_candidate", + release_id=UK_DENSE_RELEASE_ID, + # The joint-surface file keeps the ``_local`` family name: the size + # evaluator and the dense pre-flight glob ``microcosm_uk_*_local.h5`` / + # ``*.local_gates.json`` across dense and exact-count runs alike, and the + # published dense artifact is minted by the assembler. + dataset_filename_template="microcosm_uk_{vintage}_local.h5", + gate_report_filename_template="microcosm_uk_{vintage}_local.local_gates.json", + evidence_shape="rowwise_candidate_manifest", + ladder_required=True, + local_rows=True, + holdout=True, +) + +UK_ROWWISE_POSTURES: dict[str, UKRowwisePosture] = { + UK_ROWWISE_NATIONAL_POSTURE.role: UK_ROWWISE_NATIONAL_POSTURE, + UK_ROWWISE_DENSE_POSTURE.role: UK_ROWWISE_DENSE_POSTURE, +} + + +def uk_rowwise_posture(role: object) -> UKRowwisePosture: + """The posture of a declared release role, refusing any other token.""" + + if not isinstance(role, str) or role not in UK_ROWWISE_POSTURES: + raise ValueError( + f"release role must be one of {UK_ROWWISE_RELEASE_ROLES}, got {role!r}." + ) + return UK_ROWWISE_POSTURES[role] diff --git a/packages/microcosm-build/tests/test_gate_battery_contract_pins.py b/packages/microcosm-build/tests/test_gate_battery_contract_pins.py index cdf822c1b..bfabcf43e 100644 --- a/packages/microcosm-build/tests/test_gate_battery_contract_pins.py +++ b/packages/microcosm-build/tests/test_gate_battery_contract_pins.py @@ -391,6 +391,24 @@ def test_release_id_mirrors_the_build_shard(self) -> None: ) assert data_contract._UK_DENSE_CUT_TAG_RE.fullmatch(UK_DENSE_RELEASE_ID) is None + def test_dataset_filename_mirrors_the_assembler(self) -> None: + import importlib.util + from pathlib import Path + + spec = importlib.util.spec_from_file_location( + "assemble_uk_dense_release_dir", + Path(__file__).resolve().parents[3] + / "tools" + / "assemble_uk_dense_release_dir.py", + ) + assembler = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(assembler) + assert data_contract._UK_DENSE_DATASET_FILENAME == assembler._DATASET_FILENAME + assert ( + data_contract._UK_DENSE_DATASET_FILENAME == "microcosm_uk_2024_25_dense.h5" + ) + def test_entry_ids_mirror_the_local_battery_scope(self) -> None: from microcosm.build.uk_runtime.calibration_run import UK_LOCAL_GATE_SCOPE diff --git a/packages/microcosm-build/tests/test_staging_v2.py b/packages/microcosm-build/tests/test_staging_v2.py index 466908215..37ee574a2 100644 --- a/packages/microcosm-build/tests/test_staging_v2.py +++ b/packages/microcosm-build/tests/test_staging_v2.py @@ -759,11 +759,32 @@ def test_version_2_cli_uses_country_owned_repository_configuration(monkeypatch): args = parser.parse_args([]) assert args.staging_repo_id == "example/configured-staging" + assert args.staging_upload_interval_seconds == 30.0 assert not hasattr(args, "staging_prefix") with pytest.raises(SystemExit): parser.parse_args(["--staging-prefix", "candidate-runs"]) +def test_version_2_cli_lets_a_long_running_command_slow_its_upload_cadence(): + """A multi-hour solve at 30 s exhausts the Hub's commit budget (v20 run).""" + + repository = StagingRepositoryConfig( + default_repo_id="example/default-staging", + repo_id_environment_variable="EXAMPLE_STAGING_REPO_ID", + ) + parser = argparse.ArgumentParser() + add_staging_arguments( + parser, repository=repository, default_upload_interval_seconds=300 + ) + assert parser.parse_args([]).staging_upload_interval_seconds == 300.0 + assert ( + parser.parse_args( + ["--staging-upload-interval-seconds", "45"] + ).staging_upload_interval_seconds + == 45.0 + ) + + def test_typed_artifacts_reject_one_scalar_individual_record(tmp_path): telemetry = _recorder(tmp_path) artifact = tmp_path / "individual.json" diff --git a/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py b/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py index f24377d61..b0c5be921 100644 --- a/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py +++ b/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py @@ -219,7 +219,7 @@ def fake_run(**kwargs): assert call["calibration_year"] == 2025 assert call["exclusion_receipt"] == {"excluded": "reviewed"} assert call["doctrine"].epochs == 128 - assert call["doctrine_overrides"] == {"epochs": {"default": 256, "effective": 128}} + assert call["doctrine_overrides"] == {"epochs": {"default": 1500, "effective": 128}} assert isinstance(call["measure_resolver"], FakeResolver) assert ( call["measure_resolver"].kwargs["simulation_source"] == call["paths"].input_h5 diff --git a/packages/microcosm-build/tests/test_uk_dense_release_assembler.py b/packages/microcosm-build/tests/test_uk_dense_release_assembler.py index 4fdaae9d0..1d79862dd 100644 --- a/packages/microcosm-build/tests/test_uk_dense_release_assembler.py +++ b/packages/microcosm-build/tests/test_uk_dense_release_assembler.py @@ -117,7 +117,7 @@ def _candidate_dir(root: Path) -> tuple[Path, Path, Path]: (candidate / "logbook-spool").mkdir(parents=True) spine = root / "spine.h5" spine.write_bytes(b"spine-stand-in") - h5 = candidate / "microcosm_uk_2025_local.h5" + h5 = candidate / "microcosm_uk_2024_25_local.h5" h5.write_bytes(b"dense-candidate-stand-in") diagnostics = { "schema_version": 6, @@ -143,7 +143,7 @@ def _candidate_dir(root: Path) -> tuple[Path, Path, Path]: } (candidate / "calibration_diagnostics.json").write_text(json.dumps(diagnostics)) report = _signed_report() - (candidate / "microcosm_uk_2025_local.local_gates.json").write_text( + (candidate / "microcosm_uk_2024_25_local.local_gates.json").write_text( json.dumps(report) ) (candidate / "score_vs_incumbent.json").write_text( @@ -171,7 +171,8 @@ def _candidate_dir(root: Path) -> tuple[Path, Path, Path]: ) ) manifest = { - "schema_version": 2, + "schema_version": 4, + "release_role": "dense", "created_at": "2026-09-03T10:54:22+00:00", "git_commit": "b" * 40, "parameters": { @@ -326,8 +327,10 @@ def _candidate_dir(root: Path) -> tuple[Path, Path, Path]: "sha256": _sha(candidate / "calibration_diagnostics.json"), }, "local_gate_report": { - "path": str(candidate / "microcosm_uk_2025_local.local_gates.json"), - "sha256": _sha(candidate / "microcosm_uk_2025_local.local_gates.json"), + "path": str(candidate / "microcosm_uk_2024_25_local.local_gates.json"), + "sha256": _sha( + candidate / "microcosm_uk_2024_25_local.local_gates.json" + ), }, }, } @@ -400,20 +403,20 @@ def test_assembler_stages_a_contract_valid_dense_release( assert release_dir.is_dir() for name in dc._UK_DENSE_REQUIRED_RELEASE_FILES: assert (release_dir / name).is_file(), name - published = candidate / "microcosm_uk_2025_dense.h5" + published = candidate / "microcosm_uk_2024_25_dense.h5" assert ( published.read_bytes() - == (candidate / "microcosm_uk_2025_local.h5").read_bytes() + == (candidate / "microcosm_uk_2024_25_local.h5").read_bytes() ) manifest = json.loads((release_dir / "release_manifest.json").read_text()) assert manifest["dataset_role"] == "non_default_local_area" assert manifest["default_datasets"] == {} and manifest["is_default"] is False assert ( - manifest["artifacts"]["microcosm_uk_2025_dense"]["revision"] + manifest["artifacts"]["microcosm_uk_2024_25_dense"]["revision"] == summary["cut_tag"] ) assert ( - manifest["artifacts"]["microcosm_uk_2025_dense"]["repo_id"] + manifest["artifacts"]["microcosm_uk_2024_25_dense"]["repo_id"] == "policyengine/populace-uk-private" ) assert any( @@ -549,7 +552,7 @@ def test_assembler_refuses_tampered_candidate_bytes( monkeypatch.setenv("MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY", KEY) assembler = _load("assemble_uk_dense_release_dir") candidate, spine, incumbent = _candidate_dir(tmp_path) - (candidate / "microcosm_uk_2025_local.h5").write_bytes(b"tampered") + (candidate / "microcosm_uk_2024_25_local.h5").write_bytes(b"tampered") with pytest.raises(SystemExit, match="candidate bytes"): assembler.main( [ @@ -752,3 +755,45 @@ def test_assembler_rejects_expired_or_missing_measure_approval( _load("assemble_uk_dense_release_dir").main( _assemble_args(candidate, spine, incumbent, tmp_path / "r") ) + + +def test_assembler_refuses_a_national_role_manifest( + tmp_path: Path, monkeypatch +) -> None: + """The dense assembler covers the dense line only (microcosm#823).""" + + monkeypatch.setenv("MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY", KEY) + assembler = _load("assemble_uk_dense_release_dir") + candidate, spine, incumbent = _candidate_dir(tmp_path) + manifest_path = candidate / "rowwise_candidate_manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["release_role"] = "national" + manifest_path.write_text(json.dumps(manifest)) + with pytest.raises(SystemExit, match="release_role is 'national'"): + assembler.main( + [ + "--candidate-dir", + str(candidate), + "--spine-h5", + str(spine), + "--incumbent-manifest", + str(incumbent), + "--out-dir", + str(tmp_path / "releases"), + ] + ) + del manifest["release_role"] + manifest_path.write_text(json.dumps(manifest)) + with pytest.raises(SystemExit, match="release_role is None"): + assembler.main( + [ + "--candidate-dir", + str(candidate), + "--spine-h5", + str(spine), + "--incumbent-manifest", + str(incumbent), + "--out-dir", + str(tmp_path / "releases"), + ] + ) diff --git a/packages/microcosm-build/tests/test_uk_local_release_preflight.py b/packages/microcosm-build/tests/test_uk_local_release_preflight.py index 1b11b918c..76169bff2 100644 --- a/packages/microcosm-build/tests/test_uk_local_release_preflight.py +++ b/packages/microcosm-build/tests/test_uk_local_release_preflight.py @@ -28,11 +28,12 @@ def _good_candidate(tmp_path: Path) -> Path: candidate = tmp_path / "candidate" (candidate / "logbook-spool").mkdir(parents=True) (candidate / "logbook-spool" / "row.json").write_text("{}") - h5 = candidate / "microcosm_uk_2025_local.h5" + h5 = candidate / "microcosm_uk_2024_25_local.h5" h5.write_bytes(b"h5") import hashlib manifest = { + "release_role": "dense", "parameters": { "release_candidate": True, "epochs": 1500, @@ -117,7 +118,7 @@ def _good_candidate(tmp_path: Path) -> Path: }, } (candidate / "rowwise_candidate_manifest.json").write_text(json.dumps(manifest)) - (candidate / "microcosm_uk_2025_local.local_gates.json").write_text( + (candidate / "microcosm_uk_2024_25_local.local_gates.json").write_text( json.dumps(_signed_report()) ) return candidate @@ -206,6 +207,24 @@ def test_good_candidate_dir_passes(tmp_path: Path) -> None: ) +@pytest.mark.parametrize("role", ["national", None]) +def test_preflight_refuses_a_national_role_manifest(tmp_path: Path, role) -> None: + """This pre-flight is the dense line's; any other role is refused by name.""" + + module = _load() + candidate = _good_candidate(tmp_path) + manifest_path = candidate / "rowwise_candidate_manifest.json" + manifest = json.loads(manifest_path.read_text()) + if role is None: + del manifest["release_role"] + else: + manifest["release_role"] = role + manifest_path.write_text(json.dumps(manifest)) + failures = module.check_candidate_dir(candidate, today=date(2026, 9, 4)) + assert len(failures) == 1 + assert f"manifest.release_role is {role!r}" in failures[0] + + @pytest.mark.parametrize( ("mutate", "needle"), [ @@ -254,11 +273,11 @@ def test_each_missing_flag_is_named(tmp_path: Path, mutate, needle: str) -> None candidate = _good_candidate(tmp_path) manifest = json.loads((candidate / "rowwise_candidate_manifest.json").read_text()) report = json.loads( - (candidate / "microcosm_uk_2025_local.local_gates.json").read_text() + (candidate / "microcosm_uk_2024_25_local.local_gates.json").read_text() ) mutate(manifest, report) (candidate / "rowwise_candidate_manifest.json").write_text(json.dumps(manifest)) - (candidate / "microcosm_uk_2025_local.local_gates.json").write_text( + (candidate / "microcosm_uk_2024_25_local.local_gates.json").write_text( json.dumps(report) ) failures = module.check_candidate_dir(candidate, today=date(2026, 9, 4)) diff --git a/packages/microcosm-build/tests/test_uk_national_calibration.py b/packages/microcosm-build/tests/test_uk_national_calibration.py index 76dce2a8a..53af50b7d 100644 --- a/packages/microcosm-build/tests/test_uk_national_calibration.py +++ b/packages/microcosm-build/tests/test_uk_national_calibration.py @@ -862,20 +862,23 @@ def test_post_solve_fence_requires_calibrated_kind_and_mass_record() -> None: def test_national_doctrine_constants_are_the_declared_contract() -> None: - assert UK_NATIONAL_SOLVE_EPOCHS == 256 + # Ruling 2026-09-20 (microcosm#823): the certified-cut posture every + # campaign national run overrode to — 1,500 epochs, family_equal — is + # the doctrine, so a certified national cut records no overrides. + assert UK_NATIONAL_SOLVE_EPOCHS == 1500 assert UK_NATIONAL_LEARNING_RATE == 0.02 assert UK_NATIONAL_MAX_WEIGHT_RATIO == 10.0 assert UK_NATIONAL_SEED == 0 assert UK_NATIONAL_TARGET_LOSS_CAP == 10.0 assert UK_NATIONAL_L0_LAMBDA == 0.0 assert UK_NATIONAL_MASS_RULE == "free" - # María's ruling (2026-08-24): family_equal is vocabulary, never the - # default — she passes it as an explicit per-run override. - assert UK_NATIONAL_TARGET_WEIGHT_RULE == "uniform" + # "uniform" (the 2026-08-24 default) stays vocabulary: an explicit, + # receipted per-run override, never silently reachable. + assert UK_NATIONAL_TARGET_WEIGHT_RULE == "family_equal" assert UK_NATIONAL_SOLVE_DOCTRINE == UKNationalSolveDoctrine() assert UK_NATIONAL_SOLVE_DOCTRINE.scale_rule == "default_target_loss_scales" - assert UK_NATIONAL_SOLVE_DOCTRINE.target_weight_rule == "uniform" - assert UKNationalSolveDoctrine(target_weight_rule="family_equal") + assert UK_NATIONAL_SOLVE_DOCTRINE.target_weight_rule == "family_equal" + assert UKNationalSolveDoctrine(target_weight_rule="uniform") def test_uk_doctrine_with_overrides_receipts_effective_diffs_only() -> None: @@ -893,18 +896,18 @@ def test_uk_doctrine_with_overrides_receipts_effective_diffs_only() -> None: doctrine, receipt = uk_doctrine_with_overrides( epochs=128, learning_rate=0.01, - target_weight_rule="family_equal", + target_weight_rule="uniform", target_loss_cap=5.0, ) assert doctrine.epochs == 128 assert doctrine.learning_rate == 0.01 - assert doctrine.target_weight_rule == "family_equal" + assert doctrine.target_weight_rule == "uniform" assert doctrine.target_loss_cap == 5.0 assert receipt == { - "epochs": {"default": 256, "effective": 128}, + "epochs": {"default": 1500, "effective": 128}, "learning_rate": {"default": 0.02, "effective": 0.01}, "target_loss_cap": {"default": 10.0, "effective": 5.0}, - "target_weight_rule": {"default": "uniform", "effective": "family_equal"}, + "target_weight_rule": {"default": "family_equal", "effective": "uniform"}, } diff --git a/packages/microcosm-build/tests/test_uk_rowwise_candidate.py b/packages/microcosm-build/tests/test_uk_rowwise_candidate.py index ccf011108..5cb41fc0e 100644 --- a/packages/microcosm-build/tests/test_uk_rowwise_candidate.py +++ b/packages/microcosm-build/tests/test_uk_rowwise_candidate.py @@ -455,6 +455,8 @@ def test_candidate_build_writes_calibrated_h5_and_evidence( [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -471,9 +473,7 @@ def test_candidate_build_writes_calibrated_h5_and_evidence( == 0 ) - candidate_h5 = output_dir / builder.CANDIDATE_FILENAME_TEMPLATE.format( - calibration_year=2025 - ) + candidate_h5 = output_dir / "microcosm_uk_2024_25_local.h5" expected_sidecars = { builder.MANIFEST_FILENAME, builder.SOLVE_DIAGNOSTICS_FILENAME, @@ -481,7 +481,7 @@ def test_candidate_build_writes_calibrated_h5_and_evidence( builder.PAST_CAP_FILENAME, builder.CALIBRATION_DIAGNOSTICS_FILENAME, builder.LOCAL_REGISTRY_FILENAME, - builder.LOCAL_GATE_REPORT_FILENAME_TEMPLATE.format(calibration_year=2025), + "microcosm_uk_2024_25_local.local_gates.json", } assert candidate_h5.exists() assert expected_sidecars <= {path.name for path in output_dir.iterdir()} @@ -509,6 +509,20 @@ def test_candidate_build_writes_calibrated_h5_and_evidence( assert record.declared_factor == pytest.approx(record.new_total / record.old_total) manifest = json.loads((output_dir / builder.MANIFEST_FILENAME).read_text()) + # The manifest declares the role it was built under (microcosm#823): the + # dense pre-flight and assembler refuse any other, and the parameters + # carry the role's doctrine block verbatim. + assert manifest["schema_version"] == 4 + assert manifest["release_role"] == "dense" + assert manifest["release_id"] == "microcosm-uk-2024-25-dense" + assert manifest["parameters"]["release_role"] == "dense" + assert manifest["parameters"]["n_clones"] == 2 + assert manifest["parameters"]["doctrine"] == ( + builder.UK_ROWWISE_DENSE_POSTURE.doctrine_bounds() + ) + assert manifest["outputs"]["dataset"]["path"].endswith( + "microcosm_uk_2024_25_local.h5" + ) assert manifest["candidate_scope"] == "adjudicated_partial" assert manifest["bound_target_families"] == ["census_households/constituency"] adjudications = manifest["binding_adjudications"] @@ -676,6 +690,8 @@ def forbidden(*_args, **_kwargs): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -787,19 +803,23 @@ def compact_sampler_forbidden(*_args, **_kwargs): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", str(output_dir), ] ).n_clones - == builder.UK_LOCAL_CLONE_COUNT + == builder.UK_ROWWISE_DENSE_POSTURE.clone_count ) with pytest.raises(ValueError, match="must equal --n-clones"): builder.main( [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -817,6 +837,8 @@ def compact_sampler_forbidden(*_args, **_kwargs): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -878,6 +900,8 @@ def test_candidate_clone_count_planning_is_dry_run_only(tmp_path) -> None: [ "--input-h5", str(tmp_path / "missing.h5"), + "--release-role", + "dense", "--ladder", str(tmp_path / "missing.npz"), "--out", @@ -1361,6 +1385,8 @@ def support_summary(household, ladder_arg): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1402,6 +1428,8 @@ def support_summary(household, ladder_arg): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1417,7 +1445,7 @@ def support_summary(household, ladder_arg): == 0 ) f100 = json.loads((f100_out / builder.MANIFEST_FILENAME).read_text()) - assert f100["schema_version"] == 3 + assert f100["schema_version"] == 4 # The written rowwise artifact carries the shared ``clone_index`` name on # every table: the compact national loader must refuse it (flattening # rule) and the rowwise reader must undo the export rename. @@ -1489,6 +1517,8 @@ def support_summary(household, ladder_arg): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1558,6 +1588,8 @@ def failing_gate(*_args, **_kwargs): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1576,9 +1608,7 @@ def failing_gate(*_args, **_kwargs): assert len(rows) == 1 row = rows[0] assert row.disposition == "failed" - gate_report_path = output_dir / builder.LOCAL_GATE_REPORT_FILENAME_TEMPLATE.format( - calibration_year=2025 - ) + gate_report_path = output_dir / "microcosm_uk_2024_25_local.local_gates.json" assert gate_report_path.exists() assert row.gate_verdicts["uk_local_geography_ladder_post_calibration"] == { "verdict": "failed", @@ -1620,6 +1650,8 @@ def test_candidate_binding_adjudication_failure_records_failed_row( [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1674,6 +1706,8 @@ def failing_ladder_load(_path): [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1814,6 +1848,8 @@ def test_candidate_dry_run_refuses_ladder_sidecar_collision( [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -1837,8 +1873,8 @@ def test_candidate_publication_rolls_back_on_interrupt( staging_dir.mkdir() output_paths = builder._output_paths( output_dir, - source_year=2023, - calibration_year=2025, + posture=builder.UK_ROWWISE_DENSE_POSTURE, + vintage="2024_25", ) staged = {key: staging_dir / path.name for key, path in output_paths.items()} for path in staged.values(): @@ -1871,6 +1907,8 @@ def _joint_f100_args(input_h5: Path, ladder_path: Path, output_dir: Path) -> lis return [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -2066,6 +2104,8 @@ def test_release_candidate_refuses_non_doctrine_solve_settings(tmp_path) -> None base = [ "--input-h5", str(tmp_path / "spine.h5"), + "--release-role", + "dense", "--input-sha256", pin, "--ladder", @@ -2085,8 +2125,8 @@ def test_release_candidate_refuses_non_doctrine_solve_settings(tmp_path) -> None # The doctrine defaults are the release posture: nothing to refuse. args = builder._parse_args(base) builder._validate_cli_args(args) - assert args.n_clones == builder.UK_LOCAL_CLONE_COUNT == 15 - assert args.epochs == builder.UK_LOCAL_SOLVE_EPOCHS == 1500 + assert args.n_clones == builder.UK_ROWWISE_DENSE_POSTURE.clone_count == 15 + assert args.epochs == builder.UK_ROWWISE_DENSE_POSTURE.epochs == 1500 assert args.target_weight_rule == "grain_equal" with pytest.raises(ValueError, match=r"--epochs != doctrine 1500"): @@ -2105,6 +2145,8 @@ def test_candidate_requires_pinned_ledger_inputs(tmp_path) -> None: [ "--input-h5", str(tmp_path / "spine.h5"), + "--release-role", + "dense", "--input-sha256", "0" * 64, "--ladder", @@ -2201,6 +2243,8 @@ def test_size_candidate_exports_compact_links_and_cannot_claim_dense_release( [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), *flags, @@ -2231,7 +2275,7 @@ def test_size_candidate_exports_compact_links_and_cannot_claim_dense_release( manifest["weights"]["stretch_reference"] == "normalized_horvitz_thompson_w_over_q" ) - path = out / builder.CANDIDATE_FILENAME_TEMPLATE.format(calibration_year=2025) + path = out / "microcosm_uk_2024_25_local.h5" with pd.HDFStore(path, "r") as store: households = store["household"] persons = store["person"] @@ -2307,6 +2351,8 @@ def test_selection_seed_requires_a_dataset_size(tmp_path): [ "--input-h5", str(tmp_path / "spine.h5"), + "--release-role", + "dense", "--ladder", str(tmp_path / "ladder.npz"), "--out", @@ -2336,6 +2382,8 @@ def test_selection_pi_hi_is_candidate_only_and_bounded(tmp_path, argv_tail, mess [ "--input-h5", str(tmp_path / "spine.h5"), + "--release-role", + "dense", "--ladder", str(tmp_path / "ladder.npz"), "--out", @@ -2349,7 +2397,10 @@ def test_selection_pi_hi_is_candidate_only_and_bounded(tmp_path, argv_tail, mess def test_dense_candidate_manifest_has_no_size_sidecars(tmp_path): builder = _load_builder_module() - paths = builder._output_paths(tmp_path, source_year=2024, calibration_year=2025) + paths = builder._output_paths( + tmp_path, posture=builder.UK_ROWWISE_DENSE_POSTURE, vintage="2024_25" + ) + assert paths["dataset"].name == "microcosm_uk_2024_25_local.h5" assert paths["dense_reference"].name == builder.DENSE_REFERENCE_DIAGNOSTICS_FILENAME assert paths["selection"].name == builder.DATASET_SIZE_SELECTION_FILENAME assert builder._SIZE_RUN_ONLY_OUTPUTS == {"dense_reference", "selection"} @@ -2361,6 +2412,8 @@ def test_size_cli_refuses_promotion_without_separate_certification(tmp_path): [ "--input-h5", str(tmp_path / "spine.h5"), + "--release-role", + "dense", "--ladder", str(tmp_path / "ladder.npz"), "--out", @@ -2402,6 +2455,8 @@ def test_size_candidate_checkpoints_before_the_draw_and_resumes_from_it( common = [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), *_configure_households_only_inputs( @@ -2434,7 +2489,10 @@ def test_size_candidate_checkpoints_before_the_draw_and_resumes_from_it( assert checkpoint["identity"]["epochs"] == 2 # The identity carries the solve doctrine; the provenance names the # writing run (reported on resume, not compared). - assert checkpoint["identity"]["doctrine"] == builder._doctrine_bounds() + assert checkpoint["identity"]["doctrine"] == builder._doctrine_bounds( + builder.UK_ROWWISE_DENSE_POSTURE + ) + assert checkpoint["identity"]["release_role"] == "dense" assert set(checkpoint["provenance"]) == {"code_pin", "build_id"} manifest = json.loads((first / builder.MANIFEST_FILENAME).read_text()) written = manifest["solve"]["dataset_size"]["checkpoint"]["written"] @@ -2530,6 +2588,8 @@ def test_size_candidate_checkpoints_before_the_draw_and_resumes_from_it( [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), "--out", @@ -2693,6 +2753,8 @@ def _build_args(input_h5, ladder_path, flags, out, *extra): return [ "--input-h5", str(input_h5), + "--release-role", + "dense", "--ladder", str(ladder_path), *flags, @@ -3292,3 +3354,145 @@ def repo_info(self, **kwargs): plan = json.loads(capsys.readouterr().out) assert plan["parameters"]["dataset_households"] is None assert not out.exists() + + +def _role_argv(tmp_path: Path, role: str, *extra: str) -> list[str]: + return [ + "--input-h5", + str(tmp_path / "spine.h5"), + "--release-role", + role, + "--input-sha256", + "2" * 64, + "--ledger-facts", + str(tmp_path / "ledger"), + "--ledger-facts-sha256", + "0" * 64, + "--ledger-manifest-sha256", + "1" * 64, + "--out", + str(tmp_path / "out"), + *extra, + ] + + +def _dense_argv(tmp_path: Path, *extra: str) -> list[str]: + return _role_argv( + tmp_path, + "dense", + "--ladder", + str(tmp_path / "ladder.npz"), + "--ladder-sha256", + "3" * 64, + *extra, + ) + + +def test_release_role_is_required(tmp_path) -> None: + builder = _load_builder_module() + argv = _dense_argv(tmp_path) + argv.remove("--release-role") + argv.remove("dense") + with pytest.raises(SystemExit): + builder._parse_args(argv) + with pytest.raises(SystemExit): + builder._parse_args([*argv, "--release-role", "local"]) + + +def test_release_role_supplies_the_solve_defaults(tmp_path) -> None: + builder = _load_builder_module() + dense = builder._parse_args(_dense_argv(tmp_path)) + posture = builder.UK_ROWWISE_DENSE_POSTURE + assert (dense.n_clones, dense.seed, dense.epochs, dense.learning_rate) == ( + posture.clone_count, + posture.seed, + posture.epochs, + posture.learning_rate, + ) + assert dense.target_weight_rule == "grain_equal" + assert dense.expected_constituency_vintage == "2024_pcon" + assert dense.staging_upload_interval_seconds == 300.0 + assert dense._explicit_arguments == frozenset() + builder._validate_cli_args(dense) + + national = builder._parse_args(_role_argv(tmp_path, "national")) + posture = builder.uk_rowwise_posture("national") + assert national._posture is posture + assert national.n_clones is None + assert (national.seed, national.epochs, national.learning_rate) == (0, 1500, 0.02) + assert national.target_weight_rule == "family_equal" + assert national.expected_constituency_vintage is None + builder._validate_cli_args(national) + explicit = builder._parse_args(_role_argv(tmp_path, "national", "--epochs", "5")) + assert explicit.epochs == 5 + assert explicit._explicit_arguments == frozenset({"epochs"}) + + +@pytest.mark.parametrize( + ("extra", "needle"), + [ + (["--target-loss-cap", "5"], "--target-loss-cap"), + (["--allow-unpinned-feed"], "--allow-unpinned-feed"), + (["--target-weight-rule", "family_equal"], "--target-weight-rule family_equal"), + ], +) +def test_dense_role_refusal_table(tmp_path, extra, needle) -> None: + builder = _load_builder_module() + args = builder._parse_args(_dense_argv(tmp_path, *extra)) + with pytest.raises(ValueError, match="--release-role dense refuses") as excinfo: + builder._validate_cli_args(args) + assert needle in str(excinfo.value) + + +def test_dense_role_requires_the_ladder(tmp_path) -> None: + builder = _load_builder_module() + argv = _role_argv(tmp_path, "dense", "--ladder-sha256", "3" * 64) + with pytest.raises(ValueError, match="requires --ladder"): + builder._validate_cli_args(builder._parse_args(argv)) + + +@pytest.mark.parametrize( + ("extra", "needle"), + [ + (["--ladder", "ladder.npz"], "--ladder"), + (["--ladder-sha256", "3" * 64], "--ladder-sha256"), + (["--expected-constituency-vintage", "2024_pcon"], "--expected-constituency"), + (["--source-year", "2023"], "--source-year"), + (["--source-lineage-modulus", "7"], "--source-lineage-modulus"), + (["--n-clones", "15"], "--n-clones"), + (["--candidate-clone-counts", "2,4", "--dry-run"], "--candidate-clone-counts"), + (["--engine-blocks", "2"], "--engine-blocks"), + (["--households-only"], "--households-only"), + (["--skip-holdout"], "--skip-holdout"), + (["--dataset-households", "10"], "--dataset-households"), + (["--selection-seed", "3"], "--selection-seed"), + (["--selection-pi-hi", "0.5"], "--selection-pi-hi"), + (["--baseline-pi-floor", "0.1"], "--baseline-pi-floor"), + (["--no-size-checkpoint"], "--no-size-checkpoint"), + (["--resume-size-checkpoint", "dir"], "--resume-size-checkpoint"), + (["--sample-fraction", "0.1"], "--sample-fraction"), + (["--sample-seed", "9"], "--sample-seed"), + (["--target-weight-rule", "grain_equal"], "--target-weight-rule grain_equal"), + ], +) +def test_national_role_refusal_table(tmp_path, extra, needle) -> None: + builder = _load_builder_module() + args = builder._parse_args(_role_argv(tmp_path, "national", *extra)) + with pytest.raises(ValueError, match="--release-role national refuses") as excinfo: + builder._validate_cli_args(args) + assert needle in str(excinfo.value) + + +def test_national_role_refuses_release_candidate_with_the_seam_reason(tmp_path): + builder = _load_builder_module() + args = builder._parse_args(_role_argv(tmp_path, "national", "--release-candidate")) + with pytest.raises(ValueError, match="cannot sign shippability"): + builder._validate_cli_args(args) + + +def test_national_role_build_is_not_yet_wired(tmp_path) -> None: + """Declared and validated, but the seam delegation is the next increment.""" + + builder = _load_builder_module() + with pytest.raises(NotImplementedError, match="microcosm#823"): + builder.main(_role_argv(tmp_path, "national", "--staging-local-only")) diff --git a/packages/microcosm-build/tests/test_uk_rowwise_posture.py b/packages/microcosm-build/tests/test_uk_rowwise_posture.py new file mode 100644 index 000000000..8d01e3c60 --- /dev/null +++ b/packages/microcosm-build/tests/test_uk_rowwise_posture.py @@ -0,0 +1,185 @@ +"""The UK rowwise release-role postures (microcosm#823).""" + +from __future__ import annotations + +import dataclasses + +import pytest + +from microcosm.build.uk_runtime.calibration_run import ( + UK_CALIBRATION_GATE_SCOPE, + UK_LOCAL_GATE_SCOPE, +) +from microcosm.build.uk_runtime.local_doctrine import ( + UK_LOCAL_CLONE_COUNT, + UK_LOCAL_MAX_WEIGHT_RATIO, + UK_LOCAL_SOLVE_DOCTRINE, + UK_LOCAL_SOLVE_EPOCHS, + UK_LOCAL_TARGET_LOSS_CAP, +) +from microcosm.build.uk_runtime.national_doctrine import ( + UK_NATIONAL_SOLVE_DOCTRINE, + uk_doctrine_with_overrides, +) +from microcosm.build.uk_runtime.release_identity import ( + UK_DENSE_RELEASE_ID, + UK_NATIONAL_RELEASE_ID, +) +from microcosm.build.uk_runtime.rowwise_posture import ( + UK_ROWWISE_DENSE_POSTURE, + UK_ROWWISE_NATIONAL_POSTURE, + UK_ROWWISE_POSTURES, + UK_ROWWISE_RELEASE_ROLES, + UKRowwisePosture, + uk_rowwise_posture, +) + + +def test_uk_dense_posture_mirrors_the_ruled_constants() -> None: + posture = UK_ROWWISE_DENSE_POSTURE + assert posture.role == "dense" + assert posture.doctrine is UK_LOCAL_SOLVE_DOCTRINE + assert posture.clone_count == UK_LOCAL_CLONE_COUNT == 15 + assert posture.epochs == UK_LOCAL_SOLVE_EPOCHS == 1500 + assert posture.learning_rate == 0.15 + assert posture.seed == 42 + assert posture.target_weight_rule == "grain_equal" + assert posture.allowed_target_weight_rules == ("grain_equal", "uniform") + assert posture.expected_constituency_vintage == "2024_pcon" + assert posture.gate_scope == tuple(UK_LOCAL_GATE_SCOPE) + assert (posture.gate_posture, posture.gate_policy_suffix) == ( + "local_candidate", + "local_candidate", + ) + assert posture.pipeline == "uk-local-candidate" + assert posture.build_id_prefix == "uk-local-candidate-" + assert posture.staging_operation_id == "uk_rowwise_candidate" + assert posture.release_id == UK_DENSE_RELEASE_ID + assert posture.dataset_filename("2024_25") == "microcosm_uk_2024_25_local.h5" + assert ( + posture.gate_report_filename("2024_25") + == "microcosm_uk_2024_25_local.local_gates.json" + ) + assert posture.evidence_shape == "rowwise_candidate_manifest" + assert (posture.ladder_required, posture.local_rows, posture.holdout) == ( + True, + True, + True, + ) + # Exactly the six keys the dense pre-flight has pinned since #762. + assert posture.doctrine_bounds() == { + "target_loss_cap": UK_LOCAL_TARGET_LOSS_CAP, + "max_weight_ratio": UK_LOCAL_MAX_WEIGHT_RATIO, + "scale_rule": "default_target_loss_scales", + "target_weight_rule": "grain_equal", + "solve_epochs": 1500, + "clone_count": 15, + } + + +def test_uk_national_posture_is_the_seam_doctrine_with_no_overrides() -> None: + posture = UK_ROWWISE_NATIONAL_POSTURE + assert posture.role == "national" + assert posture.doctrine is UK_NATIONAL_SOLVE_DOCTRINE + assert posture.clone_count is None + assert (posture.epochs, posture.learning_rate, posture.seed) == (1500, 0.02, 0) + assert posture.target_weight_rule == "family_equal" + assert posture.allowed_target_weight_rules == ("family_equal", "uniform") + assert posture.expected_constituency_vintage is None + assert posture.gate_scope == tuple(UK_CALIBRATION_GATE_SCOPE) + assert (posture.gate_posture, posture.gate_policy_suffix) == ( + "calibration_seam", + "calibration_seam_scope", + ) + assert posture.pipeline == "uk-frs-calibration" + assert posture.build_id_prefix == "uk-frs-calibration-attempt-" + assert posture.staging_operation_id == "uk_national_calibration" + assert posture.release_id == UK_NATIONAL_RELEASE_ID + assert posture.dataset_filename("2024_25") == "microcosm_uk_2024_25.h5" + assert ( + posture.gate_report_filename("2024_25") + == "microcosm_uk_2024_25.terminal_gates.json" + ) + assert posture.evidence_shape == "calibration_seam_build_record" + assert (posture.ladder_required, posture.local_rows, posture.holdout) == ( + False, + False, + False, + ) + # The posture is the doctrine itself: a certified cut records no + # overrides, and "uniform" is a receipted override, never the default. + doctrine, receipt = uk_doctrine_with_overrides( + epochs=posture.epochs, + learning_rate=posture.learning_rate, + target_weight_rule=posture.target_weight_rule, + ) + assert doctrine == posture.doctrine + assert receipt == {} + _, receipt = uk_doctrine_with_overrides(target_weight_rule="uniform") + assert receipt == { + "target_weight_rule": {"default": "family_equal", "effective": "uniform"} + } + assert posture.doctrine_bounds() == { + "target_loss_cap": 10.0, + "max_weight_ratio": 10.0, + "scale_rule": "default_target_loss_scales", + "target_weight_rule": "family_equal", + "solve_epochs": 1500, + "clone_count": None, + "learning_rate": 0.02, + "seed": 0, + "mass_rule": "free", + "l0_lambda": 0.0, + } + + +def test_uk_rowwise_role_map_is_closed() -> None: + assert UK_ROWWISE_RELEASE_ROLES == ("national", "dense") + assert set(UK_ROWWISE_POSTURES) == set(UK_ROWWISE_RELEASE_ROLES) + assert uk_rowwise_posture("dense") is UK_ROWWISE_DENSE_POSTURE + assert uk_rowwise_posture("national") is UK_ROWWISE_NATIONAL_POSTURE + for role in ("local", "", None, 1): + with pytest.raises(ValueError, match="release role must be one of"): + uk_rowwise_posture(role) + + +@pytest.mark.parametrize( + ("changes", "needle"), + [ + ({"role": "local"}, "role must be one of"), + ({"clone_count": 0}, "clone_count"), + ({"clone_count": None}, "clones exactly when it requires the ladder"), + ({"epochs": 0}, "epochs must be positive"), + ({"learning_rate": 0.0}, "learning_rate"), + ({"seed": -1}, "seed must be a non-negative"), + ({"target_weight_rule": "family_equal"}, "not among its allowed rules"), + ( + {"allowed_target_weight_rules": ("uniform", "family_equal")}, + "not among its allowed rules", + ), + ({"expected_constituency_vintage": None}, "constituency vintage"), + ({"gate_scope": ()}, "at least one gate"), + ({"pipeline": " "}, "pipeline must be non-empty"), + ({"dataset_filename_template": "microcosm_uk_local.h5"}, "{vintage}"), + ], +) +def test_uk_posture_refuses_tampered_fields(changes, needle) -> None: + with pytest.raises(ValueError, match=needle): + dataclasses.replace(UK_ROWWISE_DENSE_POSTURE, **changes) + + +def test_uk_posture_filenames_require_an_frs_vintage() -> None: + for bad in ("2025", "2024-25", "202425", ""): + with pytest.raises(ValueError, match="YYYY_YY"): + UK_ROWWISE_DENSE_POSTURE.dataset_filename(bad) + + +def test_uk_posture_rule_must_be_its_doctrines() -> None: + with pytest.raises(ValueError, match="must be its doctrine's"): + UKRowwisePosture( + **{ + **dataclasses.asdict(UK_ROWWISE_DENSE_POSTURE), + "doctrine": UK_LOCAL_SOLVE_DOCTRINE, + "target_weight_rule": "uniform", + } + ) diff --git a/packages/microcosm-data/src/microcosm/data/contract.py b/packages/microcosm-data/src/microcosm/data/contract.py index a56eafbbd..f086ea672 100644 --- a/packages/microcosm-data/src/microcosm/data/contract.py +++ b/packages/microcosm-data/src/microcosm/data/contract.py @@ -614,6 +614,10 @@ def compatibility_claim_declarer_error(declared_by: object) -> str | None: _UK_DENSE_CUT_TAG_RE = re.compile( re.escape(_UK_DENSE_RELEASE_ID) + r"-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{8}" ) +#: The dense line's published microdata filename; mirrors +#: ``tools/assemble_uk_dense_release_dir.py::_DATASET_FILENAME`` (lockstep test +#: in the build shard's contract-pin suite). +_UK_DENSE_DATASET_FILENAME = "microcosm_uk_2024_25_dense.h5" _UK_DENSE_GATE_REPORT_FILE = "uk_local_gates.json" _UK_DENSE_SCORE_RECEIPT_FILE = "score_vs_incumbent.json" _UK_DENSE_SOURCE_COVERAGE_FILE = "uk_source_coverage.json" @@ -6200,7 +6204,7 @@ def _check_uk_dense_surface_files( ) return try: - dataset = _artifact_by_path(release_manifest, "microcosm_uk_2025_dense.h5") + dataset = _artifact_by_path(release_manifest, _UK_DENSE_DATASET_FILENAME) expected = { "candidate_dataset_sha256": dataset["sha256"], "candidate_manifest_sha256": hashes["rowwise_candidate_manifest.json"], diff --git a/packages/microcosm-data/tests/test_uk_dense_release_contract.py b/packages/microcosm-data/tests/test_uk_dense_release_contract.py index 939f75c04..1224f0416 100644 --- a/packages/microcosm-data/tests/test_uk_dense_release_contract.py +++ b/packages/microcosm-data/tests/test_uk_dense_release_contract.py @@ -243,7 +243,7 @@ def write_dense_bundle( for name, payload in payloads.items(): (release_dir / name).write_bytes(payload) h5_bytes = b"dense-h5-stand-in" - (root / "microcosm_uk_2025_dense.h5").write_bytes(h5_bytes) + (root / "microcosm_uk_2024_25_dense.h5").write_bytes(h5_bytes) def artifact(kind, path, sha): return { @@ -263,8 +263,8 @@ def artifact(kind, path, sha): "namespace": namespace, "build": {"build_id": release_id}, "artifacts": { - "microcosm_uk_2025_dense": artifact( - "microdata", "microcosm_uk_2025_dense.h5", _sha(h5_bytes) + "microcosm_uk_2024_25_dense": artifact( + "microdata", "microcosm_uk_2024_25_dense.h5", _sha(h5_bytes) ), **{ Path(name).stem: artifact("diagnostics", name, _sha(payload)) @@ -278,7 +278,7 @@ def artifact(kind, path, sha): (release_dir / "release_manifest.json").write_text(manifest_text) ledger = {name: _sha(payload) for name, payload in payloads.items()} ledger["release_manifest.json"] = _sha(manifest_text.encode()) - ledger["microcosm_uk_2025_dense.h5"] = _sha(h5_bytes) + ledger["microcosm_uk_2024_25_dense.h5"] = _sha(h5_bytes) (release_dir / "sha256sums.txt").write_text( "".join(f"{digest} {name}\n" for name, digest in sorted(ledger.items())) ) diff --git a/tools/assemble_uk_dense_release_dir.py b/tools/assemble_uk_dense_release_dir.py index 8c73c4465..8c2696888 100644 --- a/tools/assemble_uk_dense_release_dir.py +++ b/tools/assemble_uk_dense_release_dir.py @@ -43,7 +43,11 @@ _REPO_ID = "policyengine/populace-uk-private" _NAMESPACE = "uk_dense" -_DATASET_KEY = "microcosm_uk_2025_dense" +# The published dense artifact's key and filename (microcosm#823: every +# 2024-25 line carries the FRS release vintage in its name). The contract +# mirrors the filename as ``_UK_DENSE_DATASET_FILENAME``; a lockstep test +# pins the two. +_DATASET_KEY = "microcosm_uk_2024_25_dense" _DATASET_FILENAME = f"{_DATASET_KEY}.h5" _ATTEMPT_SUFFIX = re.compile(r"[0-9]{8}T[0-9]{6}Z-[0-9a-f]{8}") _RUNTIME_PACKAGES = ("policyengine-core", "policyengine-uk", "microcosm-data") @@ -195,6 +199,12 @@ def _assemble(args: argparse.Namespace) -> dict[str, object]: candidate_dir / "rowwise_candidate_manifest.json", label="rowwise_candidate_manifest.json", ) + if manifest.get("release_role") != "dense": + raise SystemExit( + f"error: manifest.release_role is {manifest.get('release_role')!r}, " + "not 'dense': this assembler covers the dense line only (a " + "candidate built before the release role existed is rebuilt)" + ) outputs = _mapping(manifest.get("outputs"), "manifest.outputs") identity = _mapping(manifest.get("identity"), "manifest.identity") # The national assembler's rule: a release carries the staging telemetry diff --git a/tools/build_uk_rowwise_candidate.py b/tools/build_uk_rowwise_candidate.py index 114b8b3ee..88b719de6 100644 --- a/tools/build_uk_rowwise_candidate.py +++ b/tools/build_uk_rowwise_candidate.py @@ -8,6 +8,13 @@ The pinned Ledger arguments are mandatory; ``--households-only`` binds only the Chronicle census-household constituency targets from the same registry. + +``--release-role`` declares which UK dataset line the run builds and is +required: ``dense`` is the joint K-clone surface described above under the +local doctrine; ``national`` builds the certified national line without +cloning, on national targets only, under the calibration-seam doctrine +(microcosm#823). The role fixes every solve default and refuses the other +role's flags, so the declared role is checked against the parameters. """ from __future__ import annotations @@ -79,11 +86,6 @@ from microcosm.build.target_materialization import resolve_target_measures from microcosm.build.uk_runtime import ( UK_GATE_REGISTRY, - UK_LOCAL_CLONE_COUNT, - UK_LOCAL_MAX_WEIGHT_RATIO, - UK_LOCAL_SOLVE_DOCTRINE, - UK_LOCAL_SOLVE_EPOCHS, - UK_LOCAL_TARGET_LOSS_CAP, CalibrationFrameAdapter, UKLadderRowwiseDatasetResult, UkOaLadder, @@ -144,6 +146,12 @@ UK_SAMPLE_SEED_DEFAULT, sample_uk_spine_frame, ) +from microcosm.build.uk_runtime.rowwise_posture import ( + UK_ROWWISE_DENSE_POSTURE, + UK_ROWWISE_RELEASE_ROLES, + UKRowwisePosture, + uk_rowwise_posture, +) from microcosm.build.uk_runtime.staging import ( UK_STAGED_DATASET_PREFIX, UK_STAGED_DATASET_REPOSITORY, @@ -154,16 +162,15 @@ BOUND_TARGET_FAMILIES = ("census_households/constituency",) BOUND_NATIONAL_TARGETS: tuple[str, ...] = () -CANDIDATE_FILENAME_TEMPLATE = "microcosm_uk_{calibration_year}_local.h5" -LOCAL_GATE_REPORT_FILENAME_TEMPLATE = ( - "microcosm_uk_{calibration_year}_local.local_gates.json" -) MANIFEST_FILENAME = "rowwise_candidate_manifest.json" SOLVE_DIAGNOSTICS_FILENAME = "solve_diagnostics.csv" CALIBRATION_DIAGNOSTICS_FILENAME = "calibration_diagnostics.json" AREA_SUPPORT_FILENAME = "area_support_summary.csv" PAST_CAP_FILENAME = "past_cap_census.json" LOCAL_REGISTRY_FILENAME = "local_target_registry.json" +#: National-role outputs (the calibration seam's evidence shape). +BUILD_RECORD_FILENAME = "build_record.json" +NATIONAL_REGISTRY_FILENAME = "national_target_registry.json" DENSE_REFERENCE_DIAGNOSTICS_FILENAME = "dense_reference_diagnostics.csv" DATASET_SIZE_SELECTION_FILENAME = "dataset_size_selection.csv" @@ -174,9 +181,17 @@ _TARGET_RECORDS: int | None = None _L0_LAMBDA = 0.0 _BUDGET_ITERS = 10 -_UK_CANDIDATE_PIPELINE = "uk-local-candidate" -_LOCAL_GATE_POLICY_SUFFIX = "local_candidate" -_STAGING_OPERATION_ID = "uk_rowwise_candidate" +# The dense role's Logbook pipeline and gate-policy suffix, kept as module +# names for the Logbook helpers and the contract-pin tests; the posture +# record (``rowwise_posture.py``) is the source of truth for both roles. +_UK_CANDIDATE_PIPELINE = UK_ROWWISE_DENSE_POSTURE.pipeline +_LOCAL_GATE_POLICY_SUFFIX = UK_ROWWISE_DENSE_POSTURE.gate_policy_suffix +# Best-effort telemetry upload cadence. The Hub allows about 128 commits per +# hour per repository and one cycle is up to eight single-file commits, so +# the shared 30-second default exhausts the budget on a multi-hour solve +# and loses uploads (the v20 national run did); five minutes keeps a +# 1,500-epoch run well inside it. +_STAGING_UPLOAD_INTERVAL_SECONDS = 300.0 # Staging telemetry keeps one row per forwarded epoch in # calibration_progress.json and one event in events.ndjson, both under the # contract's 5 MiB remote cap. A size run at 2,000 epochs solves the dense @@ -219,13 +234,24 @@ def __init__( def _new_candidate_build_id( *, seed: int, timestamp: datetime, rung: str = "f100" ) -> str: + """The dense role's attempt id; the national role mints the seam's.""" + instant = timestamp.astimezone(UTC) return ( - f"uk-local-candidate-{rung}-s{seed}-" + f"{UK_ROWWISE_DENSE_POSTURE.build_id_prefix}{rung}-s{seed}-" f"{instant.strftime('%Y%m%dT%H%M%SZ')}-{uuid.uuid4().hex[:8]}" ) +def _posture_of(args: argparse.Namespace) -> UKRowwisePosture: + """The release-role posture bound to parsed arguments.""" + + posture = getattr(args, "_posture", None) + if not isinstance(posture, UKRowwisePosture): + raise RuntimeError("arguments carry no release-role posture; parse them first.") + return posture + + def _candidate_clone_counts_argument(value: str) -> tuple[int, ...]: parts = value.split(",") if not value.strip() or any(not part.strip() for part in parts): @@ -578,7 +604,9 @@ def _size_checkpoint_identity( checkpointed dense solve and search were made with. The draw threshold is deliberately absent: re-drawing at another threshold is the point. """ + posture = _posture_of(args) return { + "release_role": posture.role, "dataset_pin": dict(pins["dataset"]), "ladder_pin": dict(pins["ladder"]), "ledger_facts_sha256": args.ledger_facts_sha256, @@ -587,7 +615,7 @@ def _size_checkpoint_identity( "selection_seed": int( args.seed if args.selection_seed is None else args.selection_seed ), - "n_clones": int(args.n_clones), + "n_clones": None if args.n_clones is None else int(args.n_clones), "dataset_households": args.dataset_households, "epochs": int(args.epochs), "learning_rate": float(args.learning_rate), @@ -604,7 +632,7 @@ def _size_checkpoint_identity( # The solve doctrine the dense solve and the search run under: a # resume after a doctrine change must refuse, not run under the old # bound while the manifest declares the new one. - "doctrine": _doctrine_bounds(), + "doctrine": _doctrine_bounds(posture), } @@ -687,6 +715,18 @@ def _record_candidate_error( def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--release-role", + choices=UK_ROWWISE_RELEASE_ROLES, + required=True, + help=( + "Which UK dataset line this run builds: 'national' (no cloning, " + "national targets only, the calibration-seam doctrine) or 'dense' " + "(the K-clone joint national + local surface under the local " + "doctrine). The role supplies every unset solve default and " + "refuses the other role's flags." + ), + ) parser.add_argument( "--input-h5", type=Path, @@ -701,8 +741,7 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser.add_argument( "--ladder", type=Path, - required=True, - help="Full-UK OA geography ladder NPZ.", + help="Full-UK OA geography ladder NPZ (required by the dense role).", ) parser.add_argument( "--ladder-sha256", @@ -716,8 +755,28 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser.add_argument("--register-json", type=Path) parser.add_argument( "--target-weight-rule", - choices=("uniform", "grain_equal"), - default=UK_LOCAL_SOLVE_DOCTRINE.target_weight_rule, + choices=("uniform", "grain_equal", "family_equal"), + help=( + "Target-weighting rule; defaults to the role's doctrine " + "(dense: grain_equal, national: family_equal). Any other admitted " + "rule is a receipted override." + ), + ) + parser.add_argument( + "--target-loss-cap", + type=float, + help=( + "National role only: receipted override of the seam doctrine's " + "per-target loss cap." + ), + ) + parser.add_argument( + "--allow-unpinned-feed", + action="store_true", + help=( + "National role only: allow a Ledger artifact whose feed commit is " + "not the committed Chronicle pin (development runs)." + ), ) parser.add_argument("--release-candidate", action="store_true") parser.add_argument( @@ -737,13 +796,17 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: type=int, help="Exact output household count after informed L0 and refit; pool clone K is unchanged. Candidate-only until size certification.", ) - parser.add_argument("--n-clones", type=int, default=UK_LOCAL_CLONE_COUNT) + parser.add_argument( + "--n-clones", + type=int, + help="Dense role only; defaults to the doctrine clone count.", + ) parser.add_argument( "--candidate-clone-counts", type=_candidate_clone_counts_argument, help="Dry-run only comma-separated candidate clone counts.", ) - parser.add_argument("--seed", type=int, default=42) + parser.add_argument("--seed", type=int, help="Defaults to the role's seed.") parser.add_argument( "--selection-seed", type=int, @@ -812,7 +875,7 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser.add_argument( "--sample-seed", type=int, - default=UK_SAMPLE_SEED_DEFAULT, + help=f"Dense role only; defaults to {UK_SAMPLE_SEED_DEFAULT}.", ) parser.add_argument( "--engine-blocks", @@ -826,12 +889,17 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: help="Survey year recorded for lineage (calibration uses the FRS release year).", ) parser.add_argument("--source-lineage-modulus", type=int) - parser.add_argument("--epochs", type=int, default=UK_LOCAL_SOLVE_EPOCHS) - parser.add_argument("--learning-rate", type=float, default=0.15) + parser.add_argument( + "--epochs", type=int, help="Defaults to the role's doctrine solve length." + ) + parser.add_argument( + "--learning-rate", + type=float, + help="Defaults to the role's learning rate (dense 0.15, national 0.02).", + ) parser.add_argument( "--expected-constituency-vintage", - default="2024_pcon", - help="Constituency vintage required from the ladder.", + help="Dense role only: constituency vintage required from the ladder.", ) parser.add_argument( "--dry-run", @@ -848,19 +916,71 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: "POPULACE_LOGBOOK_PREV_ROW_DIGEST is used, then genesis null." ), ) - add_staging_arguments(parser, repository=UK_STAGING_REPOSITORY) + add_staging_arguments( + parser, + repository=UK_STAGING_REPOSITORY, + default_upload_interval_seconds=_STAGING_UPLOAD_INTERVAL_SECONDS, + ) add_staged_dataset_arguments(parser, repository=UK_STAGED_DATASET_REPOSITORY) args = parser.parse_args(argv) validate_staging_arguments(parser, args) validate_staged_dataset_arguments(parser, args) + _resolve_role_arguments(args) return args +#: Solve arguments whose argparse default is ``None`` so an explicit value can +#: be told from the role's default: the other role's refusal table keys on +#: what was actually given. +_ROLE_DEFAULTED_ARGUMENTS = ( + "n_clones", + "seed", + "sample_seed", + "epochs", + "learning_rate", + "target_weight_rule", + "expected_constituency_vintage", +) + + +def _resolve_role_arguments(args: argparse.Namespace) -> UKRowwisePosture: + """Bind the declared role's posture and fill its defaults into unset arguments.""" + + posture = uk_rowwise_posture(args.release_role) + args._explicit_arguments = frozenset( + name for name in _ROLE_DEFAULTED_ARGUMENTS if getattr(args, name) is not None + ) + if args.n_clones is None: + args.n_clones = posture.clone_count + if args.seed is None: + args.seed = posture.seed + if args.sample_seed is None: + args.sample_seed = UK_SAMPLE_SEED_DEFAULT + if args.epochs is None: + args.epochs = posture.epochs + if args.learning_rate is None: + args.learning_rate = posture.learning_rate + if args.target_weight_rule is None: + args.target_weight_rule = posture.target_weight_rule + if args.expected_constituency_vintage is None: + args.expected_constituency_vintage = posture.expected_constituency_vintage + args._posture = posture + return posture + + def main(argv: list[str] | None = None) -> int: """Run the rowwise candidate build.""" args = _parse_args(argv) _validate_cli_args(args) + posture = _posture_of(args) + if posture.role == "national": + raise NotImplementedError( + "--release-role national is declared and validated here, but its " + "build flow (delegation to the calibration seam) lands in the next " + "increment of microcosm#823; run tools/calibrate_uk_national_dataset.py " + "until then." + ) if args.candidate_clone_counts is not None and not args.dry_run: raise ValueError("--candidate-clone-counts is valid only with --dry-run.") if _CONSERVE_MASS: @@ -879,7 +999,7 @@ def main(argv: list[str] | None = None) -> int: _preflight_staged_dataset(args) started_at = time.perf_counter() started_ts = datetime.now(UTC) - digest = preflight_digest(_UK_CANDIDATE_PIPELINE) + digest = preflight_digest(posture.pipeline) state = AttemptState( build_id=_new_candidate_build_id( seed=args.seed, @@ -967,8 +1087,10 @@ def _run_candidate( append_phase(state, "configured") append_phase(state, "inputs_pinned") national_frame, _national_provenance = load_uk_national_frame(input_h5) - calibration_year = int(load_uk_frs_release().calibration_year) + frs_release = load_uk_frs_release() + calibration_year = int(frs_release.calibration_year) args._calibration_year = calibration_year + args._frs_vintage = str(frs_release.vintage) if args.households_only: args._spine_provenance = {} else: @@ -1005,8 +1127,8 @@ def _run_candidate( ) output_paths = _output_paths( out_dir, - source_year=source_year, - calibration_year=calibration_year, + posture=_posture_of(args), + vintage=args._frs_vintage, ) _validate_output_paths( output_paths, @@ -1053,11 +1175,12 @@ def _run_candidate( "uprating: " + str(joint_inputs["census_household_uprating"].get("reason")) ) + posture = _posture_of(args) doctrine, doctrine_override = uk_local_doctrine_with_overrides( - UK_LOCAL_SOLVE_DOCTRINE, + posture.doctrine, ( {} - if args.target_weight_rule == UK_LOCAL_SOLVE_DOCTRINE.target_weight_rule + if args.target_weight_rule == posture.target_weight_rule else {"target_weight_rule": args.target_weight_rule} ), ) @@ -1656,11 +1779,12 @@ def _create_staging_telemetry( return None local_only = bool(args.staging_local_only) out_dir = args.out.expanduser().resolve() + posture = _posture_of(args) return StagingTelemetryV2( run_id=args.staging_run_id or state.build_id, country_code="GB", - operation_id=_STAGING_OPERATION_ID, - pipeline_id=_UK_CANDIDATE_PIPELINE, + operation_id=posture.staging_operation_id, + pipeline_id=posture.pipeline, pipeline_version=metadata.version("microcosm-build"), candidate_id=args.staging_candidate_id or state.build_id, local_dir=args.staging_dir or out_dir / "staging", @@ -1678,8 +1802,26 @@ def _stage( event_status: str = "started", **details: Any, ) -> None: - if telemetry is not None: + """Forward one stage event to the best-effort telemetry. + + A contract or content refusal of the event is reported and the event + dropped; the build must never abort on its own progress report. Each + event is judged on its own details, so a refused event does not silence + the ones that follow. + """ + + if telemetry is None: + return + try: telemetry.stage(stage_id, event_status=event_status, **details) + except StagingContractError as error: + print( + f"warning: staging telemetry refused the {stage_id!r} stage event " + f"({type(error).__name__}: {error}); the event is not staged, the " + "build continues.", + file=sys.stderr, + flush=True, + ) def _staging_epoch_every(args: argparse.Namespace) -> int: @@ -2446,6 +2588,7 @@ def _joint_dry_run_plan( return { "schema_version": 3, "build_kind": "uk_rowwise_calibrated_candidate_plan", + "release_role": _posture_of(args).role, "dry_run": True, "survey_year": source_year, "calibration_year": joint_inputs["calibration_year"], @@ -2912,6 +3055,7 @@ def _dry_run_plan( return { "schema_version": 3, "build_kind": "uk_rowwise_calibrated_candidate_plan", + "release_role": _posture_of(args).role, "dry_run": True, "candidate_scope": "adjudicated_partial", "bound_target_families": list(args._bound_families), @@ -3161,9 +3305,12 @@ def _manifest( } ] ) + posture = _posture_of(args) return { - "schema_version": 3, + "schema_version": 4, "build_kind": "uk_rowwise_calibrated_candidate", + "release_role": posture.role, + "release_id": posture.release_id, "candidate_scope": "adjudicated_partial", "created_at": datetime.now(UTC).isoformat(), "git_commit": _git_commit(), @@ -3257,7 +3404,7 @@ def _manifest( "reason": str(calibration_record.reason), }, "abs_delta": abs(new_total - old_total), - "declared_stretch_bound": float(UK_LOCAL_MAX_WEIGHT_RATIO), + "declared_stretch_bound": float(posture.doctrine.max_weight_ratio), "stretch_reference": "pool_design" if solve.size_receipt is None else "normalized_horvitz_thompson_w_over_q", @@ -3416,8 +3563,10 @@ def _manifest( def _parameters(args: argparse.Namespace, *, source_year: int) -> dict[str, Any]: + posture = _posture_of(args) return { - "n_clones": int(args.n_clones), + "release_role": posture.role, + "n_clones": None if args.n_clones is None else int(args.n_clones), "dataset_households": args.dataset_households, "seed": int(args.seed), "selection_seed": None @@ -3447,8 +3596,12 @@ def _parameters(args: argparse.Namespace, *, source_year: int) -> dict[str, Any] "skip_holdout": bool(args.skip_holdout), "epochs": int(args.epochs), "learning_rate": float(args.learning_rate), - "expected_constituency_vintage": str(args.expected_constituency_vintage), - "doctrine": _doctrine_bounds(), + "expected_constituency_vintage": ( + None + if args.expected_constituency_vintage is None + else str(args.expected_constituency_vintage) + ), + "doctrine": _doctrine_bounds(posture), "solve_options": { "conserve_mass": _CONSERVE_MASS, "target_records": _TARGET_RECORDS, @@ -3588,15 +3741,8 @@ def _local_output_registry( return TargetRegistry(specs, country="uk") -def _doctrine_bounds() -> dict[str, Any]: - return { - "target_loss_cap": float(UK_LOCAL_TARGET_LOSS_CAP), - "max_weight_ratio": float(UK_LOCAL_MAX_WEIGHT_RATIO), - "scale_rule": UK_LOCAL_SOLVE_DOCTRINE.scale_rule, - "target_weight_rule": UK_LOCAL_SOLVE_DOCTRINE.target_weight_rule, - "solve_epochs": int(UK_LOCAL_SOLVE_EPOCHS), - "clone_count": int(UK_LOCAL_CLONE_COUNT), - } +def _doctrine_bounds(posture: UKRowwisePosture) -> dict[str, Any]: + return posture.doctrine_bounds() def _gate_payload(gate: GateResult, *, phase: str) -> dict[str, Any]: @@ -3667,6 +3813,13 @@ def _validate_support_summary(support: pd.DataFrame) -> None: def _validate_cli_args(args: argparse.Namespace) -> None: + posture = _posture_of(args) + # The declared role is checked against the parameters first: the other + # role's flags are refused by name before any value is range-checked. + if posture.role == "national": + _refuse_dense_role_arguments(args, posture) + else: + _refuse_national_role_arguments(args, posture) if args.selection_seed is not None and args.dataset_households is None: raise ValueError("--selection-seed requires --dataset-households.") if not (0.0 < args.selection_pi_hi <= 1.0): @@ -3706,10 +3859,15 @@ def _validate_cli_args(args: argparse.Namespace) -> None: "--ledger-facts, --ledger-facts-sha256, and " "--ledger-manifest-sha256 are mandatory and must be supplied together." ) - if args.input_sha256 is None or args.ladder_sha256 is None: - raise ValueError( - "the joint registry path requires --input-sha256 and --ladder-sha256." - ) + if posture.ladder_required: + if args.ladder is None: + raise ValueError("--release-role dense requires --ladder.") + if args.input_sha256 is None or args.ladder_sha256 is None: + raise ValueError( + "the joint registry path requires --input-sha256 and --ladder-sha256." + ) + elif args.input_sha256 is None: + raise ValueError("--release-role national requires --input-sha256.") if args.release_candidate: required_release = { "--input-sha256": args.input_sha256, @@ -3727,12 +3885,12 @@ def _validate_cli_args(args: argparse.Namespace) -> None: + ", ".join(missing_release) ) refused = [] - if args.target_weight_rule != UK_LOCAL_SOLVE_DOCTRINE.target_weight_rule: + if args.target_weight_rule != posture.target_weight_rule: refused.append("--target-weight-rule") - if args.epochs != UK_LOCAL_SOLVE_EPOCHS: - refused.append(f"--epochs != doctrine {UK_LOCAL_SOLVE_EPOCHS}") - if args.n_clones != UK_LOCAL_CLONE_COUNT: - refused.append(f"--n-clones != doctrine {UK_LOCAL_CLONE_COUNT}") + if args.epochs != posture.epochs: + refused.append(f"--epochs != doctrine {posture.epochs}") + if args.n_clones != posture.clone_count: + refused.append(f"--n-clones != doctrine {posture.clone_count}") if args.measure_exclusions is not None: refused.append("--measure-exclusions") if args.skip_holdout: @@ -3746,7 +3904,7 @@ def _validate_cli_args(args: argparse.Namespace) -> None: "--release-candidate refuses non-release settings: " + ", ".join(refused) ) - if args.n_clones <= 0: + if args.n_clones is not None and args.n_clones <= 0: raise ValueError("--n-clones must be positive.") if args.seed < 0: raise ValueError("--seed must be non-negative.") @@ -3767,19 +3925,119 @@ def _validate_cli_args(args: argparse.Namespace) -> None: raise ValueError("--epochs must be positive.") if not np.isfinite(args.learning_rate) or args.learning_rate <= 0: raise ValueError("--learning-rate must be positive and finite.") - if not str(args.expected_constituency_vintage).strip(): + if args.target_loss_cap is not None and ( + not np.isfinite(args.target_loss_cap) or args.target_loss_cap <= 0 + ): + raise ValueError("--target-loss-cap must be positive and finite.") + if ( + args.expected_constituency_vintage is not None + and not str(args.expected_constituency_vintage).strip() + ): raise ValueError("--expected-constituency-vintage must be non-empty.") +def _refuse_dense_role_arguments( + args: argparse.Namespace, posture: UKRowwisePosture +) -> None: + """The national role's refusal table: nothing of the clone surface may be given. + + ``--release-candidate`` is refused outright with the seam's own reason: the + calibration-seam battery covers six of the declared entries and must never + sign a shippability claim; a national cut's verdict comes only from the + release-cut certification producer (``tools/certify_uk_release_cut.py``). + """ + + if args.release_candidate: + raise ValueError( + "--release-candidate is refused on the national role: the " + "calibration seam's scoped battery cannot sign shippability; run " + "the release-cut certification producer " + "(tools/certify_uk_release_cut.py) on the finished build instead." + ) + explicit = args._explicit_arguments + refused: list[str] = [] + if args.ladder is not None: + refused.append("--ladder") + if args.ladder_sha256 is not None: + refused.append("--ladder-sha256") + if "expected_constituency_vintage" in explicit: + refused.append("--expected-constituency-vintage") + if args.source_year is not None: + refused.append("--source-year") + if args.source_lineage_modulus is not None: + refused.append("--source-lineage-modulus") + if "n_clones" in explicit: + refused.append("--n-clones") + if args.candidate_clone_counts is not None: + refused.append("--candidate-clone-counts") + if args.engine_blocks != 1: + refused.append("--engine-blocks") + if args.households_only: + refused.append("--households-only") + if args.skip_holdout: + refused.append("--skip-holdout") + if args.dataset_households is not None: + refused.append("--dataset-households") + if args.selection_seed is not None: + refused.append("--selection-seed") + if args.selection_pi_hi != 1.0: + refused.append("--selection-pi-hi") + if args.baseline_pi_floor != 0.0: + refused.append("--baseline-pi-floor") + if args.no_size_checkpoint: + refused.append("--no-size-checkpoint") + if args.resume_size_checkpoint is not None: + refused.append("--resume-size-checkpoint") + if args.sample_fraction != 1.0: + refused.append("--sample-fraction") + if "sample_seed" in explicit: + refused.append("--sample-seed") + if args.target_weight_rule not in posture.allowed_target_weight_rules: + refused.append(f"--target-weight-rule {args.target_weight_rule}") + if refused: + raise ValueError( + "--release-role national refuses the dense role's arguments: " + + ", ".join(refused) + ) + + +def _refuse_national_role_arguments( + args: argparse.Namespace, posture: UKRowwisePosture +) -> None: + """The dense role's refusal table: the seam's knobs are not its own.""" + + refused: list[str] = [] + if args.target_loss_cap is not None: + refused.append("--target-loss-cap") + if args.allow_unpinned_feed: + refused.append("--allow-unpinned-feed") + if args.target_weight_rule not in posture.allowed_target_weight_rules: + refused.append(f"--target-weight-rule {args.target_weight_rule}") + if refused: + raise ValueError( + "--release-role dense refuses the national role's arguments: " + + ", ".join(refused) + ) + + def _output_paths( out_dir: Path, *, - source_year: int, - calibration_year: int, + posture: UKRowwisePosture, + vintage: str, ) -> dict[str, Path]: - dataset = out_dir / CANDIDATE_FILENAME_TEMPLATE.format( - calibration_year=calibration_year - ) + """The role's output paths for one FRS release vintage (``2024_25``).""" + + dataset = out_dir / posture.dataset_filename(vintage) + if posture.role == "national": + return { + "dataset": dataset, + "manifest": out_dir / MANIFEST_FILENAME, + "calibration_diagnostics": out_dir / CALIBRATION_DIAGNOSTICS_FILENAME, + "build_record": out_dir / BUILD_RECORD_FILENAME, + "terminal_gates": out_dir / posture.gate_report_filename(vintage), + "national_registry": out_dir / NATIONAL_REGISTRY_FILENAME, + } return { "dataset": dataset, "manifest": out_dir / MANIFEST_FILENAME, @@ -3787,8 +4045,7 @@ def _output_paths( "support": out_dir / AREA_SUPPORT_FILENAME, "past_cap": out_dir / PAST_CAP_FILENAME, "calibration_diagnostics": out_dir / CALIBRATION_DIAGNOSTICS_FILENAME, - "local_gates": out_dir - / LOCAL_GATE_REPORT_FILENAME_TEMPLATE.format(calibration_year=calibration_year), + "local_gates": out_dir / posture.gate_report_filename(vintage), "local_registry": out_dir / LOCAL_REGISTRY_FILENAME, "dense_reference": out_dir / DENSE_REFERENCE_DIAGNOSTICS_FILENAME, "selection": out_dir / DATASET_SIZE_SELECTION_FILENAME, diff --git a/tools/preflight_uk_local_release_candidate.py b/tools/preflight_uk_local_release_candidate.py index fcf03bbd9..fd7269fc8 100644 --- a/tools/preflight_uk_local_release_candidate.py +++ b/tools/preflight_uk_local_release_candidate.py @@ -119,6 +119,14 @@ def check_candidate_dir(candidate_dir: Path, *, today: date | None = None) -> li if not manifest_path.exists(): return [f"manifest missing: {manifest_path}"] manifest = json.loads(manifest_path.read_text()) + # The manifest declares its release role (microcosm#823); this pre-flight + # is the dense line's and refuses every other role, including a + # pre-role candidate, which is rebuilt rather than grandfathered. + if manifest.get("release_role") != "dense": + return [ + f"manifest.release_role is {manifest.get('release_role')!r}, not " + "'dense': this pre-flight covers the dense line only." + ] parameters = manifest.get("parameters", {}) if parameters.get("release_candidate") is not True: failures.append("parameters.release_candidate is not true.") From ec3b224a515441438a75c1df0f4b2ca2459bdef9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sun, 20 Sep 2026 19:28:01 +0100 Subject: [PATCH 02/11] Re-pin the childcare gate test to the family_equal doctrine rule (#823) The 2026-09-20 ruling moved UK_NATIONAL_TARGET_WEIGHT_RULE to family_equal; this test still pinned the previous default. Found by the local spine-uk and uk:build lanes (2,638 and 2,294 other tests green). Co-Authored-By: Claude Fable 5.1 --- packages/microcosm-build/tests/test_uk_childcare_targets.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/packages/microcosm-build/tests/test_uk_childcare_targets.py b/packages/microcosm-build/tests/test_uk_childcare_targets.py index 332e86e29..831427372 100644 --- a/packages/microcosm-build/tests/test_uk_childcare_targets.py +++ b/packages/microcosm-build/tests/test_uk_childcare_targets.py @@ -63,4 +63,5 @@ def test_childcare_activation_preserves_existing_calibration_gates() -> None: assert by_id["uk_calibration_reference_coverage"]["criticality"] == ( "release_blocking" ) - assert UK_NATIONAL_SOLVE_DOCTRINE.target_weight_rule == "uniform" + # Ruling 2026-09-20 (microcosm#823): the doctrine rule is family_equal. + assert UK_NATIONAL_SOLVE_DOCTRINE.target_weight_rule == "family_equal" From ddf9f626dd3ca5ac3a1229fbc86ffa5f8810b257 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:45:07 +0100 Subject: [PATCH 03/11] Build the UK national line through the rowwise driver's national role (#823) `tools/build_uk_rowwise_candidate.py --release-role national` now builds the certified national line by delegating to the calibration seam library (uk_runtime.calibration_run.run_uk_calibration): the measure resolver is built from the input file (direct_h5), the solve runs under the seam doctrine (1,500 epochs, family_equal, learning rate 0.02, seed 0) with explicit --epochs/--learning-rate/--target-weight-rule/--target-loss-cap as receipted overrides, the six calibration-seam gates run in-process, and the seam writes the H5, the diagnostics, the signed gate report, the build record and the uk/frs Logbook row exactly as the seam command did, so certification and assembly are untouched. Around it the driver adds what the dense role has: the pinned input, the frozen national_target_registry.json (the scorer's register), the rowwise manifest (build_kind uk_national_calibrated_candidate, release_role, release_id, the role's doctrine block, the seam's solve and weights blocks, fit by family, gate statuses, releasable false), staging telemetry under the attempt id (operation uk_national_calibration, pipeline uk-frs-calibration) with the two reviewed artifacts, and the staged bundle under staged// in one commit. --dry-run compiles the register and prints the plan. Seam library: run_uk_calibration accepts a caller-minted attempt id; new_uk_calibration_attempt_id, UK_CALIBRATION_PIPELINE and UK_CALIBRATION_ATTEMPT_ID_PREFIX are public; the committed Chronicle feed-pin check moves to uk_runtime.chronicle_feed (require_committed_uk_chronicle_feed_pin, UKChronicleFeedPinError) and the seam CLI wraps it into its previous SystemExit until it retires. Staging: the last-mile gap the v20 national run met (three consecutive rate-limit failures paused remote writes, so complete()'s forced upload was skipped and the remote progress.json stayed "running") is closed: completion and failure re-open the best-effort session for a bounded, backed-off terminal flush (0/15/60 s), warning once if all three miss. Co-Authored-By: Claude Fable 5.1 --- .../uk-rowwise-national-role-823.added.md | 1 + .../uk-staging-terminal-flush-823.fixed.md | 1 + .../src/microcosm/build/staging_storage.py | 18 +- .../src/microcosm/build/staging_v2.py | 38 +- .../build/uk_runtime/calibration_run.py | 38 +- .../build/uk_runtime/chronicle_feed.py | 36 ++ .../build/uk_runtime/rowwise_posture.py | 6 +- .../microcosm-build/tests/test_staging_v2.py | 80 +++ .../tests/test_uk_calibration_run.py | 51 ++ .../tests/test_uk_chronicle_feed.py | 38 ++ .../tests/test_uk_rowwise_candidate.py | 8 - .../tests/test_uk_rowwise_national_role.py | 509 +++++++++++++++++ tools/build_uk_rowwise_candidate.py | 539 +++++++++++++++++- tools/calibrate_uk_national_dataset.py | 24 +- 14 files changed, 1338 insertions(+), 49 deletions(-) create mode 100644 changelog.d/uk-rowwise-national-role-823.added.md create mode 100644 changelog.d/uk-staging-terminal-flush-823.fixed.md create mode 100644 packages/microcosm-build/tests/test_uk_rowwise_national_role.py diff --git a/changelog.d/uk-rowwise-national-role-823.added.md b/changelog.d/uk-rowwise-national-role-823.added.md new file mode 100644 index 000000000..77e3a9740 --- /dev/null +++ b/changelog.d/uk-rowwise-national-role-823.added.md @@ -0,0 +1 @@ +`tools/build_uk_rowwise_candidate.py --release-role national` now builds the certified national line through the calibration seam library: no cloning, national targets only, the seam doctrine (1,500 epochs, `family_equal`, learning rate 0.02, seed 0) with explicit flags as receipted overrides, the six calibration-seam gates, the seam-shaped `build_record.json` that `tools/certify_uk_release_cut.py` certifies, a frozen `national_target_registry.json` for the scorer, and the same staging telemetry (operation `uk_national_calibration`, run id = the attempt id) and staged bundle as the dense role. `run_uk_calibration` accepts a caller-minted attempt id and the committed Chronicle feed-pin check moved to `uk_runtime.chronicle_feed`. diff --git a/changelog.d/uk-staging-terminal-flush-823.fixed.md b/changelog.d/uk-staging-terminal-flush-823.fixed.md new file mode 100644 index 000000000..03fb3b83b --- /dev/null +++ b/changelog.d/uk-staging-terminal-flush-823.fixed.md @@ -0,0 +1 @@ +Staging telemetry now delivers its terminal state even when mid-run upload failures had paused remote writes: completion and failure re-open the best-effort session for a bounded, backed-off final flush, so a run's remote `progress.json` no longer stays `running` after a rate-limited Hub (the v20 national run's last-mile gap). diff --git a/packages/microcosm-build/src/microcosm/build/staging_storage.py b/packages/microcosm-build/src/microcosm/build/staging_storage.py index 7f9ffc4ed..59ed736b9 100644 --- a/packages/microcosm-build/src/microcosm/build/staging_storage.py +++ b/packages/microcosm-build/src/microcosm/build/staging_storage.py @@ -166,9 +166,7 @@ def credential_can_write(self) -> bool | None: entity = entity if isinstance(entity, Mapping) else {} name = str(entity.get("name") or "") kind = str(entity.get("type") or "") - covers = name == self.repo_id or ( - kind in {"user", "org"} and name == owner - ) + covers = name == self.repo_id or (kind in {"user", "org"} and name == owner) if covers and "repo.write" in (scope.get("permissions") or []): return True return False @@ -289,9 +287,7 @@ def upload( self.storage.upload(local_path, path_in_repo) except Exception as error: self.consecutive_failures += 1 - became_disabled = ( - self.consecutive_failures >= self.max_consecutive_failures - ) + became_disabled = self.consecutive_failures >= self.max_consecutive_failures if became_disabled: self.enabled = False return UploadResult( @@ -310,3 +306,13 @@ def upload( disabled=False, became_disabled=False, ) + + def reopen(self) -> None: + """Re-enable writes after a pause, for a terminal flush of the run state. + + The attempt and success counters are kept: the delivery summary + still reports every write the run made. + """ + + self.consecutive_failures = 0 + self.enabled = True diff --git a/packages/microcosm-build/src/microcosm/build/staging_v2.py b/packages/microcosm-build/src/microcosm/build/staging_v2.py index 87850b2e1..1a9ec3b34 100644 --- a/packages/microcosm-build/src/microcosm/build/staging_v2.py +++ b/packages/microcosm-build/src/microcosm/build/staging_v2.py @@ -727,6 +727,7 @@ def __init__( clock: Callable[[], str] = _utc_now, monotonic: Callable[[], float] = time.monotonic, content_policy: StagingContentPolicy | None = None, + sleep: Callable[[float], None] = time.sleep, ) -> None: self.run_id = _safe_identifier(run_id, label="run_id") self.candidate_id = _safe_identifier(candidate_id, label="candidate_id") @@ -762,6 +763,7 @@ def __init__( self.upload_interval_seconds = max(0.0, float(upload_interval_seconds)) self._clock = clock self._monotonic = monotonic + self._sleep = sleep self._content_policy = content_policy or StagingContentPolicy() self._transport = ( HuggingFaceDatasetStorage(self.repo_id, api=api) @@ -968,7 +970,7 @@ def fail( timestamp=self.updated_at, ) self._persist_bundle() - self._maybe_upload(force=True) + self._terminal_upload() def complete(self, *, message: str = "Staging run completed.") -> None: self._require_running("complete the run") @@ -985,7 +987,7 @@ def complete(self, *, message: str = "Staging run completed.") -> None: timestamp=self.updated_at, ) self._persist_bundle() - self._maybe_upload(force=True) + self._terminal_upload() def verify_remote(self) -> None: if self.status == "running": @@ -1210,6 +1212,38 @@ def _reconcile_remote_delivery_metadata(self) -> None: ) self._persist_bundle() + #: Delays before each terminal upload attempt. Mid-run, three consecutive + #: failures pause remote writes for the rest of the run so a rate-limited + #: Hub is not hammered; the terminal state is different: a run whose remote + #: copy stays "running" forever is the last-mile gap the v20 national run + #: met (its final state was delivered by hand), so completion and failure + #: re-open the session for a bounded, backed-off final flush. + _TERMINAL_UPLOAD_DELAYS: tuple[float, ...] = (0.0, 15.0, 60.0) + + def _terminal_upload(self) -> None: + """Deliver the terminal state even after failures paused remote writes.""" + + if self._transport is None or self._upload_session is None: + return + for delay in self._TERMINAL_UPLOAD_DELAYS: + if delay > 0.0: + self._sleep(delay) + self._upload_session.reopen() + self._remote_disabled = False + self._consecutive_upload_failures = 0 + self._maybe_upload(force=True) + if ( + not self._remote_disabled + and self._delivery["last_error_code"] != "UPLOAD_FAILED" + ): + return + print( + "warning: the staging run's terminal state did not reach " + f"{self.repo_id} after {len(self._TERMINAL_UPLOAD_DELAYS)} attempts; " + "the local bundle is complete.", + file=sys.stderr, + ) + def _maybe_upload(self, *, force: bool = False) -> None: if self._transport is None or self._remote_disabled: return diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py index 123013c87..6698adb16 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/calibration_run.py @@ -67,7 +67,12 @@ # The FRS line's spine, staging, imputation and calibration stages share one # hash chain (logbook/README.md): the dataset token names the base data, not # the build mechanism, so calibration derives the ratified `uk/frs` scope. -_PIPELINE = "uk-frs-calibration" +#: The national calibration seam's Logbook pipeline and attempt-id prefix; +#: the rowwise driver's national release role mints ids with them +#: (microcosm#823) so a run built either way reads as one pipeline. +UK_CALIBRATION_PIPELINE = "uk-frs-calibration" +UK_CALIBRATION_ATTEMPT_ID_PREFIX = "uk-frs-calibration-attempt-" +_PIPELINE = UK_CALIBRATION_PIPELINE @dataclass(frozen=True) @@ -277,11 +282,29 @@ def run_uk_calibration( staging_delivery: Mapping[str, object] | None = None, staging_finalizer: Callable[[], None] | None = None, staging_delivery_provider: Callable[[], Mapping[str, object]] | None = None, + build_id: str | None = None, ) -> UKCalibrationRunResult: - """Run the UK national calibration seam and write its sidecars.""" + """Run the UK national calibration seam and write its sidecars. + + ``build_id`` lets a caller that opened staging telemetry before the run + (the rowwise driver's national role) mint the attempt id first, so the + telemetry run id and the Logbook row agree; it must carry the seam's + attempt prefix. Omitted, the seam mints one. + """ started_at = time.perf_counter() started_ts = datetime.now(UTC) + if build_id is None: + build_id = new_uk_calibration_attempt_id(timestamp=started_ts) + elif not ( + isinstance(build_id, str) + and build_id.startswith(UK_CALIBRATION_ATTEMPT_ID_PREFIX) + and len(build_id) > len(UK_CALIBRATION_ATTEMPT_ID_PREFIX) + ): + raise ValueError( + "a caller-minted calibration attempt id must start with " + f"{UK_CALIBRATION_ATTEMPT_ID_PREFIX!r}, got {build_id!r}." + ) # Pure-argument validation precedes every environment probe: an # incoherent register/receipt/band-edge triple must refuse identically # whether or not a git checkout or Logbook chain is reachable. @@ -318,7 +341,7 @@ def run_uk_calibration( state = AttemptState( # Attempts are distinct rows even when they re-run one release: both # the local chain and the store refuse a repeated build id. - build_id=_new_calibration_attempt_id(timestamp=started_ts), + build_id=build_id, identity_digest=hashlib.sha256(canonical_json_bytes(run_config)).hexdigest(), input_pins_digest=role_pins_digest(source_pins), phases_reached=["attempt_started"], @@ -370,14 +393,19 @@ def run_uk_calibration( raise -def _new_calibration_attempt_id(*, timestamp: datetime) -> str: +def new_uk_calibration_attempt_id(*, timestamp: datetime) -> str: + """Mint a calibration attempt id: the seam prefix, the instant, eight hex.""" + instant = timestamp.astimezone(UTC) return ( - "uk-frs-calibration-attempt-" + f"{UK_CALIBRATION_ATTEMPT_ID_PREFIX}" f"{instant.strftime('%Y%m%dT%H%M%SZ')}-{uuid.uuid4().hex[:8]}" ) +_new_calibration_attempt_id = new_uk_calibration_attempt_id + + def _validate_band_edge_registry( *, register_registry: TargetRegistry, diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/chronicle_feed.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/chronicle_feed.py index 1c8e9d9c5..a66791aa9 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/chronicle_feed.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/chronicle_feed.py @@ -47,6 +47,42 @@ def to_dict(self) -> dict[str, object]: return result +class UKChronicleFeedPinError(ValueError): + """A Ledger artifact is not the committed UK national Chronicle feed.""" + + +def require_committed_uk_chronicle_feed_pin( + facts_sha256: str, + *, + manifest_sha256: str | None, + allow_unpinned_feed: bool, + pin: UKChronicleFeed | None = None, +) -> UKChronicleFeed: + """Refuse a Ledger artifact whose digests are not the committed feed pin. + + Returns the pin the artifact was checked against. ``allow_unpinned_feed`` + admits a mismatch for an explicitly reviewed diagnostic run; the caller + records the override in the run provenance. + """ + + pin = pin or load_uk_chronicle_feed() + mismatches = [] + for label, loaded, committed in ( + ("facts", facts_sha256, pin.facts_sha256), + ("manifest", manifest_sha256, pin.manifest_sha256), + ): + if loaded != committed: + mismatches.append(f"{label}: loaded {loaded}, committed {committed}") + if mismatches and not allow_unpinned_feed: + raise UKChronicleFeedPinError( + "Chronicle artifact differs from the committed UK national feed pin: " + + "; ".join(mismatches) + + "; pass --allow-unpinned-feed only for an explicitly reviewed " + "diagnostic run" + ) + return pin + + def _feed_path(): return files("microcosm.build.uk").joinpath("chronicle_feed.json") diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py index 60a5002df..59c6b0216 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/rowwise_posture.py @@ -29,7 +29,9 @@ import numpy as np from microcosm.build.uk_runtime.calibration_run import ( + UK_CALIBRATION_ATTEMPT_ID_PREFIX, UK_CALIBRATION_GATE_SCOPE, + UK_CALIBRATION_PIPELINE, UK_LOCAL_GATE_SCOPE, ) from microcosm.build.uk_runtime.local_doctrine import ( @@ -242,8 +244,8 @@ def _vintage(vintage: str) -> str: gate_scope=tuple(UK_CALIBRATION_GATE_SCOPE), gate_posture="calibration_seam", gate_policy_suffix="calibration_seam_scope", - pipeline="uk-frs-calibration", - build_id_prefix="uk-frs-calibration-attempt-", + pipeline=UK_CALIBRATION_PIPELINE, + build_id_prefix=UK_CALIBRATION_ATTEMPT_ID_PREFIX, staging_operation_id="uk_national_calibration", release_id=UK_NATIONAL_RELEASE_ID, dataset_filename_template="microcosm_uk_{vintage}.h5", diff --git a/packages/microcosm-build/tests/test_staging_v2.py b/packages/microcosm-build/tests/test_staging_v2.py index 37ee574a2..45f65a95a 100644 --- a/packages/microcosm-build/tests/test_staging_v2.py +++ b/packages/microcosm-build/tests/test_staging_v2.py @@ -1017,3 +1017,83 @@ def whoami(self): ).credential_can_write() is None ) + + +class _RateLimitedApi(MemoryApi): + """Fail the next ``failures_left`` writes, as a rate-limited Hub does.""" + + def __init__(self, root: Path) -> None: + super().__init__(root) + self.failures_left = 0 + self.calls = 0 + + def upload_file(self, **kwargs): + self.calls += 1 + if self.failures_left > 0: + self.failures_left -= 1 + raise RuntimeError("429 rate limited token=do-not-record") + super().upload_file(**kwargs) + + +def _paused_remote_recorder(tmp_path): + api = _RateLimitedApi(tmp_path) + sleeps: list[float] = [] + telemetry = _recorder( + tmp_path, + delivery_mode="local_and_remote", + repo_id="policyengine/populace-uk-staging", + api=api, + upload_interval_seconds=0, + sleep=sleeps.append, + ) + telemetry.set_sample(_sample()) + # Three consecutive failures pause remote writes for the rest of the run. + api.failures_left = 3 + telemetry.stage("calibrating", force_upload=True) + assert telemetry.delivery_summary["last_error_code"] == "UPLOAD_FAILED" + calls_paused = api.calls + telemetry.stage("diagnostics", force_upload=True) + assert api.calls == calls_paused + return telemetry, api, sleeps + + +def test_completion_flushes_the_terminal_state_after_paused_uploads(tmp_path): + """The v20 last-mile gap: a paused run left its remote copy 'running'.""" + + telemetry, api, sleeps = _paused_remote_recorder(tmp_path) + # The first terminal attempt meets the limit again; the second, after a + # backed-off wait, lands. + api.failures_left = 3 + telemetry.complete() + assert sleeps == [15.0] + prefix = telemetry.repo_run_prefix + assert json.loads(api.files[f"{prefix}/progress.json"])["status"] == "completed" + assert {f"{prefix}/run_manifest.json", f"{prefix}/events.ndjson"} <= set(api.files) + delivery = telemetry.delivery_summary + assert delivery["last_error_code"] is None + assert delivery["upload_successes"] >= 3 + assert delivery["upload_attempts"] > delivery["upload_successes"] + + +def test_failure_flushes_the_terminal_state_after_paused_uploads(tmp_path): + telemetry, api, sleeps = _paused_remote_recorder(tmp_path) + telemetry.fail(RuntimeError("solver exploded")) + assert sleeps == [] + prefix = telemetry.repo_run_prefix + assert json.loads(api.files[f"{prefix}/progress.json"])["status"] == "failed" + assert telemetry.delivery_summary["last_error_code"] is None + + +def test_terminal_flush_gives_up_after_bounded_attempts(tmp_path, capsys): + telemetry, api, sleeps = _paused_remote_recorder(tmp_path) + api.failures_left = 100 + telemetry.complete() + assert sleeps == [15.0, 60.0] + assert telemetry.repo_run_prefix + "/progress.json" not in api.files + delivery = telemetry.delivery_summary + assert delivery["last_error_code"] == "UPLOAD_FAILED" + err = capsys.readouterr().err + assert "terminal state did not reach" in err + assert "do-not-record" not in err + # The local bundle is complete whatever the Hub did. + assert telemetry.validate_local_bundle()["progress"]["status"] == "completed" diff --git a/packages/microcosm-build/tests/test_uk_calibration_run.py b/packages/microcosm-build/tests/test_uk_calibration_run.py index e8e59ae96..f38ec430a 100644 --- a/packages/microcosm-build/tests/test_uk_calibration_run.py +++ b/packages/microcosm-build/tests/test_uk_calibration_run.py @@ -3,6 +3,7 @@ import hashlib import hmac import json +from datetime import UTC from pathlib import Path from types import SimpleNamespace @@ -1251,3 +1252,53 @@ def test_the_uk_block_delegates_rather_than_reassembling_the_shared_one(tmp_path continue assert block[field] == value, field assert block["manifest"]["schema_version"] == shared["schema_version"] + + +def test_run_uk_calibration_accepts_a_caller_minted_attempt_id(monkeypatch, tmp_path): + """The rowwise driver mints the id before telemetry opens (microcosm#823).""" + + pytest.importorskip("tables") + monkeypatch.setattr( + calibration_run, + "uk_aggregate_admin_totals", + lambda frame, manifest: (_admin_anchor_values(), []), + ) + input_h5 = tmp_path / "input.h5" + frame = _frame() + write_uk_national_frame(frame, input_h5) + _write_spine_sidecar(input_h5, frame) + paths = _paths(tmp_path) + paths = UKCalibrationRunPaths( + input_h5=input_h5, + staging_h5=paths.staging_h5, + diagnostics_json=paths.diagnostics_json, + build_record_json=paths.build_record_json, + terminal_gate_json=paths.terminal_gate_json, + ) + minted = calibration_run.new_uk_calibration_attempt_id( + timestamp=__import__("datetime").datetime(2026, 9, 20, 12, 0, tzinfo=UTC) + ) + assert minted.startswith("uk-frs-calibration-attempt-20260920T120000Z-") + common = dict( + paths=paths, + input_sha256=_sha(input_h5), + ledger_artifact=object(), + register_registry=_registry(), + band_edge_registry=_registry(), + calibration_year=2025, + exclusion_receipt={}, + doctrine=UKNationalSolveDoctrine(epochs=1), + doctrine_overrides={}, + measure_resolver=None, + source_pins={ + "input_h5": {"sha256": _sha(input_h5), "size_bytes": 1}, + "ledger_facts": {"sha256": "a" * 64, "size_bytes": 1}, + }, + run_config_extra={"calibration_year": 2025}, + release_id="test-run", + ) + with pytest.raises(ValueError, match="uk-frs-calibration-attempt-"): + run_uk_calibration(build_id="uk-local-candidate-f100-s42-x", **common) + result = run_uk_calibration(build_id=minted, **common) + assert result.build_record["build_id"] == minted + assert result.logbook_spool is not None diff --git a/packages/microcosm-build/tests/test_uk_chronicle_feed.py b/packages/microcosm-build/tests/test_uk_chronicle_feed.py index 660312f0d..dcf5e4096 100644 --- a/packages/microcosm-build/tests/test_uk_chronicle_feed.py +++ b/packages/microcosm-build/tests/test_uk_chronicle_feed.py @@ -58,3 +58,41 @@ def test_national_feed_rejects_malformed_identity( with pytest.raises(ValueError, match=field): chronicle_feed.load_uk_chronicle_feed() + + +def test_committed_feed_pin_check_refuses_a_foreign_artifact(): + from microcosm.build.uk_runtime.chronicle_feed import ( + UKChronicleFeedPinError, + load_uk_chronicle_feed, + require_committed_uk_chronicle_feed_pin, + ) + + pin = load_uk_chronicle_feed() + assert ( + require_committed_uk_chronicle_feed_pin( + pin.facts_sha256, + manifest_sha256=pin.manifest_sha256, + allow_unpinned_feed=False, + pin=pin, + ) + is pin + ) + with pytest.raises(UKChronicleFeedPinError, match="facts: loaded 0") as excinfo: + require_committed_uk_chronicle_feed_pin( + "0" * 64, + manifest_sha256=pin.manifest_sha256, + allow_unpinned_feed=False, + pin=pin, + ) + assert "manifest:" not in str(excinfo.value) + with pytest.raises(UKChronicleFeedPinError, match="manifest: loaded None"): + require_committed_uk_chronicle_feed_pin( + pin.facts_sha256, manifest_sha256=None, allow_unpinned_feed=False, pin=pin + ) + # A reviewed diagnostic run may override; the caller records the override. + assert ( + require_committed_uk_chronicle_feed_pin( + "0" * 64, manifest_sha256=None, allow_unpinned_feed=True, pin=pin + ) + is pin + ) diff --git a/packages/microcosm-build/tests/test_uk_rowwise_candidate.py b/packages/microcosm-build/tests/test_uk_rowwise_candidate.py index 5cb41fc0e..1d8041572 100644 --- a/packages/microcosm-build/tests/test_uk_rowwise_candidate.py +++ b/packages/microcosm-build/tests/test_uk_rowwise_candidate.py @@ -3488,11 +3488,3 @@ def test_national_role_refuses_release_candidate_with_the_seam_reason(tmp_path): args = builder._parse_args(_role_argv(tmp_path, "national", "--release-candidate")) with pytest.raises(ValueError, match="cannot sign shippability"): builder._validate_cli_args(args) - - -def test_national_role_build_is_not_yet_wired(tmp_path) -> None: - """Declared and validated, but the seam delegation is the next increment.""" - - builder = _load_builder_module() - with pytest.raises(NotImplementedError, match="microcosm#823"): - builder.main(_role_argv(tmp_path, "national", "--staging-local-only")) diff --git a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py new file mode 100644 index 000000000..8242daed5 --- /dev/null +++ b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py @@ -0,0 +1,509 @@ +"""The UK rowwise driver's national release role (microcosm#823). + +The national role delegates the build to the calibration seam library, so +these tests stand on the seam run suite's synthetic frame, sidecar and +register (loaded from that module) and on the candidate suite's driver +loader and fake Hub. +""" + +from __future__ import annotations + +import hashlib +import importlib.util +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from microcosm.build.logbook import load_spool_rows +from microcosm.build.staging_v2 import validate_v2_bundle +from microcosm.build.uk_runtime import calibration_run +from microcosm.build.uk_runtime.calibration_run import UK_CALIBRATION_GATE_SCOPE +from microcosm.build.uk_runtime.national_frame import write_uk_national_frame +from microcosm.build.uk_runtime.release_identity import UK_NATIONAL_RELEASE_ID +from microcosm.calibrate import TargetRegistry + +_TESTS = Path(__file__).resolve().parent + + +def _load_test_module(name: str, filename: str): + spec = importlib.util.spec_from_file_location(name, _TESTS / filename) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +_SEAM = _load_test_module("uk_calibration_run_fixtures", "test_uk_calibration_run.py") +_CANDIDATE = _load_test_module( + "uk_rowwise_candidate_fixtures", "test_uk_rowwise_candidate.py" +) + + +def _sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +class _FakeResolver: + """Stands in for the engine-backed resolver; the seam solves the fixture + register from the frame's own columns when the resolver is ``None``.""" + + def __init__(self, **kwargs): + self.kwargs = kwargs + + +def _national_inputs(builder, monkeypatch, tmp_path: Path, *, resolver=None): + frame = _SEAM._frame() + input_h5 = tmp_path / "spine.h5" + write_uk_national_frame(frame, input_h5) + _SEAM._write_spine_sidecar(input_h5, frame) + registry = _SEAM._registry() + ledger_dir = tmp_path / "ledger" + ledger_dir.mkdir() + (ledger_dir / "consumer_facts.jsonl").write_text("{}\n") + artifact = SimpleNamespace( + facts=None, + facts_sha256="1" * 64, + manifest_sha256="2" * 64, + path=ledger_dir, + provenance=lambda: { + "facts_sha256": "1" * 64, + "manifest_sha256": "2" * 64, + "artifact_id": "synthetic-national-fixture", + }, + ) + pin = SimpleNamespace( + to_dict=lambda: {"facts_sha256": "1" * 64, "manifest_sha256": "2" * 64} + ) + monkeypatch.setattr( + builder, "load_ledger_consumer_artifact", lambda path, **kwargs: artifact + ) + monkeypatch.setattr( + builder, + "require_committed_uk_chronicle_feed_pin", + lambda facts_sha256, **kwargs: pin, + ) + monkeypatch.setattr( + builder, + "compile_uk_target_registry", + lambda facts, target_period: SimpleNamespace(registry=registry, unsupported=()), + ) + monkeypatch.setattr(builder, "load_uk_calibration_measure_exclusions", lambda p: ()) + monkeypatch.setattr( + builder, + "apply_uk_calibration_measure_exclusions", + lambda reg, exclusions: (reg, {}), + ) + monkeypatch.setattr( + builder, + "UKMeasureResolver", + (lambda **kwargs: None) if resolver is None else resolver, + ) + monkeypatch.setattr( + calibration_run, + "uk_aggregate_admin_totals", + lambda frame, manifest: (_SEAM._admin_anchor_values(), []), + ) + monkeypatch.setenv("MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY", _SEAM.SIGNING_KEY) + return input_h5, registry, artifact, pin + + +def _argv(input_h5: Path, out: Path, *extra: str) -> list[str]: + return [ + "--input-h5", + str(input_h5), + "--release-role", + "national", + "--input-sha256", + _sha(input_h5), + "--ledger-facts", + str(input_h5.parent / "ledger"), + "--ledger-facts-sha256", + "1" * 64, + "--ledger-manifest-sha256", + "2" * 64, + "--out", + str(out), + *extra, + ] + + +def _fake_seam_run(calls: list[dict]): + def fake_run(**kwargs): + calls.append(kwargs) + paths = kwargs["paths"] + paths.staging_h5.write_bytes(b"h5") + paths.diagnostics_json.write_text( + json.dumps( + { + "targets": [ + { + "name": "dwp.uc.households", + "target_name": "dwp.uc.households", + "relative_error": 0.05, + } + ] + } + ) + ) + paths.terminal_gate_json.write_text( + json.dumps( + { + "release_id": kwargs["release_id"], + "gates": { + gate: {"status": "passed"} for gate in UK_CALIBRATION_GATE_SCOPE + }, + } + ) + ) + record = { + "build_id": kwargs["build_id"], + "spine_provenance": {"spine_gate_report": {"sha256": "f" * 64}}, + "calibration": { + "weights": {"household_weight_kind": "calibrated"}, + "solve": { + "n_targets": 1, + "n_households": 4, + "initial_loss": 0.5, + "final_loss": 0.01, + "n_nonzero": 4, + }, + "effective_sample_size": 4.0, + "max_weight_ratio": 1.0, + "measure_resolution": None, + }, + } + paths.build_record_json.write_text(json.dumps(record)) + return SimpleNamespace( + build_record_sha256=_sha(paths.build_record_json), build_record=record + ) + + return fake_run + + +def test_uk_national_role_delegates_to_the_seam_library(monkeypatch, tmp_path): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, registry, artifact, pin = _national_inputs( + builder, monkeypatch, tmp_path, resolver=_FakeResolver + ) + calls: list[dict] = [] + monkeypatch.setattr(builder, "run_uk_calibration", _fake_seam_run(calls)) + out = tmp_path / "national" + + assert builder.main(_argv(input_h5, out, "--no-staging")) == 0 + + call = calls[0] + assert call["build_id"].startswith("uk-frs-calibration-attempt-") + assert call["paths"].input_h5 == input_h5 + assert call["paths"].staging_h5 == out / "microcosm_uk_2024_25.h5" + assert call["paths"].diagnostics_json == out / "calibration_diagnostics.json" + assert call["paths"].build_record_json == out / "build_record.json" + assert ( + call["paths"].terminal_gate_json + == out / "microcosm_uk_2024_25.terminal_gates.json" + ) + # The role's doctrine, with no overrides: the seam's own constants. + doctrine = call["doctrine"] + assert (doctrine.epochs, doctrine.learning_rate, doctrine.seed) == (1500, 0.02, 0) + assert doctrine.target_weight_rule == "family_equal" + assert call["doctrine_overrides"] == {} + assert call["release_id"] == UK_NATIONAL_RELEASE_ID + assert call["register_registry"] is registry + assert call["band_edge_registry"] is registry + assert call["calibration_year"] == 2025 + assert call["exclusion_receipt"] == {} + assert call["ledger_artifact"] is artifact + # The seam's direct_h5 resolver: the input file, never a live frame. + resolver = call["measure_resolver"] + assert isinstance(resolver, _FakeResolver) + assert resolver.kwargs["simulation_source"] == input_h5 + assert resolver.kwargs["frame"] is None + assert resolver.kwargs["year"] == 2025 + assert call["source_pins"]["input_h5"]["sha256"] == _sha(input_h5) + assert call["source_pins"]["ledger_facts"]["sha256"] == "1" * 64 + extra = call["run_config_extra"] + assert extra["release_role"] == "national" + assert extra["allow_unpinned_feed"] is False + assert extra["chronicle_feed_pin"] == pin.to_dict() + assert extra["rowwise_driver_parameters"]["release_role"] == "national" + assert extra["rowwise_driver_parameters"]["n_clones"] is None + assert call["staging_delivery"]["mode"] == "disabled" + assert call["staging_finalizer"] is None + assert call["progress_callback"] is None + + manifest = json.loads((out / builder.MANIFEST_FILENAME).read_text()) + assert manifest["schema_version"] == 4 + assert manifest["build_kind"] == "uk_national_calibrated_candidate" + assert manifest["release_role"] == "national" + assert manifest["release_id"] == UK_NATIONAL_RELEASE_ID + assert manifest["build_id"] == call["build_id"] + assert set(manifest["outputs"]) == { + "dataset", + "calibration_diagnostics", + "build_record", + "terminal_gate_report", + "national_target_registry", + } + assert manifest["outputs"]["build_record"]["sha256"] == _sha( + out / "build_record.json" + ) + assert manifest["solve"]["n_targets_by_kind"] == { + "national": 1, + "local": 0, + "ladder": 0, + } + assert ( + manifest["solve"]["pool_households"] == manifest["solve"]["n_households"] == 4 + ) + assert manifest["parameters"]["n_clones"] is None + assert manifest["parameters"]["doctrine"]["clone_count"] is None + assert manifest["parameters"]["doctrine"]["target_weight_rule"] == "family_equal" + assert manifest["fit"]["national_by_family"][0]["family"] == "dwp_universal_credit" + assert manifest["gate"]["scope"] == list(UK_CALIBRATION_GATE_SCOPE) + assert manifest["gate"]["posture"] == "calibration_seam" + assert manifest["failing_gate_ids"] == [] + assert manifest["releasable"] is False + assert manifest["release_posture"]["calibration_seam_gates_passed"] is True + assert ( + manifest["release_posture"]["shippable_by"] == "tools/certify_uk_release_cut.py" + ) + assert manifest["staged_dataset"]["status"] == "skipped" + assert manifest["staging_delivery"]["mode"] == "disabled" + frozen = TargetRegistry.from_json(out / "national_target_registry.json") + assert frozen.version == registry.version + + # Explicit solve flags are receipted overrides of the seam doctrine. + calls.clear() + assert ( + builder.main( + _argv( + input_h5, + tmp_path / "override", + "--no-staging", + "--epochs", + "5", + "--target-weight-rule", + "uniform", + "--target-loss-cap", + "4", + ) + ) + == 0 + ) + assert calls[0]["doctrine_overrides"] == { + "epochs": {"default": 1500, "effective": 5}, + "target_loss_cap": {"default": 10.0, "effective": 4.0}, + "target_weight_rule": {"default": "family_equal", "effective": "uniform"}, + } + + +def _sums_verify(directory: Path) -> None: + for line in (directory / "sha256sums.txt").read_text().splitlines(): + digest, name = line.split(" ", 1) + assert _sha(directory / name) == digest, name + + +def test_uk_national_role_builds_the_seam_evidence_and_stages_locally( + monkeypatch, tmp_path, capsys +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + out = tmp_path / "national" + + assert ( + builder.main(_argv(input_h5, out, "--staging-local-only", "--epochs", "5")) == 0 + ) + + manifest = json.loads((out / builder.MANIFEST_FILENAME).read_text()) + assert json.loads(capsys.readouterr().out) == manifest + run_id = manifest["build_id"] + record = json.loads((out / "build_record.json").read_text()) + assert record["build_id"] == run_id + assert record["pipeline"] == "uk-frs-calibration" + assert record["run_config"]["release_id"] == UK_NATIONAL_RELEASE_ID + assert record["run_config"]["release_role"] == "national" + assert record["run_config"]["doctrine_overrides"] == { + "epochs": {"default": 1500, "effective": 5} + } + assert record["run_config"]["doctrine"]["target_weight_rule"] == "family_equal" + # The bindings the release-cut certifier reads. + dataset = out / "microcosm_uk_2024_25.h5" + gates = out / "microcosm_uk_2024_25.terminal_gates.json" + diagnostics = out / "calibration_diagnostics.json" + assert record["artifacts"]["staging_h5"]["sha256"] == _sha(dataset) + assert record["artifacts"]["diagnostics_json"]["sha256"] == _sha(diagnostics) + assert record["artifacts"]["terminal_gate_json"]["sha256"] == _sha(gates) + assert record["spine_provenance"]["spine_gate_report"]["sha256"] + assert json.loads(diagnostics.read_text())["build"]["build_id"] == run_id + assert record["staging_delivery"]["mode"] == "local_only" + assert record["staging_delivery"] == manifest["staging_delivery"] + assert manifest["outputs"]["dataset"]["sha256"] == _sha(dataset) + assert manifest["outputs"]["build_record"]["sha256"] == _sha( + out / "build_record.json" + ) + assert manifest["build_record"]["sha256"] == _sha(out / "build_record.json") + assert set(manifest["gate"]["statuses"]) == set(UK_CALIBRATION_GATE_SCOPE) + assert manifest["solve"]["n_targets"] == 1 + assert manifest["staged_dataset"]["status"] == "skipped" + _sums_verify(out) + + bundle = validate_v2_bundle(out / "staging", run_id) + run_manifest = bundle["run_manifest"] + assert run_manifest["operation_id"] == "uk_national_calibration" + assert run_manifest["pipeline"]["id"] == "uk-frs-calibration" + assert run_manifest["run_kind"] == "calibration" + assert bundle["progress"]["status"] == "completed" + stage_ids = {event["stage_id"] for event in bundle["events"]} + assert { + "input_pinning", + "target_compilation", + "calibration", + "release_check_evaluation", + "build_record_creation", + "dataset_staging", + "complete", + } <= stage_ids + artifacts = out / "staging" / "runs" / run_id / "artifacts" + assert (artifacts / "fit_summary.json").is_file() + assert (artifacts / "staged_dataset.json").is_file() + fit_summary = json.loads((artifacts / "fit_summary.json").read_text()) + assert fit_summary["build_kind"] == "uk_national_calibrated_candidate" + assert fit_summary["fit_by_family"]["local"] == {} + assert "dwp_universal_credit" in fit_summary["fit_by_family"]["national"] + + rows = load_spool_rows(out / "logbook-spool") + assert len(rows) == 1 + assert rows[0].build_id == run_id + assert rows[0].pipeline == "uk-frs-calibration" + + +def test_uk_national_role_publishes_telemetry_and_the_bundle_to_the_hub( + monkeypatch, tmp_path +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + hub = _CANDIDATE._FakeHub() + monkeypatch.setattr(builder, "_hub_api", lambda: hub) + monkeypatch.setattr(builder, "_hub_token", lambda: "hf_test_token") + out = tmp_path / "national" + + assert ( + builder.main( + _argv( + input_h5, out, "--epochs", "5", "--staging-upload-interval-seconds", "0" + ) + ) + == 0 + ) + + manifest = json.loads((out / builder.MANIFEST_FILENAME).read_text()) + run_id = manifest["build_id"] + assert run_id.startswith("uk-frs-calibration-attempt-") + telemetry_paths = hub.paths("policyengine/populace-uk-staging") + assert telemetry_paths == sorted( + f"runs/{run_id}/{name}" + for name in ( + "run_manifest.json", + "progress.json", + "events.ndjson", + "calibration_progress.json", + "artifacts/fit_summary.json", + "artifacts/staged_dataset.json", + ) + ) + remote_progress = json.loads( + hub.files[("policyengine/populace-uk-staging", f"runs/{run_id}/progress.json")] + ) + assert remote_progress["status"] == "completed" + remote_manifest = json.loads( + hub.files[ + ("policyengine/populace-uk-staging", f"runs/{run_id}/run_manifest.json") + ] + ) + assert remote_manifest["operation_id"] == "uk_national_calibration" + delivery = manifest["staging_delivery"] + assert delivery["mode"] == "local_and_remote" + assert delivery["upload_successes"] == delivery["upload_attempts"] > 0 + assert delivery["last_error_code"] is None + + assert len(hub.commits) == 1 + commit = hub.commits[0] + assert commit["repo_id"] == "policyengine/populace-uk-private" + expected = {Path(entry["path"]).name for entry in manifest["outputs"].values()} | { + builder.MANIFEST_FILENAME, + "staged_manifest.json", + "sha256sums.txt", + } + assert commit["paths"] == sorted(f"staged/{run_id}/{name}" for name in expected) + assert "microcosm_uk_2024_25.h5" in expected and "build_record.json" in expected + staged = manifest["staged_dataset"] + assert staged["status"] == "uploaded" + assert staged["repository"] == "policyengine/populace-uk-private" + assert staged["prefix"] == f"staged/{run_id}" + remote_h5 = hub.files[ + ("policyengine/populace-uk-private", f"staged/{run_id}/microcosm_uk_2024_25.h5") + ] + assert remote_h5 == (out / "microcosm_uk_2024_25.h5").read_bytes() + record = json.loads((out / "build_record.json").read_text()) + assert record["staging_delivery"] == delivery + + +def test_uk_national_dry_run_prints_the_plan_and_writes_nothing( + monkeypatch, tmp_path, capsys +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + out = tmp_path / "national" + + assert builder.main(_argv(input_h5, out, "--dry-run")) == 0 + + plan = json.loads(capsys.readouterr().out) + assert plan["build_kind"] == "uk_national_calibrated_candidate_plan" + assert plan["release_role"] == "national" + assert plan["release_id"] == UK_NATIONAL_RELEASE_ID + assert plan["dry_run"] is True + assert plan["targets"]["active"] == plan["targets"]["compiled"] == 1 + assert plan["targets"]["register_sha256"] == registry.version + assert plan["doctrine"]["epochs"] == 1500 + assert plan["doctrine"]["target_weight_rule"] == "family_equal" + assert plan["doctrine_overrides"] == {} + assert plan["parameters"]["release_role"] == "national" + assert plan["engine"] == "not_run" + assert plan["releasable"] is False + assert not out.exists() + + +def test_uk_national_role_marks_the_staging_run_failed_on_a_refusal( + monkeypatch, tmp_path +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + monkeypatch.setattr( + builder, + "compile_uk_target_registry", + lambda facts, target_period: SimpleNamespace( + registry=None, unsupported=("dwp.uc.households",) + ), + ) + out = tmp_path / "national" + with pytest.raises(SystemExit, match="failed to compile"): + builder.main(_argv(input_h5, out, "--staging-local-only")) + runs = sorted(path.name for path in (out / "staging" / "runs").iterdir()) + assert len(runs) == 1 + bundle = validate_v2_bundle(out / "staging", runs[0]) + assert bundle["progress"]["status"] == "failed" + assert not (out / "build_record.json").exists() diff --git a/tools/build_uk_rowwise_candidate.py b/tools/build_uk_rowwise_candidate.py index 88b719de6..fe25c6843 100644 --- a/tools/build_uk_rowwise_candidate.py +++ b/tools/build_uk_rowwise_candidate.py @@ -130,10 +130,16 @@ ) from microcosm.build.uk_runtime.calibration_run import ( UK_LOCAL_GATE_SCOPE, + UKCalibrationRunPaths, finalize_uk_scoped_gate_report, + new_uk_calibration_attempt_id, + run_uk_calibration, uk_local_gate_scope_exclusions, uk_scoped_gate_manifest, ) +from microcosm.build.uk_runtime.chronicle_feed import ( + require_committed_uk_chronicle_feed_pin, +) from microcosm.build.uk_runtime.frs_release import load_uk_frs_release from microcosm.build.uk_runtime.ledger_targets import _spec_geography from microcosm.build.uk_runtime.measure_simulation import ( @@ -141,6 +147,7 @@ apply_uk_calibration_measure_exclusions, load_uk_calibration_measure_exclusions, ) +from microcosm.build.uk_runtime.national_doctrine import uk_doctrine_with_overrides from microcosm.build.uk_runtime.national_sampling import ( UK_SAMPLE_RUNG_TOKENS, UK_SAMPLE_SEED_DEFAULT, @@ -975,12 +982,12 @@ def main(argv: list[str] | None = None) -> int: _validate_cli_args(args) posture = _posture_of(args) if posture.role == "national": - raise NotImplementedError( - "--release-role national is declared and validated here, but its " - "build flow (delegation to the calibration seam) lands in the next " - "increment of microcosm#823; run tools/calibrate_uk_national_dataset.py " - "until then." - ) + if args.dry_run: + return _national_dry_run(args) + # Argument refusals above cost nothing; the credential check reaches + # the Hub, so it runs last, still before any input is read. + _preflight_staged_dataset(args) + return _run_national_role(args) if args.candidate_clone_counts is not None and not args.dry_run: raise ValueError("--candidate-clone-counts is valid only with --dry-run.") if _CONSERVE_MASS: @@ -1022,7 +1029,7 @@ def main(argv: list[str] | None = None) -> int: predecessor = resolve_predecessor(args.logbook_prev_row_digest) # Staging telemetry opens with the attempt, as in the spine builder, so an # early refusal still leaves a failed run under runs//. - telemetry = _create_staging_telemetry(args, state=state) + telemetry = _create_staging_telemetry(args, build_id=state.build_id) try: return _run_candidate( args, @@ -1689,6 +1696,509 @@ def _run_candidate( raise +def _read_json(path: Path) -> dict[str, Any]: + payload = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(payload, dict): + raise ValueError(f"{path} must hold a JSON object.") + return payload + + +def _ledger_facts_pin(artifact: Any) -> dict[str, object]: + facts_path = ( + artifact.path / "consumer_facts.jsonl" + if artifact.path.is_dir() + else artifact.path + ) + return {"sha256": artifact.facts_sha256, "size_bytes": facts_path.stat().st_size} + + +def _load_national_target_inputs(args: argparse.Namespace) -> dict[str, Any]: + """The national role's target surface: the pinned Ledger artifact, compiled. + + The artifact must be the committed Chronicle feed pin unless + ``--allow-unpinned-feed`` records a reviewed diagnostic run; the + compiled register, less the measure exclusions, is the solve surface and + the full compiled register keeps the band edges; ``--register-json`` + requires the re-derived register to be the frozen scoring surface. + """ + + artifact = load_ledger_consumer_artifact( + args.ledger_facts, + expected_facts_sha256=args.ledger_facts_sha256, + expected_manifest_sha256=args.ledger_manifest_sha256, + ) + pin = require_committed_uk_chronicle_feed_pin( + artifact.facts_sha256, + manifest_sha256=artifact.manifest_sha256, + allow_unpinned_feed=bool(args.allow_unpinned_feed), + ) + calibration_year = int(load_uk_frs_release().calibration_year) + compilation = compile_uk_target_registry( + artifact.facts, target_period=calibration_year + ) + if compilation.unsupported: + raise SystemExit( + f"{len(compilation.unsupported)} national target references " + "failed to compile" + ) + exclusions = load_uk_calibration_measure_exclusions(args.measure_exclusions) + registry, exclusion_receipt = apply_uk_calibration_measure_exclusions( + compilation.registry, exclusions + ) + if args.register_json is not None: + try: + frozen = TargetRegistry.from_json(args.register_json) + except ValueError as error: + raise SystemExit( + f"error: frozen scoring register is unusable: {error}" + ) from error + if frozen.version != registry.version: + raise SystemExit( + "re-derived register differs from the frozen scoring register: " + f"{registry.version} vs {frozen.version}" + ) + return { + "artifact": artifact, + "calibration_year": calibration_year, + "national_registry": registry, + "band_edge_registry": compilation.registry, + "measure_exclusions": exclusion_receipt, + "chronicle_feed_pin": pin.to_dict(), + } + + +def _national_doctrine_overrides(args: argparse.Namespace) -> dict[str, Any]: + """The receipted per-run overrides of the seam doctrine, explicit flags only.""" + + explicit = args._explicit_arguments + overrides: dict[str, Any] = {} + if "epochs" in explicit: + overrides["epochs"] = int(args.epochs) + if "learning_rate" in explicit: + overrides["learning_rate"] = float(args.learning_rate) + if "target_weight_rule" in explicit: + overrides["target_weight_rule"] = str(args.target_weight_rule) + if args.target_loss_cap is not None: + overrides["target_loss_cap"] = float(args.target_loss_cap) + return overrides + + +def _national_dry_run(args: argparse.Namespace) -> int: + """Compile the national target surface and print the plan; write nothing.""" + + posture = _posture_of(args) + input_h5 = _require_file(args.input_h5, label="--input-h5") + input_artifact = _artifact_info(input_h5) + _verify_requested_pin("--input-h5", input_artifact, requested=args.input_sha256) + frs_release = load_uk_frs_release() + inputs = _load_national_target_inputs(args) + doctrine, doctrine_overrides = uk_doctrine_with_overrides( + **_national_doctrine_overrides(args) + ) + plan = { + "schema_version": 3, + "build_kind": "uk_national_calibrated_candidate_plan", + "release_role": posture.role, + "release_id": posture.release_id, + "dry_run": True, + "calibration_year": inputs["calibration_year"], + "inputs": {"dataset": dict(input_artifact)}, + "targets": { + "chronicle": inputs["artifact"].provenance(), + "compiled": len(inputs["band_edge_registry"].specs), + "active": len(inputs["national_registry"].specs), + "excluded": len(inputs["measure_exclusions"]), + "register_sha256": inputs["national_registry"].version, + }, + "doctrine": { + field: getattr(doctrine, field) + for field in ( + "epochs", + "learning_rate", + "max_weight_ratio", + "seed", + "target_loss_cap", + "scale_rule", + "target_weight_rule", + "mass_rule", + "l0_lambda", + ) + }, + "doctrine_overrides": dict(doctrine_overrides), + "parameters": _parameters( + args, + source_year=_source_year( + args.source_year, time_period=str(frs_release.time_period) + ), + ), + "engine": "not_run", + "releasable": False, + } + print(_json_text(plan)) + return 0 + + +def _run_national_role(args: argparse.Namespace) -> int: + """Build the national line: the calibration seam under the driver's posture. + + The seam library (:func:`run_uk_calibration`) resolves the measures from + the input file, solves under the seam doctrine, runs the six + calibration-seam gates, writes the H5, the diagnostics, the signed gate + report, the build record and the Logbook row exactly as the retiring + seam command did, so a national cut built here is bit-for-bit the seam's. + The driver adds what the dense role has: the pinned input, the role's + doctrine and overrides, staging telemetry under the attempt id, the + rowwise manifest beside the seam's evidence, and the staged bundle. + """ + + posture = _posture_of(args) + out_dir = args.out.expanduser().resolve() + input_h5 = _require_file(args.input_h5, label="--input-h5") + if out_dir.exists() and not out_dir.is_dir(): + raise ValueError(f"--out must be a directory path, got {out_dir}.") + input_artifact = _artifact_info(input_h5) + _verify_requested_pin("--input-h5", input_artifact, requested=args.input_sha256) + # The attempt id is minted before telemetry opens so the staging run id + # and the Logbook row agree, as on the dense role. + build_id = new_uk_calibration_attempt_id(timestamp=datetime.now(UTC)) + telemetry = _create_staging_telemetry(args, build_id=build_id) + try: + return _run_national_attempt( + args, + posture=posture, + out_dir=out_dir, + input_h5=input_h5, + input_artifact=input_artifact, + build_id=build_id, + telemetry=telemetry, + ) + except BaseException as error: + _fail_staging_telemetry(telemetry, error) + raise + + +def _run_national_attempt( + args: argparse.Namespace, + *, + posture: UKRowwisePosture, + out_dir: Path, + input_h5: Path, + input_artifact: Mapping[str, Any], + build_id: str, + telemetry: StagingTelemetryV2 | None, +) -> int: + _stage( + telemetry, + "input_pinning", + "completed", + dataset_sha256=input_artifact["sha256"], + ) + if telemetry is not None: + telemetry.set_sample({"mode": "full"}) + frs_release = load_uk_frs_release() + calibration_year = int(frs_release.calibration_year) + args._calibration_year = calibration_year + args._frs_vintage = str(frs_release.vintage) + source_year = _source_year( + args.source_year, time_period=str(frs_release.time_period) + ) + output_paths = _output_paths(out_dir, posture=posture, vintage=args._frs_vintage) + _validate_output_paths(output_paths, input_h5=input_h5, ladder_path=None) + _stage(telemetry, "target_compilation", "started") + inputs = _load_national_target_inputs(args) + _stage( + telemetry, + "target_compilation", + "completed", + compiled_target_count=len(inputs["band_edge_registry"].specs), + active_target_count=len(inputs["national_registry"].specs), + ) + doctrine, doctrine_overrides = uk_doctrine_with_overrides( + **_national_doctrine_overrides(args) + ) + if doctrine.target_weight_rule != args.target_weight_rule: + raise RuntimeError( + "national doctrine override did not bind the requested rule." + ) + args._doctrine_override_receipt = doctrine_overrides + out_dir.mkdir(parents=True, exist_ok=True) + # The frozen register is the scorer's input: the same artifact this run + # solved against, by content hash. + inputs["national_registry"].to_json(output_paths["national_registry"]) + resolver = UKMeasureResolver( + simulation_source=input_h5, + scratch_dir=out_dir, + year=calibration_year, + frame=None, + ) + paths = UKCalibrationRunPaths( + input_h5=input_h5, + staging_h5=output_paths["dataset"], + diagnostics_json=output_paths["calibration_diagnostics"], + build_record_json=output_paths["build_record"], + terminal_gate_json=output_paths["terminal_gates"], + ) + evidence: dict[str, Any] = {} + + def publish_manifest() -> None: + # The seam has written its evidence; the manifest describes it and the + # staged bundle (every manifest-registered output) follows, as on the + # dense role. The staged copy predates the evidence blocks appended + # below; staged_manifest.json describes the remote side. + args._gate_report = _read_json(paths.terminal_gate_json) + manifest = _national_manifest( + args, + posture=posture, + build_id=build_id, + build_record=_read_json(paths.build_record_json), + build_record_path=paths.build_record_json, + gate_report=args._gate_report, + diagnostics=_read_json(paths.diagnostics_json), + inputs=inputs, + doctrine_overrides=doctrine_overrides, + output_paths=output_paths, + input_artifact=input_artifact, + source_year=source_year, + ) + _replace_manifest(output_paths["manifest"], manifest) + evidence["manifest"] = manifest + evidence["staged_dataset"] = _stage_dataset( + args, + manifest=manifest, + output_paths=output_paths, + run_id=build_id if telemetry is None else telemetry.run_id, + telemetry=telemetry, + ) + + def finalize_staging() -> None: + publish_manifest() + _finalize_staging_telemetry(args, telemetry) + + def event_callback(stage_id: str, status: str, details: Mapping[str, Any]) -> None: + _stage(telemetry, stage_id, status, **dict(details)) + + result = run_uk_calibration( + paths=paths, + build_id=build_id, + input_sha256=str(args.input_sha256), + ledger_artifact=inputs["artifact"], + register_registry=inputs["national_registry"], + band_edge_registry=inputs["band_edge_registry"], + calibration_year=calibration_year, + exclusion_receipt=inputs["measure_exclusions"], + doctrine=doctrine, + doctrine_overrides=doctrine_overrides, + measure_resolver=resolver, + source_pins={ + "input_h5": { + "sha256": str(args.input_sha256), + "size_bytes": int(input_artifact["bytes"]), + }, + "ledger_facts": _ledger_facts_pin(inputs["artifact"]), + }, + run_config_extra={ + "release_role": posture.role, + "calibration_year": calibration_year, + "allow_unpinned_feed": bool(args.allow_unpinned_feed), + "chronicle_feed_pin": inputs["chronicle_feed_pin"], + "rowwise_driver_parameters": _parameters(args, source_year=source_year), + }, + release_id=posture.release_id, + logbook_prev_row_digest=args.logbook_prev_row_digest, + progress_callback=( + None + if telemetry is None + else _thinned_epochs( + telemetry.calibration_progress, every=_staging_epoch_every(args) + ) + ), + event_callback=None if telemetry is None else event_callback, + staging_delivery=_staging_delivery(telemetry), + staging_finalizer=None if telemetry is None else finalize_staging, + staging_delivery_provider=( + None if telemetry is None else (lambda: telemetry.delivery_summary) + ), + ) + if telemetry is None: + publish_manifest() + manifest = evidence["manifest"] + # The seam rewrote its record with the delivery summary after the + # finalizer; the manifest binds the record as it now is. + manifest["outputs"]["build_record"] = _artifact_info(paths.build_record_json) + manifest["build_record"] = { + "path": str(paths.build_record_json), + "sha256": result.build_record_sha256, + } + manifest["staging_delivery"] = _staging_delivery(telemetry) + manifest["staged_dataset"] = evidence["staged_dataset"] + _replace_manifest(output_paths["manifest"], manifest) + if (out_dir / SHA256SUMS_FILENAME).is_file(): + # The uploaded copies list the files as uploaded; the local sums + # list the record and the manifest as they now are, evidence included. + refresh_sha256sums_entry(out_dir, paths.build_record_json.name) + refresh_sha256sums_entry(out_dir, output_paths["manifest"].name) + print(_json_text(manifest)) + return 0 + + +def _national_fit_by_family( + diagnostics: Mapping[str, Any], registry: TargetRegistry +) -> list[dict[str, object]]: + families = {str(spec.name): str(spec.family or "") for spec in registry.specs} + rows = [] + for row in diagnostics.get("targets", []): + if not isinstance(row, Mapping): + continue + error = row.get("relative_error") + if error is None: + continue + # The diagnostics name a target by its register spec name and, on + # period-suffixed rows, by a materialized name; the family lives on + # the spec. + labels = [ + str(label) + for label in (row.get("target_name"), row.get("name")) + if label is not None + ] + family = next((families[label] for label in labels if label in families), "") + rows.append( + { + "target_name": labels[0] if labels else "", + "family": family, + "abs_relative_error": abs(float(error)), + } + ) + if not rows: + return [] + return uk_fit_by_family(pd.DataFrame(rows)) + + +def _national_manifest( + args: argparse.Namespace, + *, + posture: UKRowwisePosture, + build_id: str, + build_record: Mapping[str, Any], + build_record_path: Path, + gate_report: Mapping[str, Any], + diagnostics: Mapping[str, Any], + inputs: Mapping[str, Any], + doctrine_overrides: Mapping[str, Any], + output_paths: Mapping[str, Path], + input_artifact: Mapping[str, Any], + source_year: int, +) -> dict[str, Any]: + """The rowwise manifest of a national-role run, over the seam's evidence.""" + + calibration = build_record["calibration"] + solve = calibration["solve"] + statuses = _gate_statuses(gate_report) + abs_errors = np.asarray( + [ + abs(float(row["relative_error"])) + for row in diagnostics.get("targets", []) + if isinstance(row, Mapping) and row.get("relative_error") is not None + ], + dtype=np.float64, + ) + fit_rows = _national_fit_by_family(diagnostics, inputs["national_registry"]) + outputs = { + "dataset": _artifact_info(output_paths["dataset"]), + "calibration_diagnostics": _artifact_info( + output_paths["calibration_diagnostics"] + ), + "build_record": _artifact_info(build_record_path), + "terminal_gate_report": _artifact_info(output_paths["terminal_gates"]), + "national_target_registry": _artifact_info(output_paths["national_registry"]), + } + return { + "schema_version": 4, + "build_kind": "uk_national_calibrated_candidate", + "release_role": posture.role, + "release_id": posture.release_id, + "build_id": build_id, + "candidate_scope": "national", + "created_at": datetime.now(UTC).isoformat(), + "git_commit": _git_commit(), + "git_dirty": _git_dirty(), + "parameters": _parameters(args, source_year=source_year), + "inputs": {"dataset": dict(input_artifact)}, + "identity": { + "spine": { + **dict(input_artifact), + "spine_provenance": dict(build_record["spine_provenance"]), + }, + "targets": {"chronicle": inputs["artifact"].provenance()}, + "code": {"git_commit": _git_commit(), "git_dirty": _git_dirty()}, + "runtime": runtime_provenance(), + "sampling": {"mode": "full"}, + "survey_year": source_year, + "calibration_year": int(inputs["calibration_year"]), + }, + "sampling": {"mode": "full"}, + "outputs": outputs, + "weights": dict(calibration["weights"]), + "solve": { + "n_targets": int(solve["n_targets"]), + "n_targets_by_kind": { + "national": int(solve["n_targets"]), + "local": 0, + "ladder": 0, + }, + "n_households": int(solve["n_households"]), + "pool_households": int(solve["n_households"]), + "initial_loss": float(solve["initial_loss"]), + "final_loss": float(solve["final_loss"]), + "n_nonzero": int(solve["n_nonzero"]), + "max_abs_relative_error": ( + float(abs_errors.max()) if abs_errors.size else None + ), + "median_abs_relative_error": ( + float(np.median(abs_errors)) if abs_errors.size else None + ), + "effective_sample_size": calibration.get("effective_sample_size"), + "max_weight_ratio": calibration.get("max_weight_ratio"), + "target_weight_rule": str(args.target_weight_rule), + "target_weight_rule_override": dict( + doctrine_overrides.get("target_weight_rule", {}) + ), + "doctrine_overrides": dict(doctrine_overrides), + "measure_resolution": calibration.get("measure_resolution"), + }, + "fit": { + "national_by_family": fit_rows, + "weakest_families": sorted( + fit_rows, + key=lambda row: ( + -float(row["worst_abs_relative_error"]), + row["family"], + ), + )[:10], + }, + "gate": { + "scope": list(posture.gate_scope), + "posture": posture.gate_posture, + "release_id": gate_report.get("release_id"), + "statuses": statuses, + }, + "failing_gate_ids": sorted( + gate_id for gate_id, status in statuses.items() if status != "passed" + ), + "releasable": False, + "release_posture": { + "release_candidate": False, + "shippable_by": "tools/certify_uk_release_cut.py", + "calibration_seam_gates_passed": bool(statuses) + and all(status == "passed" for status in statuses.values()), + }, + "measure_exclusions": dict(inputs["measure_exclusions"]), + "build_record": { + "path": str(build_record_path), + "sha256": outputs["build_record"]["sha256"], + }, + } + + def _hub_api() -> Any: """The Hub client used for telemetry and the staged dataset (test seam).""" @@ -1773,7 +2283,7 @@ def _require_write_credential(storage: HuggingFaceDatasetStorage, *, hint: str) def _create_staging_telemetry( - args: argparse.Namespace, *, state: AttemptState + args: argparse.Namespace, *, build_id: str ) -> StagingTelemetryV2 | None: if args.no_staging: return None @@ -1781,12 +2291,12 @@ def _create_staging_telemetry( out_dir = args.out.expanduser().resolve() posture = _posture_of(args) return StagingTelemetryV2( - run_id=args.staging_run_id or state.build_id, + run_id=args.staging_run_id or build_id, country_code="GB", operation_id=posture.staging_operation_id, pipeline_id=posture.pipeline, pipeline_version=metadata.version("microcosm-build"), - candidate_id=args.staging_candidate_id or state.build_id, + candidate_id=args.staging_candidate_id or build_id, local_dir=args.staging_dir or out_dir / "staging", run_kind="calibration", delivery_mode="local_only" if local_only else "local_and_remote", @@ -3992,6 +4502,9 @@ def _refuse_dense_role_arguments( refused.append("--sample-fraction") if "sample_seed" in explicit: refused.append("--sample-seed") + if "seed" in explicit and args.seed != posture.seed: + # The seam doctrine's seed is a reviewed constant, not a knob. + refused.append(f"--seed != doctrine {posture.seed}") if args.target_weight_rule not in posture.allowed_target_weight_rules: refused.append(f"--target-weight-rule {args.target_weight_rule}") if refused: @@ -4056,12 +4569,14 @@ def _validate_output_paths( output_paths: Mapping[str, Path], *, input_h5: Path, - ladder_path: Path, + ladder_path: Path | None, ) -> None: resolved = {name: path.resolve() for name, path in output_paths.items()} if len(set(resolved.values())) != len(resolved): raise ValueError("candidate output paths must be distinct.") - protected = {input_h5.resolve(), ladder_path.resolve()} + protected = {input_h5.resolve()} + if ladder_path is not None: + protected.add(ladder_path.resolve()) collisions = sorted(str(path) for path in resolved.values() if path in protected) if collisions: raise ValueError( diff --git a/tools/calibrate_uk_national_dataset.py b/tools/calibrate_uk_national_dataset.py index de6b70dc7..0cdda8d05 100644 --- a/tools/calibrate_uk_national_dataset.py +++ b/tools/calibrate_uk_national_dataset.py @@ -36,7 +36,9 @@ ) from microcosm.build.uk_runtime.chronicle_feed import ( UKChronicleFeed, + UKChronicleFeedPinError, load_uk_chronicle_feed, + require_committed_uk_chronicle_feed_pin, ) from microcosm.build.uk_runtime.frs_release import load_uk_frs_release from microcosm.build.uk_runtime.ledger_targets import compile_uk_target_registry @@ -343,21 +345,15 @@ def _check_committed_ledger_feed_pin( allow_unpinned_feed: bool, pin: UKChronicleFeed | None = None, ) -> None: - pin = pin or load_uk_chronicle_feed() - mismatches = [] - for label, loaded, committed in ( - ("facts", facts_sha256, pin.facts_sha256), - ("manifest", manifest_sha256, pin.manifest_sha256), - ): - if loaded != committed: - mismatches.append(f"{label}: loaded {loaded}, committed {committed}") - if mismatches and not allow_unpinned_feed: - raise SystemExit( - "error: Chronicle artifact differs from the committed UK national feed pin: " - + "; ".join(mismatches) - + "; pass " - "--allow-unpinned-feed only for an explicitly reviewed diagnostic run" + try: + require_committed_uk_chronicle_feed_pin( + facts_sha256, + manifest_sha256=manifest_sha256, + allow_unpinned_feed=allow_unpinned_feed, + pin=pin, ) + except UKChronicleFeedPinError as error: + raise SystemExit(f"error: {error}") from error def _validate_distinct_paths(paths: dict[str, Path]) -> None: From 0ccf8f55df70ec71517b0de192c32a99a91f60bd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:51:58 +0100 Subject: [PATCH 04/11] Add the seam-versus-national-role parity twin (#823) On the seam run suite's synthetic frame and register, seed 0, five epochs, no staging, tools/calibrate_uk_national_dataset.py and tools/build_uk_rowwise_candidate.py --release-role national produce identical household weights (assert_array_equal), identical diagnostics rows and identical seam build records (calibration, gate summary, register, input posture, spine provenance, doctrine and its receipt), differing only in the attempt id and the run config's release id and role fields. Green on 2026-09-20; this receipt licenses retiring the seam command in the next commit, which removes the twin with it. Co-Authored-By: Claude Fable 5.1 --- .../tests/test_uk_seam_parity_twin.py | 226 ++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 packages/microcosm-build/tests/test_uk_seam_parity_twin.py diff --git a/packages/microcosm-build/tests/test_uk_seam_parity_twin.py b/packages/microcosm-build/tests/test_uk_seam_parity_twin.py new file mode 100644 index 000000000..e6ba29d20 --- /dev/null +++ b/packages/microcosm-build/tests/test_uk_seam_parity_twin.py @@ -0,0 +1,226 @@ +"""Parity twin: the retiring seam command and the driver's national role. + +Both paths call ``run_uk_calibration`` with the same doctrine on the same +synthetic frame and register (seed 0, five epochs, no staging), so the +calibrated household weights, the diagnostics rows and the seam-shaped build +record must agree exactly. The receipt is what licenses retiring +``tools/calibrate_uk_national_dataset.py`` (microcosm#823 B3); this module +retires with it. +""" + +from __future__ import annotations + +import hashlib +import importlib.util +import json +from pathlib import Path +from types import SimpleNamespace + +import numpy as np +import pytest + +from microcosm.build.uk_runtime import calibration_run +from microcosm.build.uk_runtime.ledger_targets import UKLedgerTargetCompilation +from microcosm.build.uk_runtime.national_frame import ( + load_uk_national_frame, + write_uk_national_frame, +) + +_TESTS = Path(__file__).resolve().parent +_ROOT = _TESTS.parents[2] + + +def _load_module(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +_SEAM_TESTS = _load_module( + "uk_calibration_run_fixtures", _TESTS / "test_uk_calibration_run.py" +) +_CANDIDATE_TESTS = _load_module( + "uk_rowwise_candidate_fixtures", _TESTS / "test_uk_rowwise_candidate.py" +) + + +def _sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _stub_inputs(module, monkeypatch, *, registry, ledger_dir: Path): + from microcosm.build.uk_runtime.chronicle_feed import load_uk_chronicle_feed + + pin = load_uk_chronicle_feed() + artifact = SimpleNamespace( + facts=None, + facts_sha256=pin.facts_sha256, + manifest_sha256=pin.manifest_sha256, + path=ledger_dir, + provenance=lambda: { + "facts_sha256": pin.facts_sha256, + "manifest_sha256": pin.manifest_sha256, + "artifact_id": "synthetic-parity-fixture", + }, + ) + monkeypatch.setattr( + module, "load_ledger_consumer_artifact", lambda *args, **kwargs: artifact + ) + monkeypatch.setattr( + module, + "compile_uk_target_registry", + lambda facts, target_period: UKLedgerTargetCompilation(registry, ()), + ) + monkeypatch.setattr(module, "load_uk_calibration_measure_exclusions", lambda p: ()) + monkeypatch.setattr( + module, + "apply_uk_calibration_measure_exclusions", + lambda reg, exclusions: (reg, {}), + ) + monkeypatch.setattr(module, "UKMeasureResolver", lambda **kwargs: None) + return pin + + +def _weights(path: Path) -> np.ndarray: + frame, _provenance = load_uk_national_frame(path) + return np.asarray(frame.weights_for("household").values, dtype=np.float64) + + +def _diagnostics_rows(path: Path) -> list[tuple]: + payload = json.loads(path.read_text()) + return [ + ( + row["name"], + row["target"], + row["initial_estimate"], + row["final_estimate"], + row["relative_error"], + ) + for row in payload["targets"] + ] + + +def test_uk_seam_command_and_national_role_agree_bit_for_bit(monkeypatch, tmp_path): + pytest.importorskip("tables") + monkeypatch.setenv( + "MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY", _SEAM_TESTS.SIGNING_KEY + ) + monkeypatch.setattr( + calibration_run, + "uk_aggregate_admin_totals", + lambda frame, manifest: (_SEAM_TESTS._admin_anchor_values(), []), + ) + frame = _SEAM_TESTS._frame() + input_h5 = tmp_path / "spine.h5" + write_uk_national_frame(frame, input_h5) + _SEAM_TESTS._write_spine_sidecar(input_h5, frame) + registry = _SEAM_TESTS._registry() + ledger_dir = tmp_path / "ledger" + ledger_dir.mkdir() + (ledger_dir / "consumer_facts.jsonl").write_text("{}\n") + input_sha = _sha(input_h5) + + seam = _load_module( + "calibrate_uk_national_dataset", + _ROOT / "tools" / "calibrate_uk_national_dataset.py", + ) + pin = _stub_inputs(seam, monkeypatch, registry=registry, ledger_dir=ledger_dir) + seam_out = tmp_path / "seam" + seam_out.mkdir() + assert ( + seam.main( + [ + "--input-h5", + str(input_h5), + "--input-sha256", + input_sha, + "--ledger-facts", + str(ledger_dir), + "--ledger-facts-sha256", + pin.facts_sha256, + "--ledger-manifest-sha256", + pin.manifest_sha256, + "--staging-h5", + str(seam_out / "microcosm_uk_2024_25.h5"), + "--diagnostics-json", + str(seam_out / "calibration_diagnostics.json"), + "--build-record-json", + str(seam_out / "build_record.json"), + "--terminal-gate-json", + str(seam_out / "microcosm_uk_2024_25.terminal_gates.json"), + "--release-id", + "dev-parity-twin", + "--epochs", + "5", + "--no-staging", + ] + ) + == 0 + ) + + builder = _CANDIDATE_TESTS._load_builder_module() + _stub_inputs(builder, monkeypatch, registry=registry, ledger_dir=ledger_dir) + role_out = tmp_path / "role" + assert ( + builder.main( + [ + "--input-h5", + str(input_h5), + "--release-role", + "national", + "--input-sha256", + input_sha, + "--ledger-facts", + str(ledger_dir), + "--ledger-facts-sha256", + pin.facts_sha256, + "--ledger-manifest-sha256", + pin.manifest_sha256, + "--out", + str(role_out), + "--epochs", + "5", + "--no-staging", + ] + ) + == 0 + ) + + np.testing.assert_array_equal( + _weights(seam_out / "microcosm_uk_2024_25.h5"), + _weights(role_out / "microcosm_uk_2024_25.h5"), + ) + assert _diagnostics_rows(seam_out / "calibration_diagnostics.json") == ( + _diagnostics_rows(role_out / "calibration_diagnostics.json") + ) + seam_record = json.loads((seam_out / "build_record.json").read_text()) + role_record = json.loads((role_out / "build_record.json").read_text()) + assert seam_record["build_id"] != role_record["build_id"] + assert ( + seam_record["build_id"].split("-")[:4] == role_record["build_id"].split("-")[:4] + ) + for key in ("calibration", "gate_summary", "register", "input_posture"): + assert seam_record[key] == role_record[key], key + assert seam_record["spine_provenance"] == role_record["spine_provenance"] + seam_config = dict(seam_record["run_config"]) + role_config = dict(role_record["run_config"]) + assert seam_config.pop("release_id") == "dev-parity-twin" + assert role_config.pop("release_id") == "microcosm-uk-2024-25-national" + assert role_config.pop("release_role") == "national" + assert role_config.pop("rowwise_driver_parameters")["release_role"] == "national" + assert seam_config == role_config + assert seam_config["doctrine_overrides"] == { + "epochs": {"default": 1500, "effective": 5} + } + assert seam_record["staging_delivery"] == role_record["staging_delivery"] + seam_gates = json.loads( + (seam_out / "microcosm_uk_2024_25.terminal_gates.json").read_text() + ) + role_gates = json.loads( + (role_out / "microcosm_uk_2024_25.terminal_gates.json").read_text() + ) + assert {g: e["status"] for g, e in seam_gates["gates"].items()} == { + g: e["status"] for g, e in role_gates["gates"].items() + } From eba0e15206ef1fdee199f317ccd95e617550dc83 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sun, 20 Sep 2026 17:52:18 +0100 Subject: [PATCH 05/11] Retire the UK national calibration seam command (#823) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tools/calibrate_uk_national_dataset.py and its test module are removed: the national line is built by tools/build_uk_rowwise_candidate.py --release-role national, which delegates to the same seam library and was shown bit-for-bit equal to the command on the synthetic parity twin (the previous commit; the twin retires here with the command). The command's behaviours re-anchor on the driver: no --release-id flag (canonical ids are the role's own), operator exclusions accepted on the staging posture, a bad input sha refused, the shared staging modes, the committed feed pin checked before the register compiles and a foreign feed refused (or recorded when --allow-unpinned-feed says so), and the one Chronicle pin shared with the local census (now in the chronicle feed tests). Docs: runbook 623 and runbook 806 §1–§3 name the driver's national role and its outputs (microcosm_uk_2024_25.h5 and siblings); the staging operations guide and the README name the two UK commands. Co-Authored-By: Claude Fable 5.1 --- README.md | 11 +- ...uk-retire-national-seam-cli-823.changed.md | 1 + docs/uk-national-calibration-runbook-623.md | 55 ++- ...k-national-release-assembly-runbook-806.md | 39 +- docs/uk-staging-operations.md | 6 +- .../tests/test_uk_calibration_seam_driver.py | 446 ------------------ .../tests/test_uk_chronicle_feed.py | 19 + .../tests/test_uk_rowwise_national_role.py | 140 ++++++ .../tests/test_uk_seam_parity_twin.py | 226 --------- tools/calibrate_uk_national_dataset.py | 417 ---------------- 10 files changed, 223 insertions(+), 1137 deletions(-) create mode 100644 changelog.d/uk-retire-national-seam-cli-823.changed.md delete mode 100644 packages/microcosm-build/tests/test_uk_calibration_seam_driver.py delete mode 100644 packages/microcosm-build/tests/test_uk_seam_parity_twin.py delete mode 100644 tools/calibrate_uk_national_dataset.py diff --git a/README.md b/README.md index 6fc241c31..105facc37 100644 --- a/README.md +++ b/README.md @@ -63,11 +63,12 @@ This writes `progress.json`, `events.ndjson`, `calibration_progress.json`, and final candidate diagnostics under `runs//` without updating production `latest.json`. -The UK commands (`tools/build_uk_frs_spine.py`, -`tools/calibrate_uk_national_dataset.py`, `tools/build_uk_rowwise_candidate.py`) -stage version 2 telemetry to `policyengine/populace-uk-staging` under the same -switch. The rowwise candidate command also **stages the finished dataset -bundle** it built, dense or exact-count, under `staged//` in the +The UK commands (`tools/build_uk_frs_spine.py` and +`tools/build_uk_rowwise_candidate.py`, whose `--release-role` builds either +the national or the dense line) stage version 2 telemetry to +`policyengine/populace-uk-staging` under the same switch. The rowwise +candidate command also **stages the finished dataset bundle** it built, +national, dense or exact-count, under `staged//` in the private `policyengine/populace-uk-private` repository so the team can inspect it without publishing it: `releases/` and `latest.json` are untouched, the release contract is not consulted, and a `releasable: false` size run stages diff --git a/changelog.d/uk-retire-national-seam-cli-823.changed.md b/changelog.d/uk-retire-national-seam-cli-823.changed.md new file mode 100644 index 000000000..928b57bb7 --- /dev/null +++ b/changelog.d/uk-retire-national-seam-cli-823.changed.md @@ -0,0 +1 @@ +`tools/calibrate_uk_national_dataset.py` is retired: the national line is built by `tools/build_uk_rowwise_candidate.py --release-role national`, which delegates to the same seam library (`run_uk_calibration`) and was shown bit-for-bit equal to the command on the synthetic parity twin (weights, diagnostics rows, build record). The runbooks, the staging operations guide and the README now name the one driver. diff --git a/docs/uk-national-calibration-runbook-623.md b/docs/uk-national-calibration-runbook-623.md index 81c940e16..8b7d68bb2 100644 --- a/docs/uk-national-calibration-runbook-623.md +++ b/docs/uk-national-calibration-runbook-623.md @@ -17,29 +17,39 @@ is the release-cut producer's job, not calibration's. ## Command shape -Calibration runs through `tools/calibrate_uk_national_dataset.py`. That driver -is the only one that builds the measure resolver and applies the committed +Calibration runs through the rowwise driver's national release role, +`tools/build_uk_rowwise_candidate.py --release-role national` (microcosm#823; +it replaced the retired `tools/calibrate_uk_national_dataset.py`). The role +delegates the build to the calibration seam library +(`uk_runtime.calibration_run.run_uk_calibration`): it is the only path that +builds the measure resolver from the input file and applies the committed measure-exclusion register, and 187 of the activated references bind model outputs that no frame carries — so it is the only path on which this target -surface materializes. The June builder -(`tools/build_uk_national_dataset.py`) constructs the calibration stage without -either and aborts on the first unmaterializable reference; it also rebuilds SPI -income onto its input, which a spine artifact already carries. +surface materializes. No cloning, no ladder, national targets only. The June +builder (`tools/build_uk_national_dataset.py`) constructs the calibration +stage without either and aborts on the first unmaterializable reference; it +also rebuilds SPI income onto its input, which a spine artifact already +carries. ```bash -uv run --no-sync python tools/calibrate_uk_national_dataset.py \ +uv run --no-sync python tools/build_uk_rowwise_candidate.py --release-role national \ --input-h5 data/ukds/acceptance/623-first-calibrated-candidate/input-spine.h5 \ --input-sha256 \ --ledger-facts \ --ledger-facts-sha256 \ --ledger-manifest-sha256 \ - --staging-h5 data/ukds/acceptance/623-first-calibrated-candidate/populace_uk_2023.h5 \ - --diagnostics-json data/ukds/acceptance/623-first-calibrated-candidate/calibration_diagnostics.json \ - --build-record-json data/ukds/acceptance/623-first-calibrated-candidate/build_record.json \ - --terminal-gate-json data/ukds/acceptance/623-first-calibrated-candidate/terminal_gates.json \ - --release-id dev-623-first-calibrated-candidate + --out data/ukds/acceptance/623-first-calibrated-candidate ``` +The role names every output after the pinned FRS release vintage: +`microcosm_uk_2024_25.h5`, `calibration_diagnostics.json`, +`build_record.json`, `microcosm_uk_2024_25.terminal_gates.json`, the frozen +`national_target_registry.json` (the scorer's register) and +`rowwise_candidate_manifest.json`; `--dry-run` compiles the register and +prints the plan without solving. Staging telemetry and the staged bundle +follow the dense role's switches (`--staging-local-only`, `--no-staging`, +`--staging-read-back`, `--no-staged-dataset`); the run id is the attempt id. + The diagnostics digest is measured, not declared: the seam writes the diagnostics file, hashes its actual bytes, and only then constructs and signs the terminal gate evidence. There is no `--calibration-diagnostics-sha256` to @@ -48,12 +58,12 @@ supply, and no way for the receipt to claim an identity the file does not have. Solve parameters are per-run overrides of the declared doctrine. Since the 2026-09-20 ruling (microcosm#823) the doctrine is the campaign posture itself: 1,500 epochs, `family_equal`, learning rate 0.02, seed 0, so a doctrine run -passes no solve flags and records no overrides; `--target-weight-rule uniform` -or another `--epochs` is validated through the doctrine dataclass and echoed -as an explicit deviation in the manifest, diagnostics and build record. -`--release-candidate` refuses every override flag, and refuses -`--measure-exclusions`, so a release candidate is always solved under declared -doctrine against the committed target surface. +passes no solve flags and records no overrides; `--target-weight-rule uniform`, +another `--epochs`, `--learning-rate` or `--target-loss-cap` is validated +through the doctrine dataclass and echoed as an explicit deviation in the +manifest, diagnostics and build record. The national role refuses +`--release-candidate` outright: its six-gate battery cannot sign +shippability, which comes only from `tools/certify_uk_release_cut.py`. Signing the terminal gate report needs `MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY` in the environment. @@ -65,11 +75,11 @@ identity never depends on the incumbent's bytes: ```bash uv run --no-sync python tools/score_uk_national_candidate.py \ - --candidate-h5 data/ukds/acceptance/623-first-calibrated-candidate/populace_uk_2023.h5 \ + --candidate-h5 data/ukds/acceptance/623-first-calibrated-candidate/microcosm_uk_2024_25.h5 \ --candidate-sha256 \ --incumbent-h5 \ --incumbent-sha256 \ - --registry-json data/ukds/acceptance/623-first-calibrated-candidate/frozen-register.json \ + --registry-json data/ukds/acceptance/623-first-calibrated-candidate/national_target_registry.json \ --output-json data/ukds/acceptance/623-first-calibrated-candidate/score_vs_enhanced_frs.json ``` @@ -80,9 +90,12 @@ and both sides are scored on the same frozen register. `data/ukds/acceptance/623-first-calibrated-candidate/` should contain: +- `microcosm_uk_2024_25.h5` - `build_record.json` -- `terminal_gates.json` +- `microcosm_uk_2024_25.terminal_gates.json` - `calibration_diagnostics.json` +- `national_target_registry.json` +- `rowwise_candidate_manifest.json` - `score_vs_enhanced_frs.json` - `logbook-spool/` (one row for the attempt, whatever its disposition) diff --git a/docs/uk-national-release-assembly-runbook-806.md b/docs/uk-national-release-assembly-runbook-806.md index 8a7ba5db7..7ebe82af6 100644 --- a/docs/uk-national-release-assembly-runbook-806.md +++ b/docs/uk-national-release-assembly-runbook-806.md @@ -27,25 +27,26 @@ reference together through the run. ## 1. Calibrate the national candidate -Use the national calibration driver and record the input digest rather than -relying on a mutable path: +Use the rowwise driver's national release role and record the input digest +rather than relying on a mutable path: ```bash -uv run --no-sync python tools/calibrate_uk_national_dataset.py \ +uv run --no-sync python tools/build_uk_rowwise_candidate.py --release-role national \ --input-h5 \ --input-sha256 \ --ledger-facts \ --ledger-facts-sha256 \ --ledger-manifest-sha256 \ - --staging-h5 /microcosm_uk_2024.h5 \ - --diagnostics-json /calibration_diagnostics.json \ - --build-record-json /build_record.json \ - --terminal-gate-json /terminal_gates.json \ - --release-id dev-uk-national-calibration + --out ``` -Use only the campaign doctrine overrides that were separately adjudicated. -The build record's id has the form +The role's doctrine is the campaign posture (1,500 epochs, `family_equal`, +learning rate 0.02, seed 0): a certified cut passes no solve flags and records +no overrides. It writes `microcosm_uk_2024_25.h5`, +`calibration_diagnostics.json`, `build_record.json`, +`microcosm_uk_2024_25.terminal_gates.json`, `national_target_registry.json` +and `rowwise_candidate_manifest.json` into ``. The build +record's id has the form `uk-frs-calibration-attempt--`; assembly derives the per-cut tag from that suffix. @@ -58,13 +59,13 @@ and compose the signed certification: ```bash uv run --no-sync python tools/certify_uk_release_cut.py \ - --candidate-h5 /microcosm_uk_2024.h5 \ + --candidate-h5 /microcosm_uk_2024_25.h5 \ --candidate-sha256 \ - --candidate-name microcosm_uk_2024 \ + --candidate-name microcosm_uk_2024_25 \ --spine-h5 \ --diagnostics-json /calibration_diagnostics.json \ --build-record-json /build_record.json \ - --seam-gate-report /terminal_gates.json \ + --seam-gate-report /microcosm_uk_2024_25.terminal_gates.json \ --ledger-facts \ --ledger-facts-sha256 \ --ledger-manifest-sha256 \ @@ -74,8 +75,8 @@ uv run --no-sync python tools/certify_uk_release_cut.py \ ``` With the default paths, this writes -`microcosm_uk_2024.release_cut_gates.json` and -`microcosm_uk_2024.release_certification.json` next to the candidate. Continue +`microcosm_uk_2024_25.release_cut_gates.json` and +`microcosm_uk_2024_25.release_certification.json` next to the candidate. Continue only when the certification says `shippable: true`. ## 3. Assemble the release directory @@ -86,13 +87,13 @@ and validates the finished directory: ```bash uv run --no-sync python tools/assemble_uk_release_dir.py \ - --candidate-h5 /microcosm_uk_2024.h5 \ + --candidate-h5 /microcosm_uk_2024_25.h5 \ --spine-h5 \ - --certification-json /microcosm_uk_2024.release_certification.json \ + --certification-json /microcosm_uk_2024_25.release_certification.json \ --build-record-json /build_record.json \ --diagnostics-json /calibration_diagnostics.json \ - --seam-gate-report /terminal_gates.json \ - --release-cut-gate-json /microcosm_uk_2024.release_cut_gates.json \ + --seam-gate-report /microcosm_uk_2024_25.terminal_gates.json \ + --release-cut-gate-json /microcosm_uk_2024_25.release_cut_gates.json \ --score-receipt /score_vs_enhanced_frs.json \ --out-dir releases ``` diff --git a/docs/uk-staging-operations.md b/docs/uk-staging-operations.md index 4f91a7495..5b7150625 100644 --- a/docs/uk-staging-operations.md +++ b/docs/uk-staging-operations.md @@ -115,9 +115,9 @@ read-only credential. ## Command modes and files -The three UK commands (`tools/build_uk_frs_spine.py`, -`tools/calibrate_uk_national_dataset.py` and -`tools/build_uk_rowwise_candidate.py`) support these staging modes: +The two UK commands (`tools/build_uk_frs_spine.py` and +`tools/build_uk_rowwise_candidate.py` in either release role, `national` or +`dense`) support these staging modes: - Default: local version 2 files plus best-effort delivery to `policyengine/populace-uk-staging`. diff --git a/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py b/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py deleted file mode 100644 index b0c5be921..000000000 --- a/packages/microcosm-build/tests/test_uk_calibration_seam_driver.py +++ /dev/null @@ -1,446 +0,0 @@ -from __future__ import annotations - -import importlib.util -import json -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from microcosm.build.staging_v2 import validate_v2_bundle -from microcosm.build.uk_runtime.ledger_targets import UKLedgerTargetCompilation -from microcosm.calibrate import TargetRegistry, TargetSpec - - -def _load_driver_module(): - root = Path(__file__).resolve().parents[3] - path = root / "tools" / "calibrate_uk_national_dataset.py" - spec = importlib.util.spec_from_file_location("calibrate_uk_national_dataset", path) - module = importlib.util.module_from_spec(spec) - assert spec.loader is not None - spec.loader.exec_module(module) - return module - - -def _registry(): - return TargetRegistry( - [ - TargetSpec( - name="dwp.uc.households", - entity="benunit", - measure="dwp/uc/households", - value=1.0, - source="test", - metadata={"contract_target_id": "dwp.uc.households"}, - ) - ], - country="uk", - ) - - -def _args(tmp_path: Path) -> list[str]: - paths = { - "input": tmp_path / "input.h5", - "ledger": tmp_path / "ledger", - "staging": tmp_path / "staging.h5", - "diagnostics": tmp_path / "diagnostics.json", - "record": tmp_path / "record.json", - } - paths["ledger"].mkdir(exist_ok=True) - for key, path in paths.items(): - if key != "ledger": - path.write_bytes(key.encode()) - return [ - "--input-h5", - str(paths["input"]), - "--input-sha256", - "a" * 64, - "--ledger-facts", - str(paths["ledger"]), - "--ledger-facts-sha256", - "b" * 64, - "--ledger-manifest-sha256", - "c" * 64, - "--staging-h5", - str(paths["staging"]), - "--diagnostics-json", - str(paths["diagnostics"]), - "--build-record-json", - str(paths["record"]), - "--release-id", - "dev-calibration", - "--no-staging", - ] - - -def test_driver_refuses_release_candidate_outright(tmp_path: Path): - # The seam's scoped battery covers 6 of the declared entries and must - # never sign a shippability claim (the #757 release-cut audit); the - # release verdict belongs to the release-cut certification producer. - driver = _load_driver_module() - with pytest.raises(SystemExit): - driver._parse_args(_args(tmp_path) + ["--release-candidate"]) - # The refusal is unconditional — an otherwise doctrine-clean invocation - # is refused too, not just ones with override flags. - with pytest.raises(SystemExit): - driver._parse_args(_args(tmp_path) + ["--release-candidate", "--epochs", "128"]) - - -def test_driver_refuses_canonical_release_ids(tmp_path: Path): - # Canonical release ids name shippable candidates; the seam runs under - # staging or dev ids only, and redirects canonical ids to the - # release-cut producer. - driver = _load_driver_module() - base = _args(tmp_path) - release_index = base.index("--release-id") - for canonical in ( - "populace-uk-2024-frs-k100", - "populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z", - ): - args = [*base] - args[release_index + 1] = canonical - with pytest.raises(SystemExit): - driver._parse_args(args) - - -def test_driver_accepts_operator_exclusions_on_staging_posture(tmp_path: Path): - driver = _load_driver_module() - exclusions = tmp_path / "operator.json" - exclusions.write_text("{}", encoding="utf-8") - parsed = driver._parse_args( - _args(tmp_path) + ["--measure-exclusions", str(exclusions)] - ) - assert parsed.measure_exclusions == exclusions - - -def test_driver_refuses_bad_sha_and_path_alias(tmp_path: Path): - driver = _load_driver_module() - base_args = _args(tmp_path) - with pytest.raises(SystemExit): - driver._parse_args([*base_args[:3], "not-a-sha", *base_args[4:]]) - - args = _args(tmp_path) - staging_index = args.index("--staging-h5") + 1 - args[staging_index] = args[args.index("--input-h5") + 1] - with pytest.raises(SystemExit, match="distinct paths"): - driver._parse_args(args) - - -def test_driver_refuses_feed_outside_the_committed_pin_without_override(): - driver = _load_driver_module() - - with pytest.raises(SystemExit, match="committed UK national feed pin"): - driver._check_committed_ledger_feed_pin( - "b" * 64, - manifest_sha256=driver.load_uk_chronicle_feed().manifest_sha256, - allow_unpinned_feed=False, - ) - - driver._check_committed_ledger_feed_pin( - "b" * 64, - manifest_sha256="c" * 64, - allow_unpinned_feed=True, - ) - - -@pytest.mark.parametrize("allow_unpinned_feed", [False, True]) -def test_driver_threads_registry_exclusions_resolver_and_overrides( - monkeypatch, - tmp_path, - capsys, - allow_unpinned_feed, -): - driver = _load_driver_module() - calls = [] - registry = _registry() - pruned_registry = TargetRegistry([], country="uk") - pin = driver.load_uk_chronicle_feed() - artifact = SimpleNamespace( - path=tmp_path / "ledger", - facts=({"fact": 1},), - facts_sha256="b" * 64 if allow_unpinned_feed else pin.facts_sha256, - manifest_sha256="c" * 64 if allow_unpinned_feed else pin.manifest_sha256, - ) - artifact.path.mkdir() - (artifact.path / "consumer_facts.jsonl").write_text("{}", encoding="utf-8") - monkeypatch.setattr( - driver, "load_ledger_consumer_artifact", lambda *a, **k: artifact - ) - monkeypatch.setattr( - driver, - "compile_uk_target_registry", - lambda facts, target_period: UKLedgerTargetCompilation(registry, ()), - ) - monkeypatch.setattr( - driver, - "load_uk_frs_release", - lambda: SimpleNamespace(calibration_year=2025), - ) - monkeypatch.setattr( - driver, "load_uk_calibration_measure_exclusions", lambda path: () - ) - monkeypatch.setattr( - driver, - "apply_uk_calibration_measure_exclusions", - lambda reg, exclusions: (pruned_registry, {"excluded": "reviewed"}), - ) - - class FakeResolver: - def __init__(self, **kwargs): - self.kwargs = kwargs - - monkeypatch.setattr(driver, "UKMeasureResolver", FakeResolver) - - def fake_run(**kwargs): - calls.append(kwargs) - return SimpleNamespace( - staging_sha256="1" * 64, - diagnostics_sha256="2" * 64, - terminal_gate_sha256="3" * 64, - build_record_sha256="4" * 64, - build_record={"gate_summary": {"uk_target_fit": "passed"}}, - ) - - monkeypatch.setattr(driver, "run_uk_calibration", fake_run) - - argv = _args(tmp_path) - # The driver verifies the input pin before the resolver reads the H5, so - # the threading test must pin the fixture file's real digest. - argv[argv.index("--input-sha256") + 1] = driver._sha256_file( - Path(argv[argv.index("--input-h5") + 1]) - ) - extra = ["--allow-unpinned-feed"] if allow_unpinned_feed else [] - result = driver.main(argv + ["--epochs", "128", *extra]) - - assert result == 0 - call = calls[0] - assert call["register_registry"] is pruned_registry - assert call["band_edge_registry"] is registry - assert call["calibration_year"] == 2025 - assert call["exclusion_receipt"] == {"excluded": "reviewed"} - assert call["doctrine"].epochs == 128 - assert call["doctrine_overrides"] == {"epochs": {"default": 1500, "effective": 128}} - assert isinstance(call["measure_resolver"], FakeResolver) - assert ( - call["measure_resolver"].kwargs["simulation_source"] == call["paths"].input_h5 - ) - assert call["source_pins"]["ledger_facts"] == { - "sha256": artifact.facts_sha256, - "size_bytes": 2, - } - assert call["run_config_extra"] == { - "calibration_year": 2025, - "allow_unpinned_feed": allow_unpinned_feed, - "chronicle_feed_pin": pin.to_dict(), - } - assert call["progress_callback"] is None - assert call["event_callback"] is None - assert call["staging_finalizer"] is None - assert call["staging_delivery_provider"] is None - assert call["staging_delivery"] == { - "contract_version": 2, - "enabled": False, - "mode": "disabled", - "run_id": None, - "configured_repository": None, - "upload_attempts": 0, - "upload_successes": 0, - "read_back": "not_requested", - "last_error_code": None, - "opt_out_reason": "--no-staging", - } - assert "uk_target_fit" in capsys.readouterr().out - - -def test_driver_exposes_shared_staging_modes(tmp_path: Path) -> None: - driver = _load_driver_module() - base = _args(tmp_path) - without_disabled = base[: base.index("--no-staging")] - - remote = driver._parse_args(without_disabled) - local = driver._parse_args([*without_disabled, "--staging-local-only"]) - - assert remote.staging_repo_id == "policyengine/populace-uk-staging" - assert not remote.staging_local_only and not remote.no_staging - assert local.staging_local_only and not local.no_staging - with pytest.raises(SystemExit): - driver._parse_args([*without_disabled, "--staging-repo-id", ""]) - with pytest.raises(SystemExit): - driver._parse_args( - [*without_disabled, "--staging-local-only", "--staging-read-back"] - ) - - -def test_driver_records_local_calibration_stage_coverage( - monkeypatch, tmp_path: Path -) -> None: - driver = _load_driver_module() - registry = _registry() - pin = driver.load_uk_chronicle_feed() - artifact = SimpleNamespace( - path=tmp_path / "ledger", - facts=({"fact": 1},), - facts_sha256=pin.facts_sha256, - manifest_sha256=pin.manifest_sha256, - ) - artifact.path.mkdir() - (artifact.path / "consumer_facts.jsonl").write_text("{}", encoding="utf-8") - monkeypatch.setattr( - driver, "load_ledger_consumer_artifact", lambda *args, **kwargs: artifact - ) - monkeypatch.setattr( - driver, - "compile_uk_target_registry", - lambda facts, target_period: UKLedgerTargetCompilation(registry, ()), - ) - monkeypatch.setattr( - driver, "load_uk_frs_release", lambda: SimpleNamespace(calibration_year=2025) - ) - monkeypatch.setattr( - driver, "load_uk_calibration_measure_exclusions", lambda path: () - ) - monkeypatch.setattr( - driver, - "apply_uk_calibration_measure_exclusions", - lambda active, exclusions: (active, {}), - ) - monkeypatch.setattr(driver, "UKMeasureResolver", lambda **kwargs: object()) - - def fake_run(**kwargs): - callback = kwargs["event_callback"] - for stage_id in ( - "input_loading", - "measure_resolution", - "calibration", - "diagnostics", - "release_check_evaluation", - "candidate_h5_creation", - "build_record_creation", - ): - callback(stage_id, "started", {}) - callback(stage_id, "completed", {"aggregate_count": 1}) - kwargs["progress_callback"]( - { - "kind": "calibration_epoch", - "epoch": 1, - "epochs": 2, - "phase": "solve", - "loss": 1.5, - "iteration": 1, - } - ) - delivery = kwargs["staging_delivery"] - if kwargs["staging_finalizer"] is not None: - kwargs["staging_finalizer"]() - delivery = kwargs["staging_delivery_provider"]() - record = { - "gate_summary": {"uk_target_fit": "passed"}, - "staging_delivery": delivery, - } - driver._write_json(kwargs["paths"].build_record_json, record) - return SimpleNamespace( - staging_sha256="1" * 64, - diagnostics_sha256="2" * 64, - terminal_gate_sha256="3" * 64, - build_record_sha256=driver._sha256_file(kwargs["paths"].build_record_json), - build_record=record, - ) - - monkeypatch.setattr(driver, "run_uk_calibration", fake_run) - argv = _args(tmp_path) - argv = argv[: argv.index("--no-staging")] - input_path = Path(argv[argv.index("--input-h5") + 1]) - argv[argv.index("--input-sha256") + 1] = driver._sha256_file(input_path) - staging_root = tmp_path / "telemetry" - argv.extend( - [ - "--staging-local-only", - "--staging-dir", - str(staging_root), - "--staging-run-id", - "calibration-stage-test", - ] - ) - - assert driver.main(argv) == 0 - - bundle = validate_v2_bundle(staging_root, "calibration-stage-test") - assert bundle["calibration_progress"]["events"][0]["epoch"] == 1 - completed = { - event["stage_id"] - for event in bundle["events"] - if event["status"] == "completed" - } - assert { - "target_compilation", - "input_loading", - "measure_resolution", - "calibration", - "diagnostics", - "release_check_evaluation", - "candidate_h5_creation", - "build_record_creation", - "complete", - } <= completed - record_path = Path(argv[argv.index("--build-record-json") + 1]) - record = json.loads(record_path.read_text()) - assert record["staging_delivery"]["mode"] == "local_only" - - -def test_driver_refuses_the_national_release_id(tmp_path: Path): - # The constant national id names a shippable release (ruling 2026-08-27); - # the seam runs under staging or dev ids only. - driver = _load_driver_module() - base = _args(tmp_path) - args = [*base] - args[base.index("--release-id") + 1] = "microcosm-uk-2024-25-national" - with pytest.raises(SystemExit): - driver._parse_args(args) - - -def test_driver_and_local_census_read_the_one_chronicle_pin(): - from microcosm.build.uk_runtime.local_target_census import _LEDGER_FACT_FEED_PIN - - driver = _load_driver_module() - pin = driver.load_uk_chronicle_feed() - driver._check_committed_ledger_feed_pin( - pin.facts_sha256, - manifest_sha256=pin.manifest_sha256, - allow_unpinned_feed=False, - ) - # The local census restates the same declaration (microcosm#890 review: - # the national and local surfaces share uk/chronicle_feed.json, so a - # re-pin is one reviewed change and the two cannot drift apart). - assert _LEDGER_FACT_FEED_PIN["facts_sha256"] == pin.facts_sha256 - assert _LEDGER_FACT_FEED_PIN["manifest_sha256"] == pin.manifest_sha256 - assert _LEDGER_FACT_FEED_PIN["source_commit"] == pin.source_commit - assert _LEDGER_FACT_FEED_PIN["fact_row_count"] == pin.fact_row_count - - -@pytest.mark.parametrize("manifest_sha256", ["c" * 64, None]) -def test_driver_refuses_unpinned_national_manifest(manifest_sha256): - driver = _load_driver_module() - with pytest.raises(SystemExit, match="manifest"): - driver._check_committed_ledger_feed_pin( - driver.load_uk_chronicle_feed().facts_sha256, - manifest_sha256=manifest_sha256, - allow_unpinned_feed=False, - ) - - -def test_driver_checks_loaded_manifest_before_compiling_targets(monkeypatch, tmp_path): - driver = _load_driver_module() - pin = driver.load_uk_chronicle_feed() - artifact = SimpleNamespace(facts_sha256=pin.facts_sha256, manifest_sha256="c" * 64) - monkeypatch.setattr( - driver, "load_ledger_consumer_artifact", lambda *a, **k: artifact - ) - monkeypatch.setattr( - driver, - "compile_uk_target_registry", - lambda *a, **k: pytest.fail( - "Targets must not compile from an unpinned artifact" - ), - ) - with pytest.raises(SystemExit, match="manifest"): - driver.main(_args(tmp_path)) diff --git a/packages/microcosm-build/tests/test_uk_chronicle_feed.py b/packages/microcosm-build/tests/test_uk_chronicle_feed.py index dcf5e4096..771ff4486 100644 --- a/packages/microcosm-build/tests/test_uk_chronicle_feed.py +++ b/packages/microcosm-build/tests/test_uk_chronicle_feed.py @@ -96,3 +96,22 @@ def test_committed_feed_pin_check_refuses_a_foreign_artifact(): ) is pin ) + + +def test_national_and_local_census_read_the_one_chronicle_pin(): + """The national and local surfaces share uk/chronicle_feed.json (#890).""" + + from microcosm.build.uk_runtime.chronicle_feed import ( + load_uk_chronicle_feed, + require_committed_uk_chronicle_feed_pin, + ) + from microcosm.build.uk_runtime.local_target_census import _LEDGER_FACT_FEED_PIN + + pin = load_uk_chronicle_feed() + require_committed_uk_chronicle_feed_pin( + pin.facts_sha256, manifest_sha256=pin.manifest_sha256, allow_unpinned_feed=False + ) + assert _LEDGER_FACT_FEED_PIN["facts_sha256"] == pin.facts_sha256 + assert _LEDGER_FACT_FEED_PIN["manifest_sha256"] == pin.manifest_sha256 + assert _LEDGER_FACT_FEED_PIN["source_commit"] == pin.source_commit + assert _LEDGER_FACT_FEED_PIN["fact_row_count"] == pin.fact_row_count diff --git a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py index 8242daed5..d24c44d4a 100644 --- a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py +++ b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py @@ -20,6 +20,11 @@ from microcosm.build.staging_v2 import validate_v2_bundle from microcosm.build.uk_runtime import calibration_run from microcosm.build.uk_runtime.calibration_run import UK_CALIBRATION_GATE_SCOPE +from microcosm.build.uk_runtime.chronicle_feed import ( + UKChronicleFeedPinError, + load_uk_chronicle_feed, + require_committed_uk_chronicle_feed_pin, +) from microcosm.build.uk_runtime.national_frame import write_uk_national_frame from microcosm.build.uk_runtime.release_identity import UK_NATIONAL_RELEASE_ID from microcosm.calibrate import TargetRegistry @@ -507,3 +512,138 @@ def test_uk_national_role_marks_the_staging_run_failed_on_a_refusal( bundle = validate_v2_bundle(out / "staging", runs[0]) assert bundle["progress"]["status"] == "failed" assert not (out / "build_record.json").exists() + + +# --- Behaviours re-anchored from the retired seam command's tests (B3) --- + + +def test_uk_national_role_has_no_release_id_flag(tmp_path) -> None: + """Canonical ids are the role's own; the seam's --release-id is gone.""" + + builder = _CANDIDATE._load_builder_module() + (tmp_path / "spine.h5").write_bytes(b"spine") + with pytest.raises(SystemExit): + builder._parse_args( + _argv(tmp_path / "spine.h5", tmp_path / "out", "--release-id", "dev-x") + ) + + +def test_uk_national_role_accepts_operator_exclusions_and_refuses_a_bad_sha( + tmp_path, +) -> None: + builder = _CANDIDATE._load_builder_module() + (tmp_path / "spine.h5").write_bytes(b"spine") + exclusions = tmp_path / "operator.json" + exclusions.write_text("{}", encoding="utf-8") + argv = _argv(tmp_path / "spine.h5", tmp_path / "out") + argv[argv.index("--input-sha256") + 1] = "0" * 64 + parsed = builder._parse_args([*argv, "--measure-exclusions", str(exclusions)]) + builder._validate_cli_args(parsed) + assert parsed.measure_exclusions == exclusions + argv[argv.index("--input-sha256") + 1] = "not-a-sha" + with pytest.raises(SystemExit): + builder._parse_args(argv) + + +def test_uk_national_role_exposes_the_shared_staging_modes(tmp_path) -> None: + builder = _CANDIDATE._load_builder_module() + (tmp_path / "spine.h5").write_bytes(b"spine") + argv = _argv(tmp_path / "spine.h5", tmp_path / "out") + argv[argv.index("--input-sha256") + 1] = "0" * 64 + remote = builder._parse_args(argv) + local = builder._parse_args([*argv, "--staging-local-only"]) + disabled = builder._parse_args([*argv, "--no-staging"]) + assert remote.staging_repo_id == "policyengine/populace-uk-staging" + assert not remote.staging_local_only and not remote.no_staging + assert local.staging_local_only and not local.no_staging + assert disabled.no_staging + with pytest.raises(SystemExit): + builder._parse_args([*argv, "--staging-repo-id", ""]) + with pytest.raises(SystemExit): + builder._parse_args([*argv, "--staging-local-only", "--staging-read-back"]) + + +def _foreign_artifact(ledger_dir: Path, *, facts_sha256: str, manifest_sha256): + return SimpleNamespace( + facts=None, + facts_sha256=facts_sha256, + manifest_sha256=manifest_sha256, + path=ledger_dir, + provenance=lambda: { + "facts_sha256": facts_sha256, + "manifest_sha256": manifest_sha256, + "artifact_id": "foreign", + }, + ) + + +@pytest.mark.parametrize("manifest_sha256", ["c" * 64, None]) +def test_uk_national_role_refuses_a_feed_outside_the_committed_pin( + monkeypatch, tmp_path, manifest_sha256 +) -> None: + """The committed pin is checked before the register compiles.""" + + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + # The fixture stubs the check; this test wants the real one. + monkeypatch.setattr( + builder, + "require_committed_uk_chronicle_feed_pin", + require_committed_uk_chronicle_feed_pin, + ) + pin = load_uk_chronicle_feed() + artifact = _foreign_artifact( + tmp_path / "ledger", + facts_sha256=pin.facts_sha256, + manifest_sha256=manifest_sha256, + ) + monkeypatch.setattr( + builder, "load_ledger_consumer_artifact", lambda path, **kwargs: artifact + ) + + def compile_must_not_run(facts, target_period): + raise AssertionError("the register compiled before the feed pin was checked") + + monkeypatch.setattr(builder, "compile_uk_target_registry", compile_must_not_run) + out = tmp_path / "national" + with pytest.raises(UKChronicleFeedPinError, match="manifest: loaded"): + builder.main(_argv(input_h5, out, "--staging-local-only")) + # The refusal happened after telemetry opened: the local run reads failed. + runs = sorted(path.name for path in (out / "staging" / "runs").iterdir()) + assert ( + validate_v2_bundle(out / "staging", runs[0])["progress"]["status"] == "failed" + ) + + +def test_uk_national_role_records_an_unpinned_feed_override( + monkeypatch, tmp_path +) -> None: + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path, resolver=_FakeResolver + ) + monkeypatch.setattr( + builder, + "require_committed_uk_chronicle_feed_pin", + require_committed_uk_chronicle_feed_pin, + ) + artifact = _foreign_artifact( + tmp_path / "ledger", facts_sha256="0" * 64, manifest_sha256="c" * 64 + ) + monkeypatch.setattr( + builder, "load_ledger_consumer_artifact", lambda path, **kwargs: artifact + ) + calls: list[dict] = [] + monkeypatch.setattr(builder, "run_uk_calibration", _fake_seam_run(calls)) + out = tmp_path / "national" + assert ( + builder.main(_argv(input_h5, out, "--no-staging", "--allow-unpinned-feed")) == 0 + ) + extra = calls[0]["run_config_extra"] + assert extra["allow_unpinned_feed"] is True + # The pin the artifact was checked against is the committed one. + assert extra["chronicle_feed_pin"]["facts_sha256"] != "0" * 64 diff --git a/packages/microcosm-build/tests/test_uk_seam_parity_twin.py b/packages/microcosm-build/tests/test_uk_seam_parity_twin.py deleted file mode 100644 index e6ba29d20..000000000 --- a/packages/microcosm-build/tests/test_uk_seam_parity_twin.py +++ /dev/null @@ -1,226 +0,0 @@ -"""Parity twin: the retiring seam command and the driver's national role. - -Both paths call ``run_uk_calibration`` with the same doctrine on the same -synthetic frame and register (seed 0, five epochs, no staging), so the -calibrated household weights, the diagnostics rows and the seam-shaped build -record must agree exactly. The receipt is what licenses retiring -``tools/calibrate_uk_national_dataset.py`` (microcosm#823 B3); this module -retires with it. -""" - -from __future__ import annotations - -import hashlib -import importlib.util -import json -from pathlib import Path -from types import SimpleNamespace - -import numpy as np -import pytest - -from microcosm.build.uk_runtime import calibration_run -from microcosm.build.uk_runtime.ledger_targets import UKLedgerTargetCompilation -from microcosm.build.uk_runtime.national_frame import ( - load_uk_national_frame, - write_uk_national_frame, -) - -_TESTS = Path(__file__).resolve().parent -_ROOT = _TESTS.parents[2] - - -def _load_module(name: str, path: Path): - spec = importlib.util.spec_from_file_location(name, path) - module = importlib.util.module_from_spec(spec) - assert spec.loader is not None - spec.loader.exec_module(module) - return module - - -_SEAM_TESTS = _load_module( - "uk_calibration_run_fixtures", _TESTS / "test_uk_calibration_run.py" -) -_CANDIDATE_TESTS = _load_module( - "uk_rowwise_candidate_fixtures", _TESTS / "test_uk_rowwise_candidate.py" -) - - -def _sha(path: Path) -> str: - return hashlib.sha256(path.read_bytes()).hexdigest() - - -def _stub_inputs(module, monkeypatch, *, registry, ledger_dir: Path): - from microcosm.build.uk_runtime.chronicle_feed import load_uk_chronicle_feed - - pin = load_uk_chronicle_feed() - artifact = SimpleNamespace( - facts=None, - facts_sha256=pin.facts_sha256, - manifest_sha256=pin.manifest_sha256, - path=ledger_dir, - provenance=lambda: { - "facts_sha256": pin.facts_sha256, - "manifest_sha256": pin.manifest_sha256, - "artifact_id": "synthetic-parity-fixture", - }, - ) - monkeypatch.setattr( - module, "load_ledger_consumer_artifact", lambda *args, **kwargs: artifact - ) - monkeypatch.setattr( - module, - "compile_uk_target_registry", - lambda facts, target_period: UKLedgerTargetCompilation(registry, ()), - ) - monkeypatch.setattr(module, "load_uk_calibration_measure_exclusions", lambda p: ()) - monkeypatch.setattr( - module, - "apply_uk_calibration_measure_exclusions", - lambda reg, exclusions: (reg, {}), - ) - monkeypatch.setattr(module, "UKMeasureResolver", lambda **kwargs: None) - return pin - - -def _weights(path: Path) -> np.ndarray: - frame, _provenance = load_uk_national_frame(path) - return np.asarray(frame.weights_for("household").values, dtype=np.float64) - - -def _diagnostics_rows(path: Path) -> list[tuple]: - payload = json.loads(path.read_text()) - return [ - ( - row["name"], - row["target"], - row["initial_estimate"], - row["final_estimate"], - row["relative_error"], - ) - for row in payload["targets"] - ] - - -def test_uk_seam_command_and_national_role_agree_bit_for_bit(monkeypatch, tmp_path): - pytest.importorskip("tables") - monkeypatch.setenv( - "MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY", _SEAM_TESTS.SIGNING_KEY - ) - monkeypatch.setattr( - calibration_run, - "uk_aggregate_admin_totals", - lambda frame, manifest: (_SEAM_TESTS._admin_anchor_values(), []), - ) - frame = _SEAM_TESTS._frame() - input_h5 = tmp_path / "spine.h5" - write_uk_national_frame(frame, input_h5) - _SEAM_TESTS._write_spine_sidecar(input_h5, frame) - registry = _SEAM_TESTS._registry() - ledger_dir = tmp_path / "ledger" - ledger_dir.mkdir() - (ledger_dir / "consumer_facts.jsonl").write_text("{}\n") - input_sha = _sha(input_h5) - - seam = _load_module( - "calibrate_uk_national_dataset", - _ROOT / "tools" / "calibrate_uk_national_dataset.py", - ) - pin = _stub_inputs(seam, monkeypatch, registry=registry, ledger_dir=ledger_dir) - seam_out = tmp_path / "seam" - seam_out.mkdir() - assert ( - seam.main( - [ - "--input-h5", - str(input_h5), - "--input-sha256", - input_sha, - "--ledger-facts", - str(ledger_dir), - "--ledger-facts-sha256", - pin.facts_sha256, - "--ledger-manifest-sha256", - pin.manifest_sha256, - "--staging-h5", - str(seam_out / "microcosm_uk_2024_25.h5"), - "--diagnostics-json", - str(seam_out / "calibration_diagnostics.json"), - "--build-record-json", - str(seam_out / "build_record.json"), - "--terminal-gate-json", - str(seam_out / "microcosm_uk_2024_25.terminal_gates.json"), - "--release-id", - "dev-parity-twin", - "--epochs", - "5", - "--no-staging", - ] - ) - == 0 - ) - - builder = _CANDIDATE_TESTS._load_builder_module() - _stub_inputs(builder, monkeypatch, registry=registry, ledger_dir=ledger_dir) - role_out = tmp_path / "role" - assert ( - builder.main( - [ - "--input-h5", - str(input_h5), - "--release-role", - "national", - "--input-sha256", - input_sha, - "--ledger-facts", - str(ledger_dir), - "--ledger-facts-sha256", - pin.facts_sha256, - "--ledger-manifest-sha256", - pin.manifest_sha256, - "--out", - str(role_out), - "--epochs", - "5", - "--no-staging", - ] - ) - == 0 - ) - - np.testing.assert_array_equal( - _weights(seam_out / "microcosm_uk_2024_25.h5"), - _weights(role_out / "microcosm_uk_2024_25.h5"), - ) - assert _diagnostics_rows(seam_out / "calibration_diagnostics.json") == ( - _diagnostics_rows(role_out / "calibration_diagnostics.json") - ) - seam_record = json.loads((seam_out / "build_record.json").read_text()) - role_record = json.loads((role_out / "build_record.json").read_text()) - assert seam_record["build_id"] != role_record["build_id"] - assert ( - seam_record["build_id"].split("-")[:4] == role_record["build_id"].split("-")[:4] - ) - for key in ("calibration", "gate_summary", "register", "input_posture"): - assert seam_record[key] == role_record[key], key - assert seam_record["spine_provenance"] == role_record["spine_provenance"] - seam_config = dict(seam_record["run_config"]) - role_config = dict(role_record["run_config"]) - assert seam_config.pop("release_id") == "dev-parity-twin" - assert role_config.pop("release_id") == "microcosm-uk-2024-25-national" - assert role_config.pop("release_role") == "national" - assert role_config.pop("rowwise_driver_parameters")["release_role"] == "national" - assert seam_config == role_config - assert seam_config["doctrine_overrides"] == { - "epochs": {"default": 1500, "effective": 5} - } - assert seam_record["staging_delivery"] == role_record["staging_delivery"] - seam_gates = json.loads( - (seam_out / "microcosm_uk_2024_25.terminal_gates.json").read_text() - ) - role_gates = json.loads( - (role_out / "microcosm_uk_2024_25.terminal_gates.json").read_text() - ) - assert {g: e["status"] for g, e in seam_gates["gates"].items()} == { - g: e["status"] for g, e in role_gates["gates"].items() - } diff --git a/tools/calibrate_uk_national_dataset.py b/tools/calibrate_uk_national_dataset.py deleted file mode 100644 index 0cdda8d05..000000000 --- a/tools/calibrate_uk_national_dataset.py +++ /dev/null @@ -1,417 +0,0 @@ -"""Calibrate an existing UK national spine H5 without replaying source stages. - -This driver is intentionally thin: it verifies pinned inputs, compiles the -Ledger target registry, applies the reviewed measure-exclusion register, records -any explicit doctrine overrides, and delegates the calibration/gate/logbook -work to :func:`microcosm.build.uk_runtime.calibration_run.run_uk_calibration`. - -Signed deviation for v1: no sampling rungs and no checkpointing. This seam runs -full-scale only; scale ladders and resumable source-stage checkpoints belong to -the spine build lane. -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import re -from importlib import metadata -from itertools import combinations -from pathlib import Path -from typing import Any - -from microcosm.build.ledger_artifact import load_ledger_consumer_artifact -from microcosm.build.staging_cli import ( - add_staging_arguments, - validate_staging_arguments, -) -from microcosm.build.staging_v2 import ( - StagingTelemetryV2, - disabled_staging_delivery, -) -from microcosm.build.uk_runtime.calibration_run import ( - UKCalibrationRunPaths, - run_uk_calibration, -) -from microcosm.build.uk_runtime.chronicle_feed import ( - UKChronicleFeed, - UKChronicleFeedPinError, - load_uk_chronicle_feed, - require_committed_uk_chronicle_feed_pin, -) -from microcosm.build.uk_runtime.frs_release import load_uk_frs_release -from microcosm.build.uk_runtime.ledger_targets import compile_uk_target_registry -from microcosm.build.uk_runtime.measure_simulation import ( - UKMeasureResolver, - apply_uk_calibration_measure_exclusions, - load_uk_calibration_measure_exclusions, -) -from microcosm.build.uk_runtime.national_doctrine import uk_doctrine_with_overrides -from microcosm.build.uk_runtime.release_identity import UK_NATIONAL_RELEASE_ID -from microcosm.build.uk_runtime.staging import UK_STAGING_REPOSITORY -from microcosm.calibrate import TargetRegistry - -_SHA256 = re.compile(r"[0-9a-f]{64}") -_CANONICAL_UK_RELEASE_ID = re.compile( - r"populace-uk-[1-9][0-9]*-[a-z0-9_]+-k[1-9][0-9]*" -) -_UK_JUNE_RELEASE_ID = "populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z" - - -def main(argv: list[str] | None = None) -> int: - args = _parse_args(argv) - _validate_input_paths(args) - national_feed = load_uk_chronicle_feed() - telemetry = _create_staging_telemetry(args) - try: - if telemetry is not None: - telemetry.set_sample(_full_sample()) - telemetry.stage("target_compilation", event_status="started") - artifact = load_ledger_consumer_artifact( - args.ledger_facts, - expected_facts_sha256=args.ledger_facts_sha256, - expected_manifest_sha256=args.ledger_manifest_sha256, - ) - _check_committed_ledger_feed_pin( - artifact.facts_sha256, - manifest_sha256=artifact.manifest_sha256, - pin=national_feed, - allow_unpinned_feed=args.allow_unpinned_feed, - ) - calibration_year = load_uk_frs_release().calibration_year - compilation = compile_uk_target_registry( - artifact.facts, target_period=calibration_year - ) - if compilation.unsupported: - raise SystemExit( - f"{len(compilation.unsupported)} target references failed to compile" - ) - _compare_frozen_register(args.register_json, compilation.registry) - exclusions = load_uk_calibration_measure_exclusions(args.measure_exclusions) - registry, exclusion_receipt = apply_uk_calibration_measure_exclusions( - compilation.registry, exclusions - ) - if telemetry is not None: - telemetry.stage( - "target_compilation", - event_status="completed", - compiled_target_count=len(compilation.registry.specs), - active_target_count=len(registry.specs), - ) - overrides = { - key: value - for key, value in { - "epochs": args.epochs, - "target_weight_rule": args.target_weight_rule, - "learning_rate": args.learning_rate, - "target_loss_cap": args.target_loss_cap, - }.items() - if value is not None - } - doctrine, doctrine_overrides = uk_doctrine_with_overrides(**overrides) - paths = UKCalibrationRunPaths( - input_h5=args.input_h5, - staging_h5=args.staging_h5, - diagnostics_json=args.diagnostics_json, - build_record_json=args.build_record_json, - terminal_gate_json=args.terminal_gate_json, - ) - # The resolver reads the input H5 to build its simulation, so the pin is - # verified here first — no bytes are consumed before they match the CLI sha. - measured_input_sha = _sha256_file(args.input_h5) - if measured_input_sha != args.input_sha256: - raise SystemExit( - "error: --input-h5 sha mismatch: " - f"measured {measured_input_sha}, pinned {args.input_sha256}" - ) - resolver = UKMeasureResolver( - simulation_source=args.input_h5, - scratch_dir=args.staging_h5.parent, - year=calibration_year, - frame=None, - ) - - def event_callback(stage_id: str, status: str, details) -> None: - if telemetry is not None: - telemetry.stage(stage_id, event_status=status, **dict(details)) - - def finalize_staging() -> None: - assert telemetry is not None - telemetry.complete(message="UK calibration staging run completed.") - try: - if args.staging_read_back: - telemetry.verify_remote() - finally: - telemetry.validate_local_bundle() - - result = run_uk_calibration( - paths=paths, - input_sha256=args.input_sha256, - ledger_artifact=artifact, - register_registry=registry, - band_edge_registry=compilation.registry, - calibration_year=calibration_year, - exclusion_receipt=exclusion_receipt, - doctrine=doctrine, - doctrine_overrides=doctrine_overrides, - measure_resolver=resolver, - source_pins={ - "input_h5": { - "sha256": args.input_sha256, - "size_bytes": args.input_h5.stat().st_size, - }, - "ledger_facts": _ledger_facts_pin(artifact), - }, - run_config_extra={ - "calibration_year": calibration_year, - "allow_unpinned_feed": args.allow_unpinned_feed, - "chronicle_feed_pin": national_feed.to_dict(), - }, - release_id=args.release_id, - logbook_prev_row_digest=args.logbook_prev_row_digest, - progress_callback=( - telemetry.calibration_progress if telemetry is not None else None - ), - event_callback=event_callback if telemetry is not None else None, - staging_delivery=_staging_delivery(args, telemetry), - staging_finalizer=finalize_staging if telemetry is not None else None, - staging_delivery_provider=( - (lambda: telemetry.delivery_summary) if telemetry is not None else None - ), - ) - build_record_sha256 = result.build_record_sha256 - build_record = dict(result.build_record) - summary = { - "staging_h5_sha256": result.staging_sha256, - "diagnostics_sha256": result.diagnostics_sha256, - "terminal_gate_sha256": result.terminal_gate_sha256, - "build_record_sha256": build_record_sha256, - "gate_verdicts": build_record["gate_summary"], - } - print(json.dumps(summary, indent=2, sort_keys=True)) - return 0 - except BaseException as error: - if telemetry is not None and telemetry.status == "running": - try: - telemetry.fail(error) - telemetry.validate_local_bundle() - except Exception: - pass - raise - - -def _parse_args(argv: list[str] | None) -> argparse.Namespace: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("--input-h5", required=True, type=Path) - parser.add_argument("--input-sha256", required=True, type=_sha256) - parser.add_argument("--ledger-facts", required=True, type=Path) - parser.add_argument("--ledger-facts-sha256", required=True, type=_sha256) - parser.add_argument("--ledger-manifest-sha256", required=True, type=_sha256) - parser.add_argument( - "--allow-unpinned-feed", - action="store_true", - help=( - "Allow Chronicle facts or manifest hashes outside the committed UK " - "national feed pin; the override is recorded in the run manifest." - ), - ) - parser.add_argument("--staging-h5", required=True, type=Path) - parser.add_argument("--diagnostics-json", required=True, type=Path) - parser.add_argument("--build-record-json", required=True, type=Path) - parser.add_argument("--release-id", required=True) - parser.add_argument("--terminal-gate-json", type=Path) - parser.add_argument("--register-json", type=Path) - parser.add_argument("--measure-exclusions", type=Path) - parser.add_argument("--release-candidate", action="store_true") - parser.add_argument("--logbook-prev-row-digest", type=_sha256) - parser.add_argument("--epochs", type=int) - parser.add_argument("--target-weight-rule") - parser.add_argument("--learning-rate", type=float) - parser.add_argument("--target-loss-cap", type=float) - add_staging_arguments(parser, repository=UK_STAGING_REPOSITORY) - args = parser.parse_args(argv) - args.terminal_gate_json = args.terminal_gate_json or args.staging_h5.with_suffix( - ".terminal_gates.json" - ) - if args.release_candidate: - # The seam refuses release-candidate posture outright (the #757 - # release-cut audit, issue comment 5413502559): its scoped battery - # covers 6 of the declared entries and must never sign a - # shippability claim, and its evidence_absent gaps already refuse - # upstream. A candidate's verdict comes only from the release-cut - # certification producer (tools/certify_uk_release_cut.py). - parser.error( - "--release-candidate is refused on the calibration seam: the " - "seam's scoped battery cannot sign shippability; run the " - "release-cut certification producer instead" - ) - if ( - _CANONICAL_UK_RELEASE_ID.fullmatch(args.release_id) - or args.release_id == _UK_JUNE_RELEASE_ID - or args.release_id == UK_NATIONAL_RELEASE_ID - ): - parser.error( - "canonical UK release ids belong to the release-cut " - "certification producer; the seam runs under a staging or dev " - "release id" - ) - _validate_distinct_paths( - { - "--input-h5": args.input_h5, - "--ledger-facts": args.ledger_facts, - "--staging-h5": args.staging_h5, - "--diagnostics-json": args.diagnostics_json, - "--build-record-json": args.build_record_json, - "--terminal-gate-json": args.terminal_gate_json, - **({"--register-json": args.register_json} if args.register_json else {}), - **( - {"--measure-exclusions": args.measure_exclusions} - if args.measure_exclusions - else {} - ), - } - ) - validate_staging_arguments(parser, args) - return args - - -def _validate_input_paths(args: argparse.Namespace) -> None: - if not args.input_h5.is_file(): - raise SystemExit(f"error: --input-h5 does not exist: {args.input_h5}") - if not args.ledger_facts.exists(): - raise SystemExit(f"error: --ledger-facts does not exist: {args.ledger_facts}") - - -def _create_staging_telemetry(args: argparse.Namespace) -> StagingTelemetryV2 | None: - if args.no_staging: - return None - local_only = args.staging_local_only - return StagingTelemetryV2( - run_id=args.staging_run_id or f"{args.release_id}-calibration", - country_code="GB", - operation_id="uk_national_calibration", - pipeline_id="uk-frs-calibration", - pipeline_version=metadata.version("microcosm-build"), - candidate_id=args.staging_candidate_id or args.release_id, - local_dir=args.staging_dir or args.staging_h5.parent / "staging", - run_kind="calibration", - delivery_mode="local_only" if local_only else "local_and_remote", - repo_id=None if local_only else args.staging_repo_id, - upload_interval_seconds=args.staging_upload_interval_seconds, - ) - - -def _full_sample() -> dict[str, object]: - return {"mode": "full"} - - -def _staging_delivery( - args: argparse.Namespace, telemetry: StagingTelemetryV2 | None -) -> dict[str, object]: - if telemetry is None: - return disabled_staging_delivery("--no-staging") - return telemetry.delivery_summary - - -def _write_json(path: Path, payload: dict[str, object]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - temporary = path.with_name(path.name + ".tmp") - temporary.write_text( - json.dumps(payload, indent=2, sort_keys=True, allow_nan=False) + "\n", - encoding="utf-8", - ) - temporary.replace(path) - - -def _sha256(value: str) -> str: - if not _SHA256.fullmatch(value): - raise argparse.ArgumentTypeError("expected a lowercase SHA-256 digest") - return value - - -def _sha256_file(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as stream: - for chunk in iter(lambda: stream.read(1 << 20), b""): - digest.update(chunk) - return digest.hexdigest() - - -def _check_committed_ledger_feed_pin( - facts_sha256: str, - *, - manifest_sha256: str | None, - allow_unpinned_feed: bool, - pin: UKChronicleFeed | None = None, -) -> None: - try: - require_committed_uk_chronicle_feed_pin( - facts_sha256, - manifest_sha256=manifest_sha256, - allow_unpinned_feed=allow_unpinned_feed, - pin=pin, - ) - except UKChronicleFeedPinError as error: - raise SystemExit(f"error: {error}") from error - - -def _validate_distinct_paths(paths: dict[str, Path]) -> None: - resolved = {label: path.expanduser().resolve() for label, path in paths.items()} - for (left_label, left), (right_label, right) in combinations(resolved.items(), 2): - if _paths_alias(left, right): - raise SystemExit( - f"error: {left_label} and {right_label} must be distinct paths " - f"({left} aliases {right})" - ) - - -def _paths_alias(left: Path, right: Path) -> bool: - if str(left).casefold() == str(right).casefold(): - return True - try: - left_stat = left.stat() - right_stat = right.stat() - except FileNotFoundError: - return False - return (left_stat.st_dev, left_stat.st_ino) == ( - right_stat.st_dev, - right_stat.st_ino, - ) - - -def _compare_frozen_register(path: Path | None, registry: Any) -> None: - """Refuse when the compiled register is not the frozen scoring surface. - - Both sides are compared on ``TargetRegistry.version`` — the registry's own - content hash — through the validating loader, so the same artifact serves - this check and ``tools/score_uk_national_candidate.py``, and incidental - formatting cannot make two identical registers look different. - """ - - if path is None: - return - try: - frozen = TargetRegistry.from_json(path) - except ValueError as error: - raise SystemExit( - f"error: frozen scoring register is unusable: {error}" - ) from error - if frozen.version != registry.version: - raise SystemExit( - "re-derived register differs from the frozen scoring register: " - f"{registry.version} vs {frozen.version}" - ) - - -def _ledger_facts_pin(artifact: Any) -> dict[str, object]: - facts_path = ( - artifact.path / "consumer_facts.jsonl" - if artifact.path.is_dir() - else artifact.path - ) - return {"sha256": artifact.facts_sha256, "size_bytes": facts_path.stat().st_size} - - -if __name__ == "__main__": - raise SystemExit(main()) From fffb021c68cce76039a6f6e3d1564f2e825127e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Sun, 20 Sep 2026 18:19:34 +0100 Subject: [PATCH 06/11] Record the #823 consolidation receipts: the national role rebuilds v20 bit for bit and stages to the Hub MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R1: tools/build_uk_rowwise_candidate.py --release-role national on the v20 inputs (spine-s 4d9752fd…, feed c5e5bf8, doctrine 1,500 / family_equal / 0.02 / seed 0, no solve flags) reproduces the v20 seam-built dataset exactly: household weights array_equal on 52,846 households, the 638 diagnostics rows and losses equal, register f536021bbcd9, six gates passed, an empty override receipt where v20 carried two. R2: the same build with remote staging and read-back lands its telemetry under runs// (progress completed, 35/35 uploads, read-back passed) and its bundle under staged// in one commit; the fetch tool returns digest-verified files and a byte-equal H5. R3: certification is not run yet — the repository scorer refuses the 119 targets the eFRS incumbent cannot materialize; the v20 head-to-head used the evaluation repo's pruning scorer. Co-Authored-By: Claude Fable 5.1 --- experiments/823-uk-consolidation-receipts.md | 72 ++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 experiments/823-uk-consolidation-receipts.md diff --git a/experiments/823-uk-consolidation-receipts.md b/experiments/823-uk-consolidation-receipts.md new file mode 100644 index 000000000..4fe1174f5 --- /dev/null +++ b/experiments/823-uk-consolidation-receipts.md @@ -0,0 +1,72 @@ +# microcosm#823 consolidation receipts + +## R1 — the national role rebuilds the v20 seam-built dataset bit for bit (2026-09-20) + +Comparison run: `tools/build_uk_rowwise_candidate.py --release-role national` on the +v20 inputs — spine-s (`4d9752fdcd92…`, 175,920,391 bytes), Chronicle feed +`c5e5bf8` (facts `6d039dd869dc…`, manifest `20ac5d22e7d8…`), the committed feed +pin — under the ruled doctrine (1,500 epochs, `family_equal`, learning rate 0.02, +seed 0) with no solve flags, `--staging-local-only`; code = the B3 tip plus the +signed self-employment 20-30k deferral (`90bdb809`, cherry-picked as `3828183c`, +`git_dirty` false, engine 2.98.0). Output +`data/ukds/acceptance/823-consolidation/national-v20-twin/` (attempt id in its +build record). Reference: `runs/uk-623-first-calibrated/spine-assessment-v20/` +(`uk-frs-calibration-attempt-20260918T174754Z-3f928f19`, built by the retired +`tools/calibrate_uk_national_dataset.py` with `--epochs 1500 +--target-weight-rule family_equal`). + +- Household weights: `numpy.array_equal` on 52,846 households — **equal**. +- `calibration_diagnostics.json`: 638 target rows (name, target, initial and + final estimate, relative error) — **equal**; final loss 0.010414168864537107, + initial loss 0.3009223937988281, ESS 9,304.74, within-10 % 96.39 %, realized + max weight ratio 10.0 — **equal**. +- Register: 705 compiled, 67 excluded, 638 calibrated, version `f536021bbcd9` — + **equal**. +- Build record: `calibration` block, `input_posture`, `spine_provenance`, + `run_config.ledger`, `run_config.doctrine` — **equal**; six calibration-seam + gates all `passed` on both; `run_config.doctrine_overrides` is `{}` on the + twin where v20 carried `{epochs 256 → 1500, target_weight_rule uniform → + family_equal}` (the constants moved in B1). +- Wall time 456 s (v20: 852 s on a loaded machine), peak RSS 9.3 GB. + +Rowwise manifest: `uk_national_calibrated_candidate`, `release_role national`, +`release_id microcosm-uk-2024-25-national`, `staged_dataset.status skipped` +(local-only), `staging_delivery.mode local_only`; frozen +`national_target_registry.json` version `f536021bbcd9`. + +## R2 — the same build with remote staging and read-back (2026-09-20) + +Same inputs, code and doctrine as R1, remote staging on at the driver's default +300 s cadence with `--staging-read-back`; output +`data/ukds/acceptance/823-consolidation/national-v20-twin-remote/`, attempt id +`uk-frs-calibration-attempt-20260920T170811Z-ce339e7c`, wall 550 s. + +- Household weights and the 638 diagnostics rows equal to v20 (and to R1); + register `f536021bbcd9`; six gates passed; `doctrine_overrides` `{}`. +- Telemetry: `policyengine/populace-uk-staging` `runs//` holds + `run_manifest.json`, `progress.json` (status `completed`), `events.ndjson`, + `calibration_progress.json`, `artifacts/fit_summary.json`, + `artifacts/staged_dataset.json`; `staging_delivery` in both the build record + and the manifest: mode `local_and_remote`, 35 attempts / 35 successes, + `read_back passed`, no error code; no telemetry warning in the run log. +- Bundle: `policyengine/populace-uk-private` `staged//` (revision + `40439a3f…`) holds the five outputs (`microcosm_uk_2024_25.h5` 176,040,337 + bytes, `build_record.json`, `calibration_diagnostics.json`, + `microcosm_uk_2024_25.terminal_gates.json`, `national_target_registry.json`), + the manifest, `staged_manifest.json` and `sha256sums.txt`, in one commit; + `staged_dataset.status uploaded`. +- `tools/fetch_uk_staged_dataset.py --run-id ` returned every file + digest-verified and the fetched H5 is byte-equal to the local one (the copy + was deleted after the check). + +## R3 — certification of the cut (not run) + +The plan's step 3 scores the candidate against the eFRS 1.57.3 copy on the run's +frozen register with `tools/score_uk_national_candidate.py`. That scorer refuses +a register it cannot materialize on both sides, and 119 of the 638 targets are +unresolvable on the incumbent (the v20 head-to-head used the evaluation repo's +pass-2 scorer, which prunes them, receipted in its run). A repository-native +certification of a national cut therefore still needs a reviewed common +register or a pruning scorer in this repository; certification, assembly and +the inspect publication wait on that (and on María's go for anything that +reaches `releases/`). From cc23ec06f8c485273b57c9824bf498dc143b2a8c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:50:55 +0100 Subject: [PATCH 07/11] Address review round 1 on #965 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The data registry's dense variant names microcosm_uk_2024_25_dense.h5 (the file the assembler mints and the contract reads), with its test re-pinned and a third leg on the contract-pin lockstep test (assembler, contract, registry): the #900 writer-and-reader class, one hop further out. - The doctrine comment names the ruler and where the ruling is recorded (María, 2026-09-20, acknowledged on microcosm#965); the two test comments say the same. - Tests for two claimed protections: the national role refuses an explicit --seed that is not the doctrine's (and accepts the doctrine's own), and a size checkpoint written before the release role existed refuses to resume ("release_role: absent in checkpoint"). - The receipts name R1's attempt id and state the code the rehearsal ran on (eba0e152 plus the signed deferral, which is the v20 branch's own commit and not in this PR). - The staging operations guide says the pre-flight's and the assembler's release-role gate runs before --allow-missing-staging. Co-Authored-By: Claude Fable 5.1 --- changelog.d/uk-dense-filename-823.changed.md | 2 +- docs/uk-staging-operations.md | 5 ++- experiments/823-uk-consolidation-receipts.md | 13 +++++--- .../build/uk_runtime/national_doctrine.py | 3 +- .../tests/test_gate_battery_contract_pins.py | 5 +++ .../tests/test_uk_childcare_targets.py | 2 +- .../tests/test_uk_national_calibration.py | 2 +- .../tests/test_uk_rowwise_candidate.py | 31 +++++++++++++++++-- .../src/microcosm/data/registry.py | 9 +++--- .../microcosm-data/tests/test_registry.py | 2 +- 10 files changed, 56 insertions(+), 18 deletions(-) diff --git a/changelog.d/uk-dense-filename-823.changed.md b/changelog.d/uk-dense-filename-823.changed.md index a42c31b70..7fa59bc06 100644 --- a/changelog.d/uk-dense-filename-823.changed.md +++ b/changelog.d/uk-dense-filename-823.changed.md @@ -1 +1 @@ -The UK dense line's published microdata is `microcosm_uk_2024_25_dense.h5` (the FRS release vintage, matching the `microcosm-uk-2024-25-dense` release id) and the driver's joint-surface outputs are `microcosm_uk_2024_25_local.h5` and `microcosm_uk_2024_25_local.local_gates.json`; the contract mirrors the published filename with a lockstep pin. +The UK dense line's published microdata is `microcosm_uk_2024_25_dense.h5` (the FRS release vintage, matching the `microcosm-uk-2024-25-dense` release id) and the driver's joint-surface outputs are `microcosm_uk_2024_25_local.h5` and `microcosm_uk_2024_25_local.local_gates.json`; the contract and the data registry's dense variant mirror the published filename under a three-legged lockstep pin. diff --git a/docs/uk-staging-operations.md b/docs/uk-staging-operations.md index 5b7150625..cf3fe4971 100644 --- a/docs/uk-staging-operations.md +++ b/docs/uk-staging-operations.md @@ -229,7 +229,10 @@ requires the manifest's `staging_delivery` receipt and copies it into `build_manifest.json` as `staging`, where publication reads it. A dense run built before this lane carries no receipt; `--allow-missing-staging` assembles it with a recorded disabled-staging opt-out naming the override, the same -posture publication's `--allow-missing-staging` grants. +posture publication's `--allow-missing-staging` grants. Since microcosm#823 the +pre-flight and the assembler check the manifest's `release_role` before that +override, so a run built before the release role existed is refused on the +role and is rebuilt, never grandfathered. ## Smoke verification diff --git a/experiments/823-uk-consolidation-receipts.md b/experiments/823-uk-consolidation-receipts.md index 4fe1174f5..574d6b02d 100644 --- a/experiments/823-uk-consolidation-receipts.md +++ b/experiments/823-uk-consolidation-receipts.md @@ -6,11 +6,14 @@ Comparison run: `tools/build_uk_rowwise_candidate.py --release-role national` on v20 inputs — spine-s (`4d9752fdcd92…`, 175,920,391 bytes), Chronicle feed `c5e5bf8` (facts `6d039dd869dc…`, manifest `20ac5d22e7d8…`), the committed feed pin — under the ruled doctrine (1,500 epochs, `family_equal`, learning rate 0.02, -seed 0) with no solve flags, `--staging-local-only`; code = the B3 tip plus the -signed self-employment 20-30k deferral (`90bdb809`, cherry-picked as `3828183c`, -`git_dirty` false, engine 2.98.0). Output -`data/ukds/acceptance/823-consolidation/national-v20-twin/` (attempt id in its -build record). Reference: `runs/uk-623-first-calibrated/spine-assessment-v20/` +seed 0) with no solve flags, `--staging-local-only`; code = `eba0e152` (the +retirement commit, this PR's head minus the receipts commit) plus the signed +self-employment 20-30k deferral (`90bdb809`, cherry-picked as `3828183c`, +`git_dirty` false, engine 2.98.0) — the deferral is not in this PR (it is the +v20 branch's own commit) and without it the target-fit gate blocks while the +weights still compare, so the same inputs on the bare head are unverified past +the gate. Output `data/ukds/acceptance/823-consolidation/national-v20-twin/`, +attempt id `uk-frs-calibration-attempt-20260920T165920Z-5acd7515`. Reference: `runs/uk-623-first-calibrated/spine-assessment-v20/` (`uk-frs-calibration-attempt-20260918T174754Z-3f928f19`, built by the retired `tools/calibrate_uk_national_dataset.py` with `--epochs 1500 --target-weight-rule family_equal`). diff --git a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py index db667c216..21f446219 100644 --- a/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py +++ b/packages/microcosm-build/src/microcosm/build/uk_runtime/national_doctrine.py @@ -58,7 +58,8 @@ # per-run override (its own run-9 receipts show a 9-member family carrying # 5.5x the per-reference weight of a 50-member one). # -# Ruling 2026-09-20 (microcosm#823 consolidation): every campaign national +# María's ruling (2026-09-20, acknowledged on microcosm#965): every campaign +# national # run since v18 has overridden to family_equal at 1,500 epochs — the v20 # certified-cut posture (spine-s, loss 0.0104, 6/6 seam gates) carried the # receipt {epochs 256 → 1500, target_weight_rule uniform → family_equal} — diff --git a/packages/microcosm-build/tests/test_gate_battery_contract_pins.py b/packages/microcosm-build/tests/test_gate_battery_contract_pins.py index bfabcf43e..bdf461316 100644 --- a/packages/microcosm-build/tests/test_gate_battery_contract_pins.py +++ b/packages/microcosm-build/tests/test_gate_battery_contract_pins.py @@ -404,7 +404,12 @@ def test_dataset_filename_mirrors_the_assembler(self) -> None: assembler = importlib.util.module_from_spec(spec) assert spec.loader is not None spec.loader.exec_module(assembler) + from microcosm.data.registry import REGISTRY + assert data_contract._UK_DENSE_DATASET_FILENAME == assembler._DATASET_FILENAME + # The reader's leg: the registry's dense variant must name the file + # the assembler mints and the contract reads (the #900 class). + assert REGISTRY[("uk", 2025, "dense")].filename == assembler._DATASET_FILENAME assert ( data_contract._UK_DENSE_DATASET_FILENAME == "microcosm_uk_2024_25_dense.h5" ) diff --git a/packages/microcosm-build/tests/test_uk_childcare_targets.py b/packages/microcosm-build/tests/test_uk_childcare_targets.py index 831427372..34841d81f 100644 --- a/packages/microcosm-build/tests/test_uk_childcare_targets.py +++ b/packages/microcosm-build/tests/test_uk_childcare_targets.py @@ -63,5 +63,5 @@ def test_childcare_activation_preserves_existing_calibration_gates() -> None: assert by_id["uk_calibration_reference_coverage"]["criticality"] == ( "release_blocking" ) - # Ruling 2026-09-20 (microcosm#823): the doctrine rule is family_equal. + # María's ruling (2026-09-20, microcosm#965): the doctrine rule is family_equal. assert UK_NATIONAL_SOLVE_DOCTRINE.target_weight_rule == "family_equal" diff --git a/packages/microcosm-build/tests/test_uk_national_calibration.py b/packages/microcosm-build/tests/test_uk_national_calibration.py index 53af50b7d..7e4fb3c9c 100644 --- a/packages/microcosm-build/tests/test_uk_national_calibration.py +++ b/packages/microcosm-build/tests/test_uk_national_calibration.py @@ -862,7 +862,7 @@ def test_post_solve_fence_requires_calibrated_kind_and_mass_record() -> None: def test_national_doctrine_constants_are_the_declared_contract() -> None: - # Ruling 2026-09-20 (microcosm#823): the certified-cut posture every + # María's ruling (2026-09-20, microcosm#965): the certified-cut posture every # campaign national run overrode to — 1,500 epochs, family_equal — is # the doctrine, so a certified national cut records no overrides. assert UK_NATIONAL_SOLVE_EPOCHS == 1500 diff --git a/packages/microcosm-build/tests/test_uk_rowwise_candidate.py b/packages/microcosm-build/tests/test_uk_rowwise_candidate.py index 1d8041572..1bc74fffd 100644 --- a/packages/microcosm-build/tests/test_uk_rowwise_candidate.py +++ b/packages/microcosm-build/tests/test_uk_rowwise_candidate.py @@ -6,6 +6,7 @@ import hashlib import importlib.util import json +import shutil from dataclasses import replace from pathlib import Path from types import SimpleNamespace @@ -2618,9 +2619,30 @@ def test_size_candidate_checkpoints_before_the_draw_and_resumes_from_it( ) assert "written_at" not in resumed_receipt + # --------------------------------------------------------------------------- + # Staging: telemetry to runs// and the staged dataset bundle. -# --------------------------------------------------------------------------- -# Staging: telemetry to runs// and the staged dataset bundle. + # A checkpoint written before the release role existed (no release_role + # in its identity) refuses to resume: the identity is the run's + # contract, and a pre-role checkpoint is rebuilt, never grandfathered. + legacy = tmp_path / "legacy" + shutil.copytree(first, legacy) + legacy_manifest = json.loads( + (legacy / SIZE_CHECKPOINT_MANIFEST_FILENAME).read_text() + ) + del legacy_manifest["identity"]["release_role"] + (legacy / SIZE_CHECKPOINT_MANIFEST_FILENAME).write_text(json.dumps(legacy_manifest)) + with pytest.raises(ValueError, match="release_role: absent in checkpoint"): + builder.main( + [ + *common, + "--out", + str(tmp_path / "from-legacy"), + "--resume-size-checkpoint", + str(legacy), + ] + ) + assert not (tmp_path / "from-legacy" / builder.MANIFEST_FILENAME).exists() def _load_tool(name: str): @@ -3426,6 +3448,10 @@ def test_release_role_supplies_the_solve_defaults(tmp_path) -> None: explicit = builder._parse_args(_role_argv(tmp_path, "national", "--epochs", "5")) assert explicit.epochs == 5 assert explicit._explicit_arguments == frozenset({"epochs"}) + # The doctrine's own seed may be spelled out; only another seed is refused. + builder._validate_cli_args( + builder._parse_args(_role_argv(tmp_path, "national", "--seed", "0")) + ) @pytest.mark.parametrize( @@ -3472,6 +3498,7 @@ def test_dense_role_requires_the_ladder(tmp_path) -> None: (["--resume-size-checkpoint", "dir"], "--resume-size-checkpoint"), (["--sample-fraction", "0.1"], "--sample-fraction"), (["--sample-seed", "9"], "--sample-seed"), + (["--seed", "7"], "--seed != doctrine 0"), (["--target-weight-rule", "grain_equal"], "--target-weight-rule grain_equal"), ], ) diff --git a/packages/microcosm-data/src/microcosm/data/registry.py b/packages/microcosm-data/src/microcosm/data/registry.py index eaa5e190f..73898089e 100644 --- a/packages/microcosm-data/src/microcosm/data/registry.py +++ b/packages/microcosm-data/src/microcosm/data/registry.py @@ -110,15 +110,14 @@ def register(spec: DatasetSpec) -> DatasetSpec: register( DatasetSpec( - # microcosm#762 A18: the dense joint national + local UK line. Not a - # default variant — the compact national line stays the fast - # microsimulation artifact; this one carries every constituency and - # local authority at K=15 clones per spine household. + # microcosm#762 A18: the dense joint national + local UK build is + # tag-only; no pointer. It carries every constituency and local + # authority at K=15 clones per spine household. country="uk", year=2025, variant="dense", hf_repo="policyengine/populace-uk-private", - filename="microcosm_uk_2025_dense.h5", + filename="microcosm_uk_2024_25_dense.h5", engine_module="policyengine_uk.data", engine_class="UKSingleYearDataset", engine_package="policyengine-uk", diff --git a/packages/microcosm-data/tests/test_registry.py b/packages/microcosm-data/tests/test_registry.py index 4a069fef9..1d68af314 100644 --- a/packages/microcosm-data/tests/test_registry.py +++ b/packages/microcosm-data/tests/test_registry.py @@ -115,7 +115,7 @@ def test_uk_2025_dense_is_registered_off_the_default_variant() -> None: spec = REGISTRY[("uk", 2025, "dense")] assert spec.variant == "dense" != DEFAULT_VARIANT assert spec.hf_repo == "policyengine/populace-uk-private" - assert spec.filename == "microcosm_uk_2025_dense.h5" + assert spec.filename == "microcosm_uk_2024_25_dense.h5" assert spec.engine_class == "UKSingleYearDataset" assert spec.engine_package == "policyengine-uk" assert ("uk", 2025, DEFAULT_VARIANT) not in REGISTRY From acd93c12b90f9a8dd4d1dbf0ab34df34c85f01ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:22:23 +0100 Subject: [PATCH 08/11] Evaluate the national candidate against the incumbent at the end of the build The national role takes --incumbent-h5 / --incumbent-sha256 (and a label) and, after the bundle is staged and before the telemetry completes, scores the finished candidate against it on the surface both can materialize (microcosm#967's common-surface scorer, loaded lazily so this branch stands without it): score_vs_incumbent.json beside the outputs and as a reviewed telemetry artifact, rows the incumbent cannot materialize pruned from both arms and named on stderr, and a manifest evaluation block with the rule-1 verdict the release-cut certifier requires. The evaluation never blocks the build: an error is recorded and warned, staging and the exit code stand. Without the flags the manifest says not_requested. The full compiled register is frozen beside the solve register as national_contract_registry.json, the band edges a pruned scoring surface must never redraw (#803). The dense role refuses the flags. Co-Authored-By: Claude Fable 5.1 --- ...nal-role-incumbent-evaluation-823.added.md | 1 + docs/uk-national-calibration-runbook-623.md | 4 +- ...k-national-release-assembly-runbook-806.md | 20 +- docs/uk-staging-operations.md | 8 +- .../tests/test_uk_rowwise_national_role.py | 330 ++++++++++++++++++ tools/build_uk_rowwise_candidate.py | 215 +++++++++++- 6 files changed, 573 insertions(+), 5 deletions(-) create mode 100644 changelog.d/uk-national-role-incumbent-evaluation-823.added.md diff --git a/changelog.d/uk-national-role-incumbent-evaluation-823.added.md b/changelog.d/uk-national-role-incumbent-evaluation-823.added.md new file mode 100644 index 000000000..3696e92fa --- /dev/null +++ b/changelog.d/uk-national-role-incumbent-evaluation-823.added.md @@ -0,0 +1 @@ +The rowwise driver's national role evaluates the finished candidate against an incumbent (`--incumbent-h5`, `--incumbent-sha256`, `--incumbent-label`) at the end of the build, after the bundle is staged and before the telemetry completes: `score_vs_incumbent.json` beside the outputs and as a reviewed telemetry artifact, rows the incumbent cannot materialize pruned from both arms and named on stderr, and a manifest `evaluation` block with the rule-1 verdict the release-cut certifier requires. The evaluation never blocks the build; an error is recorded and warned. The full compiled register is frozen beside the solve register as `national_contract_registry.json`. The dense role refuses the flags.\n \ No newline at end of file diff --git a/docs/uk-national-calibration-runbook-623.md b/docs/uk-national-calibration-runbook-623.md index 8b7d68bb2..afdc2fc1b 100644 --- a/docs/uk-national-calibration-runbook-623.md +++ b/docs/uk-national-calibration-runbook-623.md @@ -95,8 +95,10 @@ and both sides are scored on the same frozen register. - `microcosm_uk_2024_25.terminal_gates.json` - `calibration_diagnostics.json` - `national_target_registry.json` +- `national_contract_registry.json` - `rowwise_candidate_manifest.json` -- `score_vs_enhanced_frs.json` +- `score_vs_incumbent.json` (the build's own when run with `--incumbent-h5`; + the manifest's `evaluation` block records its verdict) - `logbook-spool/` (one row for the attempt, whatever its disposition) Acceptance follows #578: the candidate must not regress incumbent battery diff --git a/docs/uk-national-release-assembly-runbook-806.md b/docs/uk-national-release-assembly-runbook-806.md index 7ebe82af6..0ce8fda7a 100644 --- a/docs/uk-national-release-assembly-runbook-806.md +++ b/docs/uk-national-release-assembly-runbook-806.md @@ -37,15 +37,31 @@ uv run --no-sync python tools/build_uk_rowwise_candidate.py --release-role natio --ledger-facts \ --ledger-facts-sha256 \ --ledger-manifest-sha256 \ + --incumbent-h5 \ + --incumbent-sha256 \ --out ``` +`--incumbent-h5` (with its digest; `--incumbent-label` names it, default +`enhanced_frs_2024_25`) makes the build evaluate the finished candidate +against the incumbent after the bundle is staged: `score_vs_incumbent.json` +lands beside the outputs with the rule-1 verdict, rows the incumbent cannot +materialize are pruned from both arms and named on stderr, the receipt rides +the staging telemetry as `artifacts/score_vs_incumbent.json`, and the manifest +records `evaluation` (`status`, `verdict`, `rule_1`, `scored_surface`, +`pruned_measures`). The evaluation never blocks the build: an error is +recorded as `status: error` and warned. Without the flags the manifest says +`not_requested`, and the certifier has no receipt to read until the candidate +is scored by hand. + The role's doctrine is the campaign posture (1,500 epochs, `family_equal`, learning rate 0.02, seed 0): a certified cut passes no solve flags and records no overrides. It writes `microcosm_uk_2024_25.h5`, `calibration_diagnostics.json`, `build_record.json`, -`microcosm_uk_2024_25.terminal_gates.json`, `national_target_registry.json` -and `rowwise_candidate_manifest.json` into ``. The build +`microcosm_uk_2024_25.terminal_gates.json`, `national_target_registry.json`, +`national_contract_registry.json` (the full compiled register the scoring +surface takes its band edges from) and `rowwise_candidate_manifest.json` +into ``. The build record's id has the form `uk-frs-calibration-attempt--`; assembly derives the per-cut tag from that suffix. diff --git a/docs/uk-staging-operations.md b/docs/uk-staging-operations.md index cf3fe4971..6fde00d9c 100644 --- a/docs/uk-staging-operations.md +++ b/docs/uk-staging-operations.md @@ -200,7 +200,13 @@ code or Logbook disposition. The telemetry run declares two reviewed artifacts, `artifacts/staged_dataset.json` (the same block) and `artifacts/fit_summary.json` (loss, fit by family, gate verdicts, the size receipt without its per-row arrays), so a run in the dashboard points at its -bundle. +bundle. A national run given `--incumbent-h5` adds a third, +`artifacts/score_vs_incumbent.json`: the rule-1 score receipt against the +incumbent on the surface both can materialize, produced after the bundle is +staged and before the telemetry completes, with the rows the incumbent cannot +materialize pruned from both arms and listed; the manifest's `evaluation` +block records its verdict, and the release-cut certifier requires that +verdict to be `passed`. Re-stage a finished directory, including runs built before this lane existed or whose upload failed, with: diff --git a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py index d24c44d4a..8dd089924 100644 --- a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py +++ b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py @@ -8,9 +8,11 @@ from __future__ import annotations +import argparse import hashlib import importlib.util import json +import shutil from pathlib import Path from types import SimpleNamespace @@ -250,6 +252,18 @@ def test_uk_national_role_delegates_to_the_seam_library(monkeypatch, tmp_path): "build_record", "terminal_gate_report", "national_target_registry", + "national_contract_registry", + } + assert manifest["outputs"]["national_contract_registry"]["sha256"] == _sha( + out / "national_contract_registry.json" + ) + assert manifest["evaluation"] == { + "status": "not_requested", + "note": ( + "no --incumbent-h5: the rule-1 score receipt the release-cut " + "certifier needs was not produced; score the candidate with " + "tools/score_uk_national_candidate.py before certification." + ), } assert manifest["outputs"]["build_record"]["sha256"] == _sha( out / "build_record.json" @@ -647,3 +661,319 @@ def test_uk_national_role_records_an_unpinned_feed_override( assert extra["allow_unpinned_feed"] is True # The pin the artifact was checked against is the committed one. assert extra["chronicle_feed_pin"]["facts_sha256"] != "0" * 64 + + +# --- the end-of-build incumbent evaluation (microcosm#823 E2) ----------------- + + +def _fake_evaluator(receipt_of, calls: list[dict]): + """A stand-in for microcosm.build.uk_runtime.candidate_score (#967).""" + + class Module: + @staticmethod + def evaluate_uk_candidate_against_incumbent(**kwargs): + calls.append(kwargs) + return receipt_of(kwargs) + + @staticmethod + def _default_measure_resolver_factory(scratch_dir, year): + return lambda path, frame: None + + @staticmethod + def pruned_warning(score): + pruned = score.get("incumbent_unresolvable_pruned") or {} + if not pruned.get("n_pruned"): + return None + return "warning: pruned from BOTH arms: " + ", ".join(pruned["measures"]) + + return Module + + +def _receipt(kwargs, *, verdict="passed", pruned_measures=()): + n_pruned = len(pruned_measures) + return { + "artifacts": { + "candidate": {"sha256": kwargs["candidate_sha256"]}, + "incumbent": { + "sha256": kwargs["incumbent_sha256"], + "label": kwargs["incumbent_label"], + }, + }, + "candidate_full_loss": 0.01, + "incumbent_full_loss": 0.2, + "candidate_target_wins": 1, + "incumbent_target_wins": 0, + "register": {"n_specs": 1}, + "incumbent_unresolvable_pruned": { + "n_pruned": n_pruned, + "n_scored": 1, + "n_surface": 1 + n_pruned, + "pruned_targets": {}, + "measures": list(pruned_measures), + "families": {"dwp_universal_credit": n_pruned} if n_pruned else {}, + "note": "pruned", + }, + "evaluation": { + "schema_version": 1, + "rule": "rule 1", + "scored_surface": { + "n_scored": 1, + "n_pruned": n_pruned, + "n_surface": 1 + n_pruned, + }, + "rule_1": { + "passed": verdict == "passed", + "candidate_full_loss": 0.01, + "incumbent_full_loss": 0.2, + "candidate_target_wins": 1, + "incumbent_target_wins": 0, + }, + "verdict": verdict, + }, + } + + +def _incumbent_argv(input_h5: Path, tmp_path: Path, *extra: str) -> list[str]: + incumbent = tmp_path / "incumbent.h5" + shutil.copyfile(input_h5, incumbent) + return [ + "--incumbent-h5", + str(incumbent), + "--incumbent-sha256", + _sha(incumbent), + "--incumbent-label", + "efrs_fixture", + *extra, + ] + + +def test_uk_dense_role_refuses_the_incumbent_flags(tmp_path) -> None: + builder = _CANDIDATE._load_builder_module() + args = argparse.Namespace( + target_loss_cap=None, + allow_unpinned_feed=False, + target_weight_rule="grain_equal", + incumbent_h5=tmp_path / "incumbent.h5", + incumbent_sha256=None, + ) + with pytest.raises(ValueError, match="--incumbent-h5/--incumbent-sha256"): + builder._refuse_national_role_arguments( + args, builder.uk_rowwise_posture("dense") + ) + + +def test_uk_national_role_requires_the_incumbent_pair(monkeypatch, tmp_path): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + with pytest.raises(ValueError, match="must be given together"): + builder.main( + _argv( + input_h5, + tmp_path / "out", + "--no-staging", + "--incumbent-h5", + str(input_h5), + ) + ) + + +def test_uk_national_role_refuses_the_incumbent_without_the_scorer() -> None: + builder = _CANDIDATE._load_builder_module() + + def missing(name: str): + raise ImportError(name) + + with pytest.raises(ValueError, match="microcosm#967"): + builder._load_candidate_evaluator(importer=missing) + + +def test_uk_national_role_evaluates_against_the_incumbent( + monkeypatch, tmp_path, capsys +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + monkeypatch.setattr(builder, "run_uk_calibration", _fake_seam_run([])) + calls: list[dict] = [] + module = _fake_evaluator( + lambda kwargs: _receipt( + kwargs, pruned_measures=("benunit.uc_calibration_child_count",) + ), + calls, + ) + monkeypatch.setattr( + builder, "_load_candidate_evaluator", lambda importer=None: module + ) + out = tmp_path / "national" + + assert ( + builder.main( + _argv(input_h5, out, "--no-staging", *_incumbent_argv(input_h5, tmp_path)) + ) + == 0 + ) + + call = calls[0] + assert call["candidate_h5"] == out / "microcosm_uk_2024_25.h5" + assert call["candidate_sha256"] == _sha(out / "microcosm_uk_2024_25.h5") + assert call["incumbent_h5"] == tmp_path / "incumbent.h5" + assert call["incumbent_sha256"] == _sha(tmp_path / "incumbent.h5") + assert call["incumbent_label"] == "efrs_fixture" + assert call["target_registry"] is registry + assert call["band_edge_registry"] is registry + assert call["calibration_year"] == 2025 + receipt_path = out / "score_vs_incumbent.json" + receipt = json.loads(receipt_path.read_text()) + assert receipt["evaluation"]["verdict"] == "passed" + manifest = json.loads((out / builder.MANIFEST_FILENAME).read_text()) + evaluation = manifest["evaluation"] + assert evaluation["status"] == "completed" + assert evaluation["verdict"] == "passed" + assert evaluation["rule_1"]["passed"] is True + assert evaluation["scored_surface"] == { + "n_scored": 1, + "n_pruned": 1, + "n_surface": 2, + } + assert evaluation["pruned_measures"] == ["benunit.uc_calibration_child_count"] + assert evaluation["pruned_families"] == {"dwp_universal_credit": 1} + assert evaluation["receipt"]["sha256"] == _sha(receipt_path) + assert evaluation["incumbent"]["label"] == "efrs_fixture" + assert evaluation["incumbent"]["sha256"] == _sha(tmp_path / "incumbent.h5") + assert manifest["outputs"]["score_receipt"]["sha256"] == _sha(receipt_path) + # The absent measure is named loudly on stderr, never hidden. + assert "benunit.uc_calibration_child_count" in capsys.readouterr().err + + +def test_uk_national_role_records_an_evaluation_error_without_failing( + monkeypatch, tmp_path, capsys +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + monkeypatch.setattr(builder, "run_uk_calibration", _fake_seam_run([])) + + def explode(kwargs): + raise RuntimeError("the incumbent frame refused to load") + + module = _fake_evaluator(explode, []) + monkeypatch.setattr( + builder, "_load_candidate_evaluator", lambda importer=None: module + ) + out = tmp_path / "national" + + assert ( + builder.main( + _argv(input_h5, out, "--no-staging", *_incumbent_argv(input_h5, tmp_path)) + ) + == 0 + ) + + manifest = json.loads((out / builder.MANIFEST_FILENAME).read_text()) + assert manifest["evaluation"]["status"] == "error" + assert "refused to load" in manifest["evaluation"]["error"] + assert manifest["evaluation"]["receipt"] is None + assert "score_receipt" not in manifest["outputs"] + assert not (out / "score_vs_incumbent.json").exists() + assert "the incumbent evaluation failed" in capsys.readouterr().err + + +def test_uk_national_role_evaluates_after_staging_the_bundle( + monkeypatch, tmp_path, capsys +): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + calls: list[dict] = [] + module = _fake_evaluator(lambda kwargs: _receipt(kwargs), calls) + monkeypatch.setattr( + builder, "_load_candidate_evaluator", lambda importer=None: module + ) + out = tmp_path / "national" + + assert ( + builder.main( + _argv( + input_h5, + out, + "--staging-local-only", + "--epochs", + "5", + *_incumbent_argv(input_h5, tmp_path), + ) + ) + == 0 + ) + + manifest = json.loads((out / builder.MANIFEST_FILENAME).read_text()) + run_id = manifest["build_id"] + assert manifest["evaluation"]["status"] == "completed" + assert manifest["evaluation"]["verdict"] == "passed" + assert manifest["outputs"]["score_receipt"]["sha256"] == _sha( + out / "score_vs_incumbent.json" + ) + # The receipt is listed in the local sums beside the record and manifest. + _sums_verify(out) + assert any( + line.endswith(" score_vs_incumbent.json") + for line in (out / "sha256sums.txt").read_text().splitlines() + ) + bundle = validate_v2_bundle(out / "staging", run_id) + assert bundle["progress"]["status"] == "completed" + events = [ + (event["stage_id"], event["status"]) + for event in bundle["events"] + if event["stage_id"] == "incumbent_evaluation" + ] + assert events == [ + ("incumbent_evaluation", "started"), + ("incumbent_evaluation", "completed"), + ] + # The evaluation ran after the bundle was staged, before completion. + stage_ids = [event["stage_id"] for event in bundle["events"]] + assert stage_ids.index("dataset_staging") < stage_ids.index("incumbent_evaluation") + assert stage_ids.index("incumbent_evaluation") < stage_ids.index("complete") + artifacts = out / "staging" / "runs" / run_id / "artifacts" + staged_receipt = json.loads((artifacts / "score_vs_incumbent.json").read_text()) + assert staged_receipt["evaluation"]["verdict"] == "passed" + assert calls[0]["candidate_sha256"] == manifest["outputs"]["dataset"]["sha256"] + + +def test_uk_national_dry_run_records_the_incumbent(monkeypatch, tmp_path, capsys): + pytest.importorskip("tables") + builder = _CANDIDATE._load_builder_module() + input_h5, _registry, _artifact, _pin = _national_inputs( + builder, monkeypatch, tmp_path + ) + module = _fake_evaluator(lambda kwargs: _receipt(kwargs), []) + monkeypatch.setattr( + builder, "_load_candidate_evaluator", lambda importer=None: module + ) + + assert ( + builder.main( + _argv( + input_h5, + tmp_path / "plan", + "--dry-run", + "--no-staging", + *_incumbent_argv(input_h5, tmp_path), + ) + ) + == 0 + ) + + plan = json.loads(capsys.readouterr().out) + assert plan["dry_run"] is True + assert plan["incumbent"]["label"] == "efrs_fixture" + assert plan["incumbent"]["sha256"] == _sha(tmp_path / "incumbent.h5") + assert not (tmp_path / "plan").exists() diff --git a/tools/build_uk_rowwise_candidate.py b/tools/build_uk_rowwise_candidate.py index fe25c6843..3123d43bc 100644 --- a/tools/build_uk_rowwise_candidate.py +++ b/tools/build_uk_rowwise_candidate.py @@ -22,6 +22,7 @@ import argparse import dataclasses import hashlib +import importlib import json import shutil import subprocess @@ -73,6 +74,7 @@ StagedDatasetBundle, disabled_staged_dataset, local_only_staged_dataset, + parse_sha256sums, refresh_sha256sums_entry, stage_bundle, write_sidecars, @@ -178,6 +180,8 @@ #: National-role outputs (the calibration seam's evidence shape). BUILD_RECORD_FILENAME = "build_record.json" NATIONAL_REGISTRY_FILENAME = "national_target_registry.json" +NATIONAL_CONTRACT_REGISTRY_FILENAME = "national_contract_registry.json" +SCORE_RECEIPT_FILENAME = "score_vs_incumbent.json" DENSE_REFERENCE_DIAGNOSTICS_FILENAME = "dense_reference_diagnostics.csv" DATASET_SIZE_SELECTION_FILENAME = "dataset_size_selection.csv" @@ -785,6 +789,26 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: "not the committed Chronicle pin (development runs)." ), ) + parser.add_argument( + "--incumbent-h5", + type=Path, + help=( + "National role only: the incumbent dataset the finished candidate " + "is evaluated against (microcosm#578 rule 1 on the surface both " + "can materialize). The evaluation runs after the bundle is staged " + "and never blocks the build; its receipt is what the release-cut " + "certifier reads." + ), + ) + parser.add_argument( + "--incumbent-sha256", + help="National role only: the incumbent's SHA-256, verified before it is read.", + ) + parser.add_argument( + "--incumbent-label", + default="enhanced_frs_2024_25", + help="National role only: the incumbent's label in the score receipt.", + ) parser.add_argument("--release-candidate", action="store_true") parser.add_argument( "--households-only", @@ -1832,6 +1856,7 @@ def _national_dry_run(args: argparse.Namespace) -> int: ), ), "engine": "not_run", + "incumbent": _incumbent_arguments(args), "releasable": False, } print(_json_text(plan)) @@ -1858,6 +1883,7 @@ def _run_national_role(args: argparse.Namespace) -> int: raise ValueError(f"--out must be a directory path, got {out_dir}.") input_artifact = _artifact_info(input_h5) _verify_requested_pin("--input-h5", input_artifact, requested=args.input_sha256) + incumbent = _incumbent_arguments(args) # The attempt id is minted before telemetry opens so the staging run id # and the Logbook row agree, as on the dense role. build_id = new_uk_calibration_attempt_id(timestamp=datetime.now(UTC)) @@ -1871,6 +1897,7 @@ def _run_national_role(args: argparse.Namespace) -> int: input_artifact=input_artifact, build_id=build_id, telemetry=telemetry, + incumbent=incumbent, ) except BaseException as error: _fail_staging_telemetry(telemetry, error) @@ -1886,6 +1913,7 @@ def _run_national_attempt( input_artifact: Mapping[str, Any], build_id: str, telemetry: StagingTelemetryV2 | None, + incumbent: Mapping[str, Any] | None = None, ) -> int: _stage( telemetry, @@ -1925,6 +1953,10 @@ def _run_national_attempt( # The frozen register is the scorer's input: the same artifact this run # solved against, by content hash. inputs["national_registry"].to_json(output_paths["national_registry"]) + # The full compiled register beside it: the band edges a pruned scoring + # surface must never redraw (#803), for the end-of-build evaluation and + # for a re-score by hand (--band-edge-registry-json). + inputs["band_edge_registry"].to_json(output_paths["contract_registry"]) resolver = UKMeasureResolver( simulation_source=input_h5, scratch_dir=out_dir, @@ -1970,8 +2002,22 @@ def publish_manifest() -> None: telemetry=telemetry, ) + def evaluate() -> None: + # After the bundle is staged (the evaluation never blocks staging), + # before the telemetry completes (the receipt rides it as an artifact). + evidence["evaluation"] = _evaluate_against_incumbent( + args, + incumbent=incumbent, + inputs=inputs, + output_paths=output_paths, + telemetry=telemetry, + calibration_year=calibration_year, + out_dir=out_dir, + ) + def finalize_staging() -> None: publish_manifest() + evaluate() _finalize_staging_telemetry(args, telemetry) def event_callback(stage_id: str, status: str, details: Mapping[str, Any]) -> None: @@ -2021,6 +2067,7 @@ def event_callback(stage_id: str, status: str, details: Mapping[str, Any]) -> No ) if telemetry is None: publish_manifest() + evaluate() manifest = evidence["manifest"] # The seam rewrote its record with the delivery summary after the # finalizer; the manifest binds the record as it now is. @@ -2031,11 +2078,18 @@ def event_callback(stage_id: str, status: str, details: Mapping[str, Any]) -> No } manifest["staging_delivery"] = _staging_delivery(telemetry) manifest["staged_dataset"] = evidence["staged_dataset"] + evaluation = evidence.get("evaluation", {"status": "not_requested"}) + manifest["evaluation"] = evaluation + if evaluation.get("status") == "completed": + manifest["outputs"]["score_receipt"] = evaluation["receipt"] _replace_manifest(output_paths["manifest"], manifest) if (out_dir / SHA256SUMS_FILENAME).is_file(): # The uploaded copies list the files as uploaded; the local sums - # list the record and the manifest as they now are, evidence included. + # list the record, the receipt and the manifest as they now are, + # evidence included. refresh_sha256sums_entry(out_dir, paths.build_record_json.name) + if evaluation.get("status") == "completed": + _list_sha256sums_entry(out_dir, output_paths["score_receipt"].name) refresh_sha256sums_entry(out_dir, output_paths["manifest"].name) print(_json_text(manifest)) return 0 @@ -2110,6 +2164,7 @@ def _national_manifest( "build_record": _artifact_info(build_record_path), "terminal_gate_report": _artifact_info(output_paths["terminal_gates"]), "national_target_registry": _artifact_info(output_paths["national_registry"]), + "national_contract_registry": _artifact_info(output_paths["contract_registry"]), } return { "schema_version": 4, @@ -4524,6 +4579,8 @@ def _refuse_national_role_arguments( refused.append("--target-loss-cap") if args.allow_unpinned_feed: refused.append("--allow-unpinned-feed") + if args.incumbent_h5 is not None or args.incumbent_sha256 is not None: + refused.append("--incumbent-h5/--incumbent-sha256") if args.target_weight_rule not in posture.allowed_target_weight_rules: refused.append(f"--target-weight-rule {args.target_weight_rule}") if refused: @@ -4533,6 +4590,160 @@ def _refuse_national_role_arguments( ) +def _load_candidate_evaluator(importer=importlib.import_module): + """The common-surface scorer (microcosm#967), loaded when an incumbent is given. + + Lazy so the driver imports without it; a build asked to evaluate refuses + up front, before any solve, when the scorer is not in the tree. + """ + + try: + return importer("microcosm.build.uk_runtime.candidate_score") + except ImportError as error: + raise ValueError( + "--incumbent-h5 needs the common-surface scorer (microcosm#967): " + "microcosm.build.uk_runtime.candidate_score is not in this tree." + ) from error + + +def _incumbent_arguments(args: argparse.Namespace) -> dict[str, Any] | None: + """The national role's optional incumbent, pinned and verified up front.""" + + if args.incumbent_h5 is None and args.incumbent_sha256 is None: + return None + if args.incumbent_h5 is None or args.incumbent_sha256 is None: + raise ValueError( + "--incumbent-h5 and --incumbent-sha256 must be given together." + ) + _load_candidate_evaluator() + incumbent_h5 = _require_file(args.incumbent_h5, label="--incumbent-h5") + info = _artifact_info(incumbent_h5) + _verify_requested_pin("--incumbent-h5", info, requested=args.incumbent_sha256) + return { + "path": str(incumbent_h5), + "sha256": info["sha256"], + "bytes": info["bytes"], + "label": str(args.incumbent_label), + } + + +def _list_sha256sums_entry(out_dir: Path, name: str) -> None: + """List a file the build wrote after the sidecars in the local sums.""" + + sums_path = out_dir / SHA256SUMS_FILENAME + entries = parse_sha256sums(sums_path.read_text(encoding="utf-8")) + if name in {listed for _, listed in entries}: + refresh_sha256sums_entry(out_dir, name) + return + digest = _artifact_info(out_dir / name)["sha256"] + sums_path.write_text( + sums_path.read_text(encoding="utf-8") + f"{digest} {name}\n", + encoding="utf-8", + ) + + +def _evaluate_against_incumbent( + args: argparse.Namespace, + *, + incumbent: Mapping[str, Any] | None, + inputs: Mapping[str, Any], + output_paths: Mapping[str, Path], + telemetry: StagingTelemetryV2 | None, + calibration_year: int, + out_dir: Path, +) -> dict[str, Any]: + """Score the finished candidate against the incumbent; never fail the build. + + Runs after the staged bundle is on the Hub and before the telemetry + completes, so the receipt rides the run as a reviewed artifact. Rows the + incumbent cannot materialize are pruned from both arms and warned about + by name; the verdict (microcosm#578 rule 1 on the common surface) is + what the release-cut certifier requires to be ``passed``. An error is + recorded and warned, never raised: staging and the exit code stand. + """ + + if incumbent is None: + return { + "status": "not_requested", + "note": ( + "no --incumbent-h5: the rule-1 score receipt the release-cut " + "certifier needs was not produced; score the candidate with " + "tools/score_uk_national_candidate.py before certification." + ), + } + _stage( + telemetry, + "incumbent_evaluation", + "started", + incumbent_sha256=incumbent["sha256"], + ) + candidate = _artifact_info(output_paths["dataset"]) + try: + module = _load_candidate_evaluator() + score = module.evaluate_uk_candidate_against_incumbent( + candidate_h5=output_paths["dataset"], + incumbent_h5=Path(incumbent["path"]), + candidate_sha256=candidate["sha256"], + incumbent_sha256=incumbent["sha256"], + target_registry=inputs["national_registry"], + calibration_year=calibration_year, + measure_resolver_factory=module._default_measure_resolver_factory( + out_dir, calibration_year + ), + candidate_label=output_paths["dataset"].stem, + incumbent_label=incumbent["label"], + band_edge_registry=inputs["band_edge_registry"], + ) + atomic_write_json(output_paths["score_receipt"], score) + except Exception as error: # noqa: BLE001 - the evaluation never fails a finished build + message = f"{type(error).__name__}: {error}"[:600] + print( + f"warning: the incumbent evaluation failed ({message}); the build's " + "evidence and staging are unaffected, and the candidate cannot be " + "certified until it is re-scored with " + "tools/score_uk_national_candidate.py.", + file=sys.stderr, + flush=True, + ) + _stage(telemetry, "incumbent_evaluation", "failed", error=message) + return { + "status": "error", + "error": message, + "incumbent": dict(incumbent), + "receipt": None, + } + warning = module.pruned_warning(score) + if warning is not None: + print(warning, file=sys.stderr, flush=True) + evaluation = score["evaluation"] + pruned = score["incumbent_unresolvable_pruned"] + _add_staging_artifact( + telemetry, + "score_vs_incumbent", + score, + artifact_kind="aggregate_diagnostics", + classification="aggregate", + ) + _stage( + telemetry, + "incumbent_evaluation", + "completed", + verdict=evaluation["verdict"], + n_scored=int(evaluation["scored_surface"]["n_scored"]), + n_pruned=int(evaluation["scored_surface"]["n_pruned"]), + ) + return { + "status": "completed", + "verdict": evaluation["verdict"], + "rule_1": dict(evaluation["rule_1"]), + "scored_surface": dict(evaluation["scored_surface"]), + "pruned_measures": list(pruned["measures"]), + "pruned_families": dict(pruned["families"]), + "receipt": _artifact_info(output_paths["score_receipt"]), + "incumbent": dict(incumbent), + } + + def _output_paths( out_dir: Path, *, @@ -4550,6 +4761,8 @@ def _output_paths( "build_record": out_dir / BUILD_RECORD_FILENAME, "terminal_gates": out_dir / posture.gate_report_filename(vintage), "national_registry": out_dir / NATIONAL_REGISTRY_FILENAME, + "contract_registry": out_dir / NATIONAL_CONTRACT_REGISTRY_FILENAME, + "score_receipt": out_dir / SCORE_RECEIPT_FILENAME, } return { "dataset": dataset, From 6104926e285faf98037c85d96334f01fcd456707 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:37:30 +0100 Subject: [PATCH 09/11] Stage the score receipt's verdict and aggregates, not its record arrays The reviewed telemetry artifact policy refuses any list of mappings, and the rehearsal build's receipt was refused on its per-target drift list, so the artifact never rode the run. The staged copy now drops the drift, the asymmetries and the resolution receipts and keeps the verdict, the pruned block and the aggregates; the full receipt stays beside the outputs and in the staged bundle. The test's receipt carries the arrays. Co-Authored-By: Claude Fable 5.1 --- .../tests/test_uk_rowwise_national_role.py | 24 +++++++++++++++++++ tools/build_uk_rowwise_candidate.py | 21 +++++++++++++++- 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py index 8dd089924..be74ec157 100644 --- a/packages/microcosm-build/tests/test_uk_rowwise_national_role.py +++ b/packages/microcosm-build/tests/test_uk_rowwise_national_role.py @@ -704,6 +704,21 @@ def _receipt(kwargs, *, verdict="passed", pruned_measures=()): "candidate_target_wins": 1, "incumbent_target_wins": 0, "register": {"n_specs": 1}, + # Record arrays the reviewed telemetry artifact must not carry. + "target_drift": [ + { + "target": "dwp.uc.households@2025", + "family": "dwp_universal_credit", + "candidate_relative_error": 0.01, + "incumbent_relative_error": 0.2, + "winner": "candidate", + } + ], + "signed_asymmetries": [{"id": "incumbent_own_registry", "description": "…"}], + "measure_resolution": { + "candidate": {"rounds": [{"round": 0}]}, + "incumbent": None, + }, "incumbent_unresolvable_pruned": { "n_pruned": n_pruned, "n_scored": 1, @@ -945,6 +960,15 @@ def test_uk_national_role_evaluates_after_staging_the_bundle( artifacts = out / "staging" / "runs" / run_id / "artifacts" staged_receipt = json.loads((artifacts / "score_vs_incumbent.json").read_text()) assert staged_receipt["evaluation"]["verdict"] == "passed" + # The staged copy keeps the verdict, the pruned block and the aggregates + # and drops the record arrays the reviewed-artifact policy refuses; the + # full receipt is beside the outputs. + assert "incumbent_unresolvable_pruned" in staged_receipt + assert staged_receipt["candidate_target_wins"] == 1 + for key in ("target_drift", "signed_asymmetries", "measure_resolution"): + assert key not in staged_receipt + full_receipt = json.loads((out / "score_vs_incumbent.json").read_text()) + assert full_receipt["target_drift"][0]["winner"] == "candidate" assert calls[0]["candidate_sha256"] == manifest["outputs"]["dataset"]["sha256"] diff --git a/tools/build_uk_rowwise_candidate.py b/tools/build_uk_rowwise_candidate.py index 3123d43bc..ed49cbd1f 100644 --- a/tools/build_uk_rowwise_candidate.py +++ b/tools/build_uk_rowwise_candidate.py @@ -4642,6 +4642,25 @@ def _list_sha256sums_entry(out_dir: Path, name: str) -> None: ) +#: Receipt blocks that are record arrays (lists of mappings): the reviewed +#: telemetry artifact policy refuses them, and the verdict, the pruned block +#: and the aggregates carry everything a reviewer reads. The full receipt stays +#: beside the outputs and in the staged bundle. +_SCORE_RECEIPT_TELEMETRY_EXCLUDED = ( + "target_drift", + "signed_asymmetries", + "measure_resolution", +) + + +def _score_receipt_telemetry_summary(score: Mapping[str, Any]) -> dict[str, Any]: + return { + key: value + for key, value in score.items() + if key not in _SCORE_RECEIPT_TELEMETRY_EXCLUDED + } + + def _evaluate_against_incumbent( args: argparse.Namespace, *, @@ -4720,7 +4739,7 @@ def _evaluate_against_incumbent( _add_staging_artifact( telemetry, "score_vs_incumbent", - score, + _score_receipt_telemetry_summary(score), artifact_kind="aggregate_diagnostics", classification="aggregate", ) From d5730fe318229854b9f43026e3cf891d77ce48bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:53:54 +0100 Subject: [PATCH 10/11] Record R3: the end-of-build evaluation passes; the first release-cut certification runs and names what blocks it Co-Authored-By: Claude Fable 5.1 --- experiments/823-uk-consolidation-receipts.md | 100 +++++++++++++++++-- 1 file changed, 90 insertions(+), 10 deletions(-) diff --git a/experiments/823-uk-consolidation-receipts.md b/experiments/823-uk-consolidation-receipts.md index 574d6b02d..8700a3296 100644 --- a/experiments/823-uk-consolidation-receipts.md +++ b/experiments/823-uk-consolidation-receipts.md @@ -62,14 +62,94 @@ Same inputs, code and doctrine as R1, remote staging on at the driver's default digest-verified and the fetched H5 is byte-equal to the local one (the copy was deleted after the check). -## R3 — certification of the cut (not run) +## R3 — the end-of-build evaluation and the first release-cut certification (2026-09-21) -The plan's step 3 scores the candidate against the eFRS 1.57.3 copy on the run's -frozen register with `tools/score_uk_national_candidate.py`. That scorer refuses -a register it cannot materialize on both sides, and 119 of the 638 targets are -unresolvable on the incumbent (the v20 head-to-head used the evaluation repo's -pass-2 scorer, which prunes them, receipted in its run). A repository-native -certification of a national cut therefore still needs a reviewed common -register or a pruning scorer in this repository; certification, assembly and -the inspect publication wait on that (and on María's go for anything that -reaches `releases/`). +María's direction (2026-09-21): certify national candidates; warn loudly, naming +every measure absent on the incumbent, instead of refusing; evaluate at the end +of every national build without blocking staging; never publish on an unpassed +evaluation. Code: scratch branch `uk-823-rehearsal-eval` (never pushed) = this +branch's `acd93c12` (the driver hook, PR #965) + PR #967's `92edabdd`, +`e0d80c37`, `4173ab38` (the v20 deferral, the scoring-route fix, the six A16 +retirements) + `9f81db51`/`a58b6c29` (the common-surface scorer, #967) + the +certifier pin fix (#967 `3f2fe52d`, picked as `d45e91be`). + +**Build.** `tools/build_uk_rowwise_candidate.py --release-role national` on +spine-s (`4d9752fdcd92…`) and feed `c5e5bf8`, no solve flags (1,500 epochs, +`family_equal`, learning rate 0.02, seed 0; override receipt `{}`), +`--staging-local-only`, `--incumbent-h5` the enhanced FRS 1.57.3 copy +(`ef34c1ae2821…`, label `enhanced_frs_2024_25_v1_57_3`). Attempt +`uk-frs-calibration-attempt-20260921T152658Z-6609d768`, 15:26:47Z → 15:35:30Z +(8m43s including the evaluation), exit 0. 644 targets (v20's 638 plus the six +retired A16 rows), 52,846 households, final loss 0.010453, max absolute relative +error 0.2524 (the signed 20-30k self-employment deferral, inside its window), +median 0.00109; the six calibration-seam gates passed. Output directory +`data/ukds/acceptance/823-consolidation/national-eval-rehearsal/`; the full +compiled register is frozen beside the solve register as +`national_contract_registry.json`. + +**Evaluation.** `score_vs_incumbent.json` (sha `749598dff826…`, listed in the +sums, `outputs.score_receipt`, manifest `evaluation` block; telemetry events +`dataset_staging` < `incumbent_evaluation` < `complete`). Verdict **passed**: +candidate full loss 0.009559 against the incumbent's 0.212540; 501 candidate +wins to 23 on the 524 common targets (register `2331a9386a2e`). 120 targets +pruned from both arms and named on stderr, as the v21 head-to-head had them: +`dwp_universal_credit` 93, `dwp_two_child_limit` 15, `ons_household_composition` +10, `hmrc_cgt` 2, over six measures the enhanced FRS never carried +(`benunit.uc_calibration_administrative_family_type`, `…_child_count`, +`…_has_child_under_one`, `benunit.uc_tcl_affected_benunit_proxy`, +`household.ons_household_type`, `person.capital_gains_asset_type`). One defect +surfaced: the telemetry copy of the receipt was refused as a record array (the +per-target drift list), so this run's telemetry carries no `score_vs_incumbent` +artifact; fixed on this branch as `6104926e` (the staged copy keeps the +verdict, the pruned block and the aggregates and drops the arrays). + +**Certification.** `tools/certify_uk_release_cut.py` with the spine H5, the +feed, the accepted input-mass reference +(`686-spine-swap/uk_input_mass_reference_2024_25_v1_56_16.json`) and the +receipt. The first attempt refused before any gate ran: the certifier's Logbook +pin roles carried a digest without `size_bytes`, so it had never run end to end +(fixed on #967 as `3f2fe52d`, with a stub-battery test). The second attempt +(`uk-frs-release-certification-attempt-20260921T153934Z`, 15:39Z → 16:50Z) ran +the 20-gate release-cut battery: **12 passed, 7 failed, 1 evidence absent**, +`GateBatteryBlockedError`, no certification composed (error receipt under +`logbook-receipts/`). Passed: `uk_aggregate_admin`, the three Ledger compile +parity gates, `uk_qrf_tail_concentration`, `uk_release_family_build_stages`, +`uk_release_input_coverage_manifest_current`, `uk_support`, +`uk_take_up_signal`, `uk_target_surface`, `uk_target_surface_local_default_2025`, +`uk_uc_capital_coherence`. The rest fall into three groups: + +- *Certifier evidence adaptation, not the candidate.* `uk_release_input_coverage` + (19 failures over 13 stage families: 13 "final household weights have kind + 'calibrated', expected reviewed kind 'importance'" and 6 "lacks the reviewed + mass-conserving MassChangeRecord": the gate re-evaluates stage-boundary + expectations on the calibrated frame). `uk_uc_deduction_combination_enum_domain` + and `uk_student_loan_plan_enum_domain` fail closed with + `'PolicyEngineUKCoverageEngine' object has no attribute '_variable'`: the + evaluator (`battery_bindings.py`) reads `engine._variable(column)` while the + certifier supplies the coverage engine, which exposes `variables()`, + `variable_entities()` and `default_values()`. `uk_weights_audit` is + evidence-absent because spine-s's sidecar carries no `fit_weight_records` + block (the rehydration returns `None` for a pre-#757 sidecar). +- *Stale reviewed registers.* `uk_export_surface` compares against + `UK_CANDIDATE_DATASET_NAME = "microcosm_uk_2024"` and the June allow-list: 49 + exported columns outside the enhanced FRS surface without an entry (person 32, + household 12, benunit 5: the `hmrc_spi_*` income components, reported benefit + flags, council tax, support-channel and clone ids, source keys), the calibrated + weight not exported as `household.household_weight`, and one reference-only + column to drop (`person.incapacity_benefit_reported`). `uk_input_mass_parity`: + two columns beyond the ±452 % fence (`dfe_education_spending` +187,609 %, + `jsa_income_reported` +909 %) and two stale exclusions now within tolerance + (`charitable_investment_gifts`, `owned_land`), over 126 checked columns and 65 + candidate-only ones. +- *Data findings.* `uk_degenerate_release_surface`: two persisted all-zero + columns (`person.incapacity_benefit_reported`, `person.is_in_approved_training`). + `uk_nonnegative_columns`: `housing_water_and_electricity_consumption` has 239 + values below zero (minimum −14,165). + +The evaluation path therefore works end to end and the receipt is certifiable +in shape; the cut does not certify today because the release-cut battery had +never been run on a candidate of this shape. María's calls: the three certifier +adaptations, the export allow-list and dataset name, the two input-mass +breaches and the two stale exclusions, the two all-zero columns and the negative +consumption values. Assembly and the inspect publication wait on a green +certification and her go. From 43a4acbcfadf8f1983851741b25643a100179d17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mar=C3=ADa=20Juaristi?= <127882282+juaristi22@users.noreply.github.com> Date: Mon, 21 Sep 2026 20:10:21 +0100 Subject: [PATCH 11/11] Record R4: the release-cut battery with the certifier-evidence fixes The R3 candidate re-certified with PR #967's four fold-in commits (stale exclusions retired, June CGT family retired with pass-through semantics, enum domains on both engines, the build-state half on the pinned spine frame, the fit-weight collector): 15 passed, 4 failed, 1 evidence absent. Input coverage and both enum gates pass; input-mass parity fails on the two breaches alone; the weights audit waits on a spine rebuilt from the fixed collector; the export allow-list and the two data findings are unchanged. Co-Authored-By: Claude Fable 5.1 --- experiments/823-uk-consolidation-receipts.md | 51 ++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/experiments/823-uk-consolidation-receipts.md b/experiments/823-uk-consolidation-receipts.md index 8700a3296..df618170c 100644 --- a/experiments/823-uk-consolidation-receipts.md +++ b/experiments/823-uk-consolidation-receipts.md @@ -153,3 +153,54 @@ adaptations, the export allow-list and dataset name, the two input-mass breaches and the two stale exclusions, the two all-zero columns and the negative consumption values. Assembly and the inspect publication wait on a green certification and her go. + +## R4 — the release-cut battery on the R3 candidate with the certifier-evidence fixes (2026-09-21) + +María's direction (2026-09-21): retire the stale exclusions, fix the fit-weight +records, give the coverage engine an enum-domain method, evaluate the +spine-stage contract on the spine's importance weights and the calibration +checks on the calibrated weights, and retire `hmrc_cgt_gains` from the June +path. Code: the four commits pushed to PR #967 (`447a24bd` stale exclusions, +`f6b8ce0c` June CGT family + pass-through semantics, `e77cc65c` enum-domain +accessor + build-state half on the spine frame + `--spine-sha256`, `dc42375b` +fit-weight collector), picked onto the scratch branch `uk-823-rehearsal-eval` +(never pushed) as `7222f78a`, `fd358654`, `be818e8e`, `d07ac0e7` on top of R3's +`d45e91be`. + +**Certification.** The same candidate and inputs as R3 (candidate +`5b88e4accbb1…`, spine-s `4d9752fdcd92…` now pinned by `--spine-sha256`, feed +`c5e5bf8`, the accepted input-mass reference, the R3 score receipt), outputs +written beside R3's as `microcosm_uk_2024_25.release_cut_gates.r4.json` (sha +`802290cefe08…`). Attempt +`uk-frs-release-certification-attempt-20260921T185339Z`, 18:53Z → 19:09Z +(16 minutes; R3 took 71), the 20-gate release-cut battery: **15 passed, 4 +failed, 1 evidence absent**, `GateBatteryBlockedError` at the terminal phase, +no certification composed. Against R3: + +- `uk_release_input_coverage` failed → **passed**: the family build-state half + read the spine frame (`details.family_build_state_frame: "spine"`), so the + 13 weight-kind failures are gone, and the 6 mass-record failures are gone + with the invented E5 reasons (five families now `weights_pass_through`) and + the retired June CGT family. The column and effective-mass halves still + read the calibrated release frame. +- `uk_uc_deduction_combination_enum_domain` and + `uk_student_loan_plan_enum_domain` failed-closed → **passed**: one column + each checked against the engine's enum domain through the new accessor, + zero invalid values. +- `uk_input_mass_parity` still fails, now on the two breaches alone + (`dfe_education_spending` +187,609 %, `jsa_income_reported` +909 %): + `stale_exclusions: []`, `expired_exclusions: []`, reviewed exclusions `{}`. +- `uk_weights_audit` still evidence-absent (`fit_weight_records`): the + collector fix lands in the spine builder, so the evidence appears only in a + spine built from `dc42375b` onward; spine-s cannot be retrofitted. +- Unchanged: `uk_export_surface` (June allow-list and + `UK_CANDIDATE_DATASET_NAME`), `uk_degenerate_release_surface` (two all-zero + columns), `uk_nonnegative_columns` (239 negative + `housing_water_and_electricity_consumption` values). The twelve R3 passes + still pass. + +The certifier-evidence group from R3 is closed except for the spine rebuild. +What blocks a certified cut is now entirely María's calls: the export +allow-list and dataset name, the two input-mass breaches, the two all-zero +columns, the negative consumption values, and a spine rebuild for the weights +audit.