Stage UK rowwise candidate builds: telemetry to runs/ and the dataset bundle to staged/

CLAUDE.md

@@ -100,7 +100,14 @@ see README "Releasing & alerts". Publication also refuses a release whose
 build recorded staging telemetry that never reached its repo
 (`--allow-missing-staging` overrides); a build that declared `--no-staging`
 publishes without the flag. Never publish or promote artifacts as a side
-effect of another task.
+effect of another task. A UK rowwise run's **staged** bundle
+(`staged/<run_id>/` in the private repository, written by the build itself)
+is inspection evidence, not a release: it never moves `releases/` or
+`latest.json` and is not loadable through the certified loader. The build's
+default is to upload that bundle (hundreds of megabytes of licensed microdata)
+to the private repository; when you run `tools/build_uk_rowwise_candidate.py`
+yourself, pass `--staging-local-only` unless the operator asked for a staged
+upload.
 
 The US native-SPM-role source-enrichment lane is a separate release type:
 `tools/build_us_spm_role_enrichment.py` creates a local candidate from the exact

README.md

@@ -63,6 +63,18 @@ This writes `progress.json`, `events.ndjson`, `calibration_progress.json`, and
 final candidate diagnostics under `runs/<run_id>/` without updating production
 `latest.json`.
 
+The UK commands (`tools/build_uk_frs_spine.py`,
+`tools/calibrate_uk_national_dataset.py`, `tools/build_uk_rowwise_candidate.py`)
+stage version 2 telemetry to `policyengine/populace-uk-staging` under the same
+switch. The rowwise candidate command also **stages the finished dataset
+bundle** it built, dense or exact-count, under `staged/<run_id>/` in the
+private `policyengine/populace-uk-private` repository so the team can inspect
+it without publishing it: `releases/` and `latest.json` are untouched, the
+release contract is not consulted, and a `releasable: false` size run stages
+like any other. Fetch a bundle with `tools/fetch_uk_staged_dataset.py`;
+re-stage a finished run directory with `tools/stage_uk_rowwise_candidate.py`.
+See [docs/uk-staging-operations.md](docs/uk-staging-operations.md).
+
 See [SYSTEM_REQUIREMENTS.md](SYSTEM_REQUIREMENTS.md) for the measured memory,
 disk, and CPU footprint of developing and building locally (and what to budget
 on a build machine — RAM is the binding constraint).

changelog.d/uk-dense-assembler-staging-evidence.changed.md

@@ -0,0 +1 @@
+`tools/assemble_uk_dense_release_dir.py` now requires the run manifest's `staging_delivery` receipt and copies it into `build_manifest.json` as `staging`, the national assembler's rule, so `microcosm-publish-release` applies its undelivered-staging refusal to dense cuts; `--allow-missing-staging` assembles a run built before the staging lane with a recorded disabled-staging opt-out.

changelog.d/uk-rowwise-staging.added.md

@@ -0,0 +1 @@
+Stage UK rowwise candidate builds (dense and `--dataset-households` runs) like the national command: version 2 telemetry to `runs/<run_id>/` with stage events, thinned per-epoch losses and delivery evidence in the manifest; the driver adopts the shared `--staging-*` options.

changelog.d/uk-staged-dataset-lane.added.md

@@ -0,0 +1 @@
+Add the staged-dataset lane: the finished rowwise bundle (every manifest-registered output, the manifest, `staged_manifest.json`, `sha256sums.txt`) is uploaded in one commit to `staged/<run_id>/` of the private UK dataset repository, keyed by the telemetry run id, with `staged_dataset` evidence in the manifest and reviewed `staged_dataset`/`fit_summary` artifacts in the telemetry run; `--staged-dataset-repo-id`, `--no-staged-dataset`, `tools/stage_uk_rowwise_candidate.py` and `tools/fetch_uk_staged_dataset.py`. A remote stage is refused up front without a Hub credential that can write the repository (read tokens and fine-grained tokens scoped elsewhere included); epoch telemetry is thinned to a bounded row count for any `--epochs`. Staging is not publication: `releases/` and `latest.json` are untouched.

docs/uk-dataset-size-plan-355.md

@@ -93,6 +93,21 @@ Never reduce `--n-clones` to request a smaller output. Full builds still need
 the dense build's peak memory and add L0/refit work; the reduction is in the
 exported dataset's storage and downstream loading/simulation footprint.
 
+Every run stages by default (see [UK staging operations](uk-staging-operations.md)):
+telemetry to `runs/<run_id>/` in `policyengine/populace-uk-staging` while it
+runs, and at the end the bundle the manifest registers, under
+`staged/<run_id>/` in `policyengine/populace-uk-private`, keyed by the same
+run id (the Logbook build id unless `--staging-run-id` is given). A size run
+stays `releasable=false`; staging does not change that, it makes the
+candidate inspectable. The run needs an ambient `HF_TOKEN` that can see the
+private repository, or `--staging-local-only` to keep everything on disk
+(`--no-staging` records a deliberate opt-out, `--no-staged-dataset` runs the
+telemetry alone). The command ends with a `staged dataset:` line naming the
+repository, prefix and revision, and the manifest carries `staging_delivery`
+and `staged_dataset`. Fetch a staged bundle with
+`tools/fetch_uk_staged_dataset.py --run-id <run_id>`; re-stage a finished
+directory with `tools/stage_uk_rowwise_candidate.py --run-dir <dir>`.
+
 ### The search stops on the draw's own feasibility; the solve is checkpointed before the draw (2026-09-09)
 
 S2 on spine-p (55,000 at `--selection-pi-hi 0.95`, 2,000 epochs) was refused at the

docs/uk-dense-release-assembly-runbook-762.md

@@ -105,8 +105,11 @@ uv run --no-sync python tools/assemble_uk_dense_release_dir.py \
 ```
 
 Assembly verifies the hash join (every manifest output against its bytes, the
-spine against its pin, the gate report against the Logbook build id), re-runs
-the candidate pre-flight, mints the cut tag
+spine against its pin, the gate report against the Logbook build id), requires
+the manifest's `staging_delivery` receipt (the run's version 2 staging
+telemetry evidence, copied into `build_manifest.json` as `staging` so
+publication can apply its undelivered-staging refusal, as on the national
+lane), re-runs the candidate pre-flight, mints the cut tag
 `microcosm-uk-2024-25-dense-<YYYYMMDDTHHMMSSZ>-<uuid8>` from the run's attempt
 id, clones the H5 beside itself as `microcosm_uk_2025_dense.h5`, stages
 `build_manifest.json`, `release_manifest.json`, `calibration_diagnostics.json`,

docs/uk-staging-operations.md

@@ -115,7 +115,9 @@ read-only credential.
 
 ## Command modes and files
 
-Both UK commands support these staging modes:
+The three UK commands (`tools/build_uk_frs_spine.py`,
+`tools/calibrate_uk_national_dataset.py` and
+`tools/build_uk_rowwise_candidate.py`) support these staging modes:
 
 - Default: local version 2 files plus best-effort delivery to
   `policyengine/populace-uk-staging`.
@@ -149,6 +151,86 @@ artifacts are permitted; population H5 files, NumPy archives, source survey
 tables, row-level extracts, archives, credentials, and environment data are
 rejected before remote storage is called.
 
+## Staged datasets
+
+Telemetry is not the dataset. The rowwise candidate command (dense K=15 and
+exact-count `--dataset-households` runs alike) also stages the bundle its
+manifest vouches for, so a run can be inspected by the team without being
+published. The two destinations share one run id:
+
+```text
+policyengine/populace-uk-staging   runs/<run_id>/...          telemetry (above)
+policyengine/populace-uk-private   staged/<run_id>/...        the bundle
+```
+
+The bundle is every file `rowwise_candidate_manifest.json` registers under
+`outputs` (the H5, `calibration_diagnostics.json`, the gate report, the CSVs
+and the registry), the manifest as built, `staged_manifest.json` (inventory
+with digests, summary fields and the telemetry run reference) and
+`sha256sums.txt`. Nothing else in the run directory is eligible: logs, the
+Logbook spool, checkpoints and evaluation trees stay local. The bundle is
+verified from disk against the manifest's own digests and written in one
+commit; the commit is recorded as the bundle's revision. `releases/` and
+`latest.json` are never touched, no tag is created, and the release contract is
+not consulted: a staged bundle is not a release and cannot be loaded as one.
+
+Modes follow the staging switch. The default uploads telemetry and the bundle;
+`--staging-local-only` keeps both on disk (the sidecars are still written);
+`--no-staging` disables both and records the opt-out; `--no-staged-dataset`
+runs telemetry alone, with the bundle neither inventoried nor uploaded. The
+repository is `--staged-dataset-repo-id` (environment
+`POPULACE_UK_STAGED_DATASET_REPO_ID`). Because the upload closes a multi-hour
+run, the command refuses to start a remote dataset stage without an ambient
+Hub credential (`HF_TOKEN`) that can see the repository **and** write it: a
+read token, or a fine-grained token scoped to another owner, is refused up
+front rather than by the Hub's 403 hours later (a fine-grained token needs
+`repo.write` on the repository or on the `policyengine` organisation). The
+telemetry stays best-effort with no pre-flight. Forwarded epochs are thinned to
+at most 2,400 rows per run, whatever `--epochs` says, so the telemetry files
+stay under the contract's 5 MiB cap; a content refusal from the telemetry is
+reported once and stops the forwarding without touching the solve.
+
+The manifest gains two evidence blocks after the bundle is on disk:
+`staging_delivery`, the validated version 2 telemetry receipt, and
+`staged_dataset` (`mode`, `repository`, `prefix`, `run_id`, `revision`,
+`status` in `uploaded`, `already_staged`, `failed`, `skipped`, a reviewed
+`error_code`, and the `files` inventory as a mapping). A failed upload is
+recorded with its code, warned on stderr, and never changes the build's exit
+code or Logbook disposition. The telemetry run declares two reviewed
+artifacts, `artifacts/staged_dataset.json` (the same block) and
+`artifacts/fit_summary.json` (loss, fit by family, gate verdicts, the size
+receipt without its per-row arrays), so a run in the dashboard points at its
+bundle.
+
+Re-stage a finished directory, including runs built before this lane existed
+or whose upload failed, with:
+
+```bash
+uv run python tools/stage_uk_rowwise_candidate.py --run-dir <run directory>
+```
+
+It is idempotent on the outputs' digests: a directory whose manifest already
+records the same outputs as uploaded is left untouched (the driver's revision
+stands); an identical remote bundle the manifest does not know about is
+recorded as `already_staged` with the bundle's own commit; a different bundle
+under the same run id is refused (`REMOTE_DIFFERS`). Fetch a bundle for a scorecard, an evaluation leg or the
+dashboard's local-directory mode with:
+
+```bash
+uv run python tools/fetch_uk_staged_dataset.py --run-id <run_id> --dest <dir>
+```
+
+Every file is checked against `sha256sums.txt`; `--h5-only` fetches the
+dataset alone. Rollback is configuration-first here too: `--no-staged-dataset`
+or `--staging-local-only` for new runs, and deleting `staged/<run_id>/` on the
+Hub for a bundle that must not remain (its manifest keeps the record). The
+dense release assembler applies the national rule to this command's runs: it
+requires the manifest's `staging_delivery` receipt and copies it into
+`build_manifest.json` as `staging`, where publication reads it. A dense run
+built before this lane carries no receipt; `--allow-missing-staging` assembles
+it with a recorded disabled-staging opt-out naming the override, the same
+posture publication's `--allow-missing-staging` grants.
+
 ## Smoke verification
 
 The UK spine command keeps fractional input sampling for scale tests. The