From d8a6fc358fd7941b9532d6ae89b5bc3997f7a615 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 17:33:37 -0400 Subject: [PATCH 01/75] Open the lane journal with what has been read and the first finding Co-Authored-By: Claude Opus 5 --- PROGRESS.md | 1639 +-------------------------------------------------- 1 file changed, 30 insertions(+), 1609 deletions(-) diff --git a/PROGRESS.md b/PROGRESS.md index 54c4c863c..90c30e851 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,1623 +1,44 @@ -# Publisher compatibility range at source-enrichment certification +# Lane journal — native retention seal (`native-retention-seal`) -Lane: `max/certify-compatible-model-range-20260914`, off `origin/main` at -`18271b28d`. Started 2026-09-14. Everything below the `---` rule at the end of -this section is prior-lane history; see "Root journals are history, not state" -in `CLAUDE.md`. +Branch `native-retention-seal`, from `origin/native-scale-transport` at +`a64f7b733`. Worktree `~/PolicyEngine/_worktrees/microcosm-native-retention`. -## State - -Pushed as draft PR -[#928](https://github.com/PolicyEngine/microcosm/pull/928). No publication of -any kind: this lane changes producer/validator source only, and builds, -certifies and publishes no artifact. - -An adversarial review lane run against this worktree wrote into it while it -worked: a reviewer checked out `origin/main` copies of the changed files for a -byte-for-byte default-path comparison, and a `git add -A` in this session -committed and pushed that reverted tree as `4b0ae3624`, briefly deleting the -feature from the PR and committing a reviewer's scratch test module. The branch -was reset to `8f82d0c6a` and the two intended commits reapplied; `git diff -8f82d0c6a ` is now exactly the docs and journal changes, with the feature -and test files byte-identical to `8f82d0c6a`. Review lanes must run in their own -worktree, not this one. - -## Problem - -`certify_source_enrichment` writes -`compatible_{model,core}_packages = [{"name": pkg, "specifier": "=="}]` -over whatever the candidate manifest held (`source_enrichment.py:940-942`), and -`_check_compatibility` then requires exactly that list at every later validation, -including publish preflight (`source_enrichment.py:610-615`). The contract layer -(`contract.py::_check_compatible_package_entries`) and both consumers (Microcosm -`loader.py::_package_certification`, policyengine.py -`provenance/certification.py::validate_release_manifest`) already accept any PEP -440 specifier set that contains the built-with version — so the exact pin is a -producer-tooling choice, not a schema limit. The consequence: each country patch -release moves the binding rather than widening it, and a data release whose H5 -bytes are unchanged still needs re-certification. - -## Plan - -1. Add a validated publisher claim at certification time, recorded in both - `release_manifest.json` and `source_enrichment.json`, with the default path - byte-identical to today. -2. Relax the certification-time equality gate to "equals the default exact pin, - or equals the claim the report records", keeping every other guard. -3. Tests for accept/reject/default/consumer-read. -4. Docs: when a range is legitimate and when it is not. - -## Done - -- Read both gates, the contract layer, both consumers and the 2026-09-12 dry-run - report that motivated the change. -- `--compatible-model-specifier` / `--compatibility-claim-declared-by` on the - source-enrichment CLI and `certify_source_enrichment`, validated by - `parse_compatibility_claim_requirement` (PEP 508, names the built-with - package, no URL/extras/marker) and `compatibility_claim_entry` (valid and - non-empty PEP 440 set, contains the tested version under the consumers' own - containment, bounded above, accountable declarer). -- The claim recorded in `source_enrichment.json` - (`compatibility.publisher_claims.model`) and `release_manifest.json` - (`compatible_model_packages[0]`, `basis: publisher_claim`, `declared_by`), - cross-checked at every later validation so a manifest widened after - certification has no declaration behind it. -- `contract.py` refuses a `publisher_claim` basis with no declarer, and a - declarer with no basis, for every release type. -- 20 tests across `test_source_enrichment.py` and `test_contract.py`; - `packages/microcosm-data/tests/` 550 passed, 2 skipped; `ruff check .` clean; - `tools/ci_test_groups.py --verify` ok. -- Docs: a "Declaring a publisher compatibility range" section in - `docs/us-native-spm-role-source-enrichment.md` with when to use a range and - when not to, a note in the `microcosm-data` README, and a changelog fragment. - -## Second session, 2026-09-14 evening - -The lane was re-entered after the first session ended at `a4e7e131c`. Nothing -was rewritten: the feature and test files are untouched, and this session's job -was to confirm the branch rather than extend it. Done here: - -- `packages/microcosm-data/tests/` re-run from scratch in the lane venv: 565 - passed, 2 skipped, matching what the PR body claims. -- A second, independent adversarial pass over the branch, run out of two - detached review worktrees so no reviewer could write into this one (the - collision recorded above must not repeat): PEP 440 containment parity against - both consumers, guard-preservation and bypass, wrapper accept/refuse measured - rather than described, mutation testing of every new guard, a claims audit of - the PR body and docs, and a blast-radius sweep for anything that reads - `compatible_model_packages`. - -## Third session, 2026-09-14 — second-pass review, part two - -The second-pass review (`review-928-r2.md`, written against `7cb8eae6e`) -confirmed the six earlier fixes and left five items. This session applies them, -each with a test that fails before and passes after. - -- **M1** — the boundedness guard bounds a claim above and never below, so - `policyengine-us<2.1` over a 2.0.1 build is accepted and certifies a consumer - running 0.9.0. Add a lower-bound probe. -- **L1** — `compatibility.narrowed_claims` is written and never read; surface it - in the validation/preflight output. -- **L2** — the "pass the flags" remediation suffix fires even when the flags - were passed this run; gate it on the claim being absent. -- **L3** — the narrowing loop calls a Core pin change a narrowed "claim", - although no producer can declare a Core range; reword. -- **I2** — doc only: a prerelease built-with version cannot carry a range, so - the exact default pin is the only option there. (Filed as such; the premise - did not survive measurement — see "Done".) - -### Done - -- **M1.** A third boundedness probe in `compatibility_claim_entry` asks whether - the claim still admits `Version(f"{tested.epoch}!0")`. Measured first: - `Version("0") in SpecifierSet("<2.1")` is `True`, and `False` for - `>=2.0.1,<2.1`, `~=2.0.1`, `==2.0.*` and `>=2.0.1,<3`. `<2.1` and `<=2.0.5` - moved from the accepted parameters to the refused ones, a - certification-level `policyengine-us<2` case was added, and the next-major - error text now cites `'>=2.0.1,<2.1'` instead of the `'<2.1'` the new probe - refuses. Four tests failed before, pass after. The probe carries the tested - version's epoch because a claim may mix epochs: over a `1!2.0.1` build, - `>=2.0.1,<1!2.1` admits `1!0` while excluding a bare `Version("0")`. The - first draft justified that backwards — claiming an epoch-0 zero sits outside - an epoch-bearing claim, when `Version("0") in SpecifierSet("<1!2.1")` is - `True` — and its test survived replacing the probe with a bare - `Version("0")`. Corrected after the third pass; the test now uses the - mixed-epoch claim and kills that mutant. -- **L1.** `recorded_narrowed_claims` reads the record back, and both validation - (`python -m microcosm.data.source_enrichment` without `--certify`) and - `microcosm-publish-release --preflight-only` print `narrowed_claims` beside - their verdict when a bundle carries one; publication repeats it on stderr, - since reaching publication does not require running the preflight first. - It reports rather than gates: an absent or malformed record reads as no - record. Three tests failed before. -- **L2.** The "pass the flags" suffix is gated on `claim_specifier is None`. - Both branches tested through a re-certification that tightens a declared - range (`>=1.998.0,<2` → `>=1.999.0,<2`): warns, no suffix. -- **L3.** Message construction moved to `_narrowing_notice`; Core reads "moves - the policyengine-core compatibility pin". The Core branch turns out to be - unreachable through `certify_source_enrichment` — the input gate re-runs the - loader qualification and requires the recorded receipt to equal the runtime, - so a moved Core version is refused first. Both the wording and that wall are - now pinned by tests. -- **I2.** The review's premise was wrong and the docs say the accurate thing - instead. Installed `packaging` 26.2 matches prereleases by default - (`SpecifierSet.contains` documents it; `Version("2.1.0rc1") in - SpecifierSet(">=2.0.1,<2.2")` is `True`). The real constraint is ordering: a - prerelease sorts below its own release, so `>=2.0.1,<2.1` and `~=2.0.1` - exclude a `2.0.1rc1` build while `>=2.0.1rc1,<2.1` and `==2.0.*` reach it and - pass all three probes. A characterization test pins all five outcomes; it - passes before and after. -- `packages/microcosm-data/tests/` 611 passed, 2 skipped; the two named files - 379 passed; `ruff check` and `ruff format --check` clean on - `packages/microcosm-data` and on every changed `.py`; repo-wide `ruff check` - clean; `tools/ci_test_groups.py --verify` ok. - -### Third adversarial pass - -Five read-only reviewers over the five changes (guard correctness, reporting -path, wording and reachability, docs/claims audit, test quality), each finding -put to an independent refuter: 33 raised, 2 survived. Both are fixed. - -- **The epoch rationale was a fabricated mechanism, and its test was inert.** - The comment and docstring said an epoch-0 zero sits outside an epoch-bearing - claim; `Version("0") in SpecifierSet("<1!2.1")` is `True`. The verifier - mutated the probe to a bare `Version("0")` and the whole file stayed green — - the test named for the epoch carry survived dropping it. The carry is - justified by mixed-epoch claims instead (`>=2.0.1,<1!2.1` over `1!2.0.1` - admits `1!0` and not `0`), which is the case the test now uses; the mutant - fails it. -- **The record reached the preflight and not the publish run.** Already closed - mid-flight, before the pass reported it: publication repeats it on stderr, - since `tools/publish_release.sh` passes its arguments straight through and - the runbook's "remove `--preflight-only`" step is a habit, not a gate. - -Two refuted findings were worth acting on anyway. The documented probe residue -is now executable on both sides (`<2.1,!=0` admits `0.9.0` exactly as -`>=2.0.1,!=3.0.0,!=99999.0.0` admits `5.0`), so the sentence describing the -guard's limit cannot drift from it. And `--certify` now reports the narrowing -it caused in its own verdict, with `_narrowed_claims` giving all four verdicts -one tolerance so none can report a bundle differently from the others. - -### Next - -- PR CI. Do not merge; do not mark ready. - -## Next - -- Whole-workspace run and PR CI to finish; hand to human review. Do not merge; - do not publish. - -# Grouped solver x calibration target snapshots integration - 2026-09-12 - -Historical note, 12 September 2026: this grouped-lane journal was subsequently -integrated with PR #914's corrected iteration identities and the actual fiscal -host observer. Its 443-test evidence and statements that host wiring is absent -describe that earlier lane. Current source and acceptance are recorded in -[the fiscal-host evidence](experiments/fiscal-target-snapshot-host-20260912.md). - -Lane: `microcosm-grouped-target-snapshot-integration-20260912`, branch -`grouped-target-snapshot-integration-20260912`. Base: fresh `origin/main` -`116d46ee9dc2aafdc68259b7c06e4c3462522e8b` (unmoved; it is exactly the shared -base both reviewed heads were cut from). Integrates the exact reviewed heads -`536f1ceefcdafda3cc619c14b4da18e012a7be57` (G, US grouped/fixed-zero Adam) and -`b43369dc49e175803f62020cc1e72fa53926aed8` (S, calibration target snapshots, -PR #914) against the read-only checklist -`grouped-snapshot-integration-review.md`. - -Everything below this section is prior-lane history and was accurate when -written; see "Root journals are history, not state" in `CLAUDE.md`. - -## State - -Complete and local. Four commits: the merge, the grouped instrumentation, the -identity recomputation, and the cross-product controls. Nothing pushed, no PR, -no release action. Root reviews and integrates. - -Scope is the shared solver seam only. `graph_fiscal_dense_calibration.py` still -calls `calibrate` without an observer, so this branch emits no snapshot for the -real US fiscal path; that host wiring is a separate step, and no observer -registry, `Node.params` callback or replay-reruns-the-optimizer claim was -invented here. See -[the lane experiment](experiments/grouped-target-snapshot-integration-20260912.md) -for scope, evidence and residual risks. - -## Done - -- Fetched `origin/main`; confirmed it is still `116d46ee9`, so no main drift - had to be preserved. Recorded as the integration base. -- Created this worktree on a new branch from that base; fast-forwarded to G, - then merged S. Verified all four pinned source hashes - (`solve.py` and `calibrate/__init__.py` on both heads) match - `grouped-snapshot-integration-review-pins.json` byte for byte before merging. -- `calibrate/__init__.py` auto-merged as a true union: `GroupedUpperBounds` - export retained alongside every snapshot export. -- `solve.py`: the single textual conflict was the `calibrate()` signature; - resolved as a union of S's `target_snapshots` and G's - `grouped_upper_bounds` / `grouped_preserve_zeros` / - `_post_projection_observer`. -- `test_us_multispine_pool_tool.py`: resolved semantically. G's unrelated US - integration delta (schema_version 2 PUMA-ladder fixture with joint - PUMA/tract/CD overlap arrays and per-layer `source` labels) merged cleanly - and is retained; the only textual conflict was the `spec_sha256` pin. -- The five source-derived identity conflicts - (`inventory_coverage.py` EXPECTED_HASHES, `us-f0-coverage.json`, - `test_spec_engine_loader.py` golden, the multispine `spec_sha256`, and the - calibrate parity `pins.json`) carry a placeholder in this merge commit. None - of them is an ours/theirs decision: both sides' values describe their own - tree, and neither describes the merged one. They are recomputed against the - final merged sources in a later commit on this branch. - -## Done (continued) - -- Instrumented `_optimize_grouped`: the grouped early return in `_optimize` - bypassed every hook S added, so grouped runs emitted only a closing snapshot. - The in-loop emission sits after the progress callback and before `backward()` - and reads the exact float32 estimate tensor the epoch's loss was computed - from. Returned weights, trajectory and RNG state are bit-identical with the - observer on and off, and no evaluation is added. -- Separated retain-best detection from the receipt for grouped runs, so a later - change that populated a grouped receipt cannot make the reused final emitter - claim a retained best. The receipt stays empty. -- Recomputed six source-derived identity artifacts against this checkout (the - five conflicted ones plus the country-bundle digests, which were not - conflicted because only G had touched them but which move for the same - reason). No value equals either branch's. The simulate and fit.qrf pins were - deliberately left alone. -- Added the checklist's cross product to the three existing grouped test files - rather than a new one, so every case reuses fixtures already there. - -## Next - -Root's independent review and integration. Open follow-ups, none owned here: -the host observer seam for the US fiscal dense calibration path, a country-scale -cadence choice backed by a real measurement, and the dashboard consumer. - ---- - -# #893 reconciliation to main's amended graph interface (amendments 19 and 20) - -Lane: `microcosm-us-launch-verified-lanes-20260910` (the live integration -worktree for PR #893, branch `microcosm-us-launch-integration-20260909`). -Started 2026-09-12 at `069d5ed9a`. Everything below the `---` rule at the end -of this section is prior-lane history; see "Root journals are history, not -state" in `CLAUDE.md`. - -## State - -Merges done, four graph pieces re-applied, graph package green (518 passed), -fit green (191), spec check / groups verify / ruff clean. Build-side consumer -suites running at the time of this entry; final results in `out.md` §5 and -`experiments/893-reconciliation-amendments-19-20-20260912.md`. Nothing -pushed, no PR, no new branch, `uv.lock` untouched. - -## Done - -- Read `CLAUDE.md`, the charter's "Interface freeze" at `23ba24770`, PR - #893's body, the Amendment 19 lane's "Scope" list; diffed the graph package - against `23ba24770` per file; AST-scanned every non-graph consumer. -- `uv sync --all-packages --locked --extra us --extra uk` exit 0. Baseline - graph suite: 5 failed / 362 passed (the five the brief names). -- `3010b7788` merge `origin/main`: graph package, fit sources, lock and - charter resolved to main's bytes; branch-only `attachments.py`, - `availability.py`, `schema.py` and the two branch-only graph tests removed - in the merge, to return only where consumed. -- `051357909` merge `23ba24770` (amendment 20, merged from the branch head - because #912 was still on CI; the dispatcher re-runs `git merge origin/main` - after it lands — expected no-op for graph/fit/lock/charter). -- `dc621c14c` seed digests re-pinned (the branch's `acs_transfer` and - `housing_inputs` plus amendment 20's `fit.qrf` move them); coverage report - regenerated; `--check` exit 0. -- `cff8fbf32` calibrate/simulate H1 pins re-recorded (the branch's solver and - `Frame.__reduce__` changes move them); parity files 27 passed. -- Pieces re-applied on main's files, each with graph-level tests the branch - never had: `752ab840f` raw-byte codec (13 US consumers; 8 codec tests), - `db1b7821a` Frame-metadata store (3 named consumers; 274 passed across the - store/population/executor/manifest files), `d1019762b` execution states - (the post-clone geography gate's typed artifact; replaces the amendment-19 - gate refusal and its test; whole graph package 518 passed), `a16eeacf8` - population observer (4 consumers; 94 passed on the executor + B/F files). -- Dropped for lack of a consumer: lazy retention (`attachments.py`, layer - 08), `schema.py`, `keys._stream_file`, the `_write_node` refactor. -- Layer map computed from `git log --name-only` attribution plus an AST - import scan with hard / name-hard / soft link classes; written to `out.md`. - -## Next +Implements Max's 2026-09-17 decision on the transport lane's report §10 +question 2: **option (b) — replace the object comparison with a content +seal**, plus 1/15 now and 1/10 when the machine is quiet, JSON segment bodies +kept, and the four row-count ceilings left to another lane. -- Record the build-side consumer results in `out.md` §5, commit the - `experiments/` copy of the report, leave `out.md` uncommitted (it is - another lane's tracked report; see the memory note). -- For Max: piece C supersedes amendment 19's gate refusal (`out.md` §8.1); - the dropped lazy retention / `_stream_file` / `_write_node` pieces (§8.2–3). - ---- - -# Amendment 19 — typed opaque artifacts on the graph interface - -Lane: `amend-typed-artifacts`, off `origin/main` at `3094bfe84`. Started -2026-09-11. Everything below the `---` rule at the end of this section is -prior-lane history; see "Root journals are history, not state" in -`CLAUDE.md`. - -## State - -Landed, reviewed, and re-verified from scratch on `amend-typed-artifacts`. The -whole-workspace run is the last command outstanding; every gate the brief names -has been re-run green in this session. Nothing pushed, no PR, no branches -created, `uv.lock` untouched. - -> Historicized 2026-09-12: the branch was pushed as PR #911 on 2026-09-11 and -> peer-gated; the whole-workspace run above was superseded by the PR's CI. The -> interface lock is now enforced — `graph-interface-lock-test` merged as #910 -> on 2026-09-11 and this branch passes it. The paragraphs below are the lane's -> record as written, not current state. - -## Scope (what is in, and what is deliberately out) - -In, from `git diff origin/main origin/microcosm-us-launch-integration-20260909 --- packages/microcosm-graph/src`: - -- `decl.py`: `ArtifactType`, `ArtifactOutput`, `ArtifactInput`, - `Node.artifact_inputs` / `Node.artifact_outputs`, their validation, their - elision from the canonical projection when empty, and the artifact-edge - arm of `compile_graph`. -- `kernel.py`: `ArtifactValue` and `KernelContext.artifacts`. -- `artifact_edges.py` (new): numeric scope payloads, scope compatibility, - typed descriptors, `typed_contracts`, `value_from_descriptor`. -- `keys.py`: `opaque_artifact_key` and the `typed_artifacts` term in - `node_key`. -- `serialize.py`, `view.py`, `manifest.py`, `executor.py`: the minimal - support for the executor to honour declared artifact inputs/outputs. - -Out, because it is not needed for artifacts (each is its own lane): - -- `SeedSource.KEYED` and `randomness.py` (`keyed_uniform`). -- `availability.py` / execution state / `unreached` / `blocked_by` / - `gate_exception` propagation, and manifest schema 4. -- `attachments.py`, `_PopulationRetention`, lazy populations, - `_population_observer`. -- `store.py` Frame-metadata storage (`microcosm-graph-frame-v2`) and the - non-finite JSON decode hooks. -- `keys.py` `_stream_file` chunked source hashing. -- `codecs.py` `SourceBytesCodec` / `load_source_bytes`; `schema.py`. -- The `_write_node` per-coordinate memory refactor. - -## Done - -- Read `CLAUDE.md`, `docs/graph-acceptance.md`, `DESIGN.md`, and the - amendment-17 precedent (`cdbf71888`, `80b63ba14`, `ed36f6cb3`). -- Measured the branch diff per file and fixed the in/out boundary above. -- `uv sync --all-packages --locked --extra us --extra uk` → exit 0. -- Captured the baseline node keys of the three toy acceptance graphs before - touching any source, so the node-key answer is measured, not asserted. -- `68a6ecc4b` (red, exit 2, 4 collection errors) → `e591c52d7`: the frozen - declaration interface, `compile_graph`'s artifact edge, the elided - canonical projection, `opaque_artifact_key`, serialization, and the view. -- `a2b6dfb0b`: the acceptance suite's B2 `KernelContext` field set, as its - own commit, matching `80b63ba14`. -- `1cce8eceb` (red, **8 of 8 failing** — the commit message and an earlier - version of this line both say 7, which is wrong; see the correction under - "Re-verification") → `15f9d2c67`: `artifact_edges.py` and - the executor, cache record, and manifest support. -- `38b9a9e4d`: amendment 19 in the charter, the relock, the changelog - fragment. `8c2e7faab`: the graph explorer's receipt payload. -- Node keys re-measured after the change: byte-identical for all 20 nodes - of the three toy graphs. -- `packages/microcosm-graph/tests` 359 passed, exit 0. - `tools/ci_test_groups.py --verify` ok, `tools/spec_engine_coverage.py - --check` 42156/42156 + 41/41, `tools/graph_acceptance_burndown.py - --verify` ok, `ruff check` and `ruff format --check` clean. - -- Ran a five-dimension adversarial review of the extraction against the - integration branch (fidelity/minimality, executor paths, identity and - store, manifest provenance, charter/lock/changelog), each finding put to - two skeptics. Nine findings; seven real and fixed here: - `3a0726f93` (a corrupt typed manifest surfaced as `NodeRejectedError` - rather than `StoreCorruptError`), `9ebb60e4b` (`ArtifactValue.key` - described as a content identity it is not; relock), `402d9a631` (a - malformed `gate_ancestry` regressed to a bare `TypeError` on manifests - with no artifacts at all; charter graph list corrected; the F2 sentence - split into its two mechanisms), `5c4a8efde` (the artifact miss decision - moved back inside the recompute fallback), `9018c4420` (the - identity-preservation claim corrected, and payloads read only on the path - that runs a kernel). Two were the documented decisions and stand. -- Added coverage the review motivated: manifest ancestry authentication, - the F2-over-bytes path, cross-version edges under all three resume - policies, every artifact declaration field being normative, A3 through a - byte edge, and a cache hit that reads no payload. - -## Re-verification, 2026-09-11 (independent of the landing session) - -Everything below was re-run from a clean read of the tree, not carried over -from the landing session's notes. - -- The node-key answer re-measured with a script that varies only the - graph-kernel code: `microcosm.graph` resolved once from this branch and once - from `origin/main`'s sources (shadowed through `PYTHONPATH`, confirmed by the - loaded `decl.py` hash `635fef92...` on the main run), with `microcosm.build` - identical in both. Six graphs, 5+5+6+9+41+8 = 74 nodes: **all 74 node keys and - all 74 canonical projections byte-identical.** The amendment's per-graph counts - are each correct. -- `docs/graph-interface.lock` re-checked against `shasum -a 256` of the two - frozen files: both match. -- Re-run green: `packages/microcosm-graph/tests` 370 passed exit 0; the - acceptance subset 113 passed exit 0; `test_graph_kernel_contract.py` 15 passed - exit 0; the `KernelContext(` consumers (calibrate/fit/frame `test_kernels.py` - plus `test_us_graph.py`, `test_uk_graph.py`) 36 passed exit 0. - `tools/ci_test_groups.py --verify` ok, `tools/spec_engine_coverage.py --check` - 42156/42156 + 41/41, `tools/graph_acceptance_burndown.py --verify` ok, - `ruff check .` clean — all exit 0. -- No consumer constructs `KernelContext` positionally: all five non-test sites - use keyword arguments, so the new field's placement could not have broken one. -- `ruff format --check .` exits 1 on 81 pre-existing files, none of them touched - by this lane (all 17 changed Python files pass `ruff format --check` - individually). CI's lint lane runs only `ruff check .`, so this is repo drift, - not a gate this lane moved. -- The red commit `1cce8eceb` records "Red: 7 of 8 fail against the executor as - it stands", and this journal repeated it. **It was 8 of 8.** Measured by - extracting the whole tree at `1cce8eceb` (`git archive | tar -x`), pointing - `PYTHONPATH` at that tree's six shard `src` directories (confirmed: - `microcosm.graph.executor` resolves into the extract, and - `microcosm.graph.artifact_edges` has no spec there, so the executor support - genuinely had not landed), and running the commit's own - `test_graph_executor.py` against its own sources: **8 failed, 63 passed**, the - 8 being exactly the amendment-19 tests the commit added. Red-first discipline - holds — the commit was redder than claimed — but the count in its message is - wrong and stays wrong, because rewriting landed history to fix a tally would - be worse than recording the correction here. -- An adversarial audit line-traced the new module: eleven non-docstring - statements of `artifact_edges.py` never executed in the whole graph suite, all - on the foreign-provenance parsing surface. Closed in `e3f69a4c4` with three - tests through the public `NodeReceipt`/`RunManifest` surface; the trace now - reports zero. The same trace showed `run_graph`'s consumer-side receipt - comparison is unreachable as a refusal — both skeptics confirmed the charter's - wording claims only the check's ordering, which does execute — so the branch - is now commented the way this file already marks such guards, rather than - chased with a test that cannot be written honestly. -- `packages/microcosm-build/tests/test_release_target_parity.py` fails two - tests locally. **Not this lane, and not CI**: both are guarded by - `_feed_or_skip` on a 131 MB pinned feed that lives *outside the repository* - (`~/PolicyEngine/_buildh-runtime/inputs/consumer_facts_buildn_v9_4.jsonl`, - dated 2026-07-23), so CI skips them; the local artifact predates #855's - hierarchy-label requirement. The same two fail identically with `origin/main`'s - graph sources swapped in, and `ledger_targets.py` imports no - `microcosm.graph`. This is the US twin of the UK instance the #791 lane already - recorded in `experiments/791-household-composition-receipts.md:111`. - -## Next - -- Whole-workspace `uv run pytest` is the only command still running. Every gate - the brief names is green at `HEAD`, and the complete set of tests this change - can reach — the graph package (373) plus the 8 test files outside it that - import `microcosm.graph` directly or through the seven source modules that do - (81) — is green at exit 0. The lane report is in `out.md`. -- For Max, in `out.md` §8: the gate-artifact-output refusal is the one interim - ruling this lane made on his behalf; the interface lock had **no automated - enforcement** when this was written (the charter's freeze was a human gate, - which is how the integration branch changed both frozen files unnoticed) — - the sibling branch `graph-interface-lock-test` (`8bd6e05ec`) added the test - and merged as #910 on 2026-09-11; this branch passes it, exit 0; plus the pre-existing - `ruff format` drift and the stale local `_buildh-runtime` feed. - ---- - -# Issue #907 — population `_storage_parts` hashes object dtype by pointer - -Lane: `fix-907-population-stamp-object-storage`, branched from -`origin/main` at `295130c9f901e08db11457f16dbdee4e2349c5ba` on 2026-09-11. -Report: `/Users/maxghenis/PolicyEngine/_recovered/scratch-backup/893/lanes/out-907-build-r1.md`. - -## State - -Implementation complete on `fix-907-population-stamp-object-storage`; ten -commits, nothing pushed, no new branch, no stash. `decl.py`, `kernel.py`, -`docs/graph-interface.lock`, every `test_acceptance_*`, `uv.lock`, spec pins -and evidence JSON are untouched — `shasum -a 256` on `decl.py`/`kernel.py` -still matches the lock byte for byte. - -## Done - -- Read `CLAUDE.md` and `docs/shared-constants.md`; confirmed the lane rules. -- Reproduced #907 directly, and end to end: two independently constructed - equal object-dtype Series give different `_storage_parts` value bytes - (PyObject addresses), which on `origin/main` surfaces as a **spurious** - `PopulationError: Structural node 'n1' changed carried storage in - household.tenure_type` for content that did not change. -- Landed the red regression in `packages/microcosm-graph/tests/` - (`test_graph_population.py`, an already-tracked file, so - `tools/ci_test_groups.py` needed no change). Final tests against the pre-fix - source: 34 failed, 24 passed, 46 deselected; direct exit 1. -- Fixed `_storage_parts` by routing any materialized object array through a - length-prefixed encoding whose body is `store._encode_object_scalar` — the - graph package's existing object-leaf codec, the one `ContentStore` writes - and reads back. A second, parallel vocabulary would have left a column - unequal to its own persisted-and-reloaded self for `pd.NA`, `pd.NaT` and - NumPy scalars. Unsupported leaves raise `PopulationError` under the static - code `storage-object-leaf` instead of being `repr()`-ed. -- Kept the masked, numeric and `StringDtype` branches byte-identical, verified - by a pre-fix/post-fix byte diff over 20 dtypes and pinned by hex-literal - characterization tests. Only object, Categorical, DatetimeTZ, Period and - Interval move — every one of them a dtype whose old bytes were addresses. -- Documented at `storage_equal` what the parts do and do not seal, with - regressions pinning the masked half, the numpy-normalization, the new - refusals, and a real `ContentStore` frame round trip. -- Adversarial review (three lenses) found two real defects, both fixed: the - refusal missed `UnicodeEncodeError` from a lone-surrogate `str`, and two - deliberate normalizations were undocumented. -- Green: `pytest packages/microcosm-graph` 399 passed; the nine graph-adjacent - build/frame/fit/calibrate files 578 passed; `test_graph_population.py` 119 - passed; `ruff check .` 0; `ruff format --check` on both changed files 0; - `tools/ci_test_groups.py --verify` 0; `tools/spec_engine_coverage.py - --check` 0 (42156/42156 fields, 41/41 inventory); `tools/graph_acceptance_ - burndown.py --verify` 0 (green 41, red 0, missing 0). - -## Next - -- Human review. One item this lane could not complete: the issue asked for a - note at `_population_stamp` in `us_runtime/survey_atomic_geography.py`, a - module that exists only at the tip of the unmerged - `origin/microcosm-us-launch-integration-20260909`. The verified wording — - corrected, because the issue's own phrasing was incomplete — is in the lane - report for whoever owns that branch. -- No pinned digest moves. The three `*_population_sha256` stamps on that - branch will take new values once this merges; their old values were process - addresses, they are recomputed on both sides of every comparison, and - `survey_origin_budget` already excludes them from the persisted identity. - ---- - -# F1 portable worker identity — CI crawl fix +This file is a session-handoff journal. Accurate when written, historical +afterwards — check git/GitHub for current truth. ## State -Complete on 2026-09-05 on `f1-portable-worker-identity`, started on -2026-09-04 at `50c9232b7597bd1f47897a3132ed08894fc11d93`. Process memoization, -real session priming, explicit live opt-outs, and all requested verification -are complete. No GitHub network, push, new branch, or stash. -Final report, exact commands, direct exits, and per-file timing tables: -`/private/tmp/microcosm-pr871-ci-crawl/out.md`. +Reading the authorities and the code at this head. Nothing implemented. ## Done -- Read the guide, prior journals, worker identity/bootstrap/launch, three live - stacked binding sites, and fixture caches. Used direct source tracing and - independent agent reviews; the GitNexus skill's tools were unavailable. -- Completed both unchanged baselines before production/test behavior edits: - identity selection 77 passed, 245 deselected, exit 0, 734.406s wall; - first 200 tests 200 passed, 122 deselected, exit 0, 2033.172s wall. - Source-order collection used `-p no:randomly`, then 122 explicit deselections; - the final selected node was - `test_primary_refuses_missing_universe_receipt_before_callback`. -- Repaired only ignored `.venv/bin/pytest` after its stale interpreter shebang - produced direct exit 127. Used the supplied synced Python 3.14.4 environment, - `uv run --no-sync`, and an external writable UV cache; no dependency sync. -- Committed fail-before memo regressions: 4 failed, 7 passed, 322 deselected; - exit 1; 30.073s wall. Implemented the memo keyed by raw lock argument and - both bound fit-control environment values, with an exported/documented clear - API and shared read-only semantic graph. Execution bindings clone the graph - so artifact mutation cannot poison subsequent identities. -- Primed real session identities for stacked-spine, H5, pool-tool, and the - indirectly reached spec-bundle suite. Removed redundant stacked/H5 caches. - Identity/source/runtime/backend/environment mutation tests explicitly opt out - and clear before/after; deliberate byte edits within a test clear between calls. - Preflight/fiscal-refresh tests already stub their live-binding paths. -- Added 14 regression cases covering memo reuse/reset, both environment keys, - invalid controls and locks, lock separation, artifact-copy isolation, real - session consistency, opted-out namespace-byte changes, and generator ordering. - Memo/session checks passed: 13 passed, 322 deselected; exit 0; 64.224s wall. -- Broader call-path audit found spec generation could precede function priming. - Committed a real ordering regression before the fixture fix: 1 failed, - 25 deselected; exit 1; 129.447s wall. Ordered priming before cached generation; - the same test then passed: 1 passed, 25 deselected; exit 0; 99.303s wall. -- Same post-fix identity selection: 90 passed, 245 deselected; exit 0; - 160.517s wall, a 4.58x speedup despite 13 added selection cases. -- Full stacked-spine file: 335 passed, 2,378 warnings, no failures/skips; - exit 0; 347.905s wall (5m 47.905s), pytest 337.74s. -- Ran actual CI process expansions from `--list GROUP:PROCESS`; `--procs` - returns process names, so the prompt's literal per-file loop was not valid. - Every process ran serially with exact CI file arguments and no cacheprovider. - An external observation-only plugin saved each file's counts and wall span. -- Initial us-qs:build before bundle priming: 1,368 passed, no skips; exit 0; - 626.070s wall. Reran the affected complete process after its final fix. -- Final CI process results (all exit 0): - - | Process | Files | Passed | Skipped | Wall seconds | - |---|---:|---:|---:|---:| - | us-qs:build | 29 | 1,369 | 0 | 654.966 | - | us-qs:frame | 5 | 84 | 8 | 111.252 | - | us-am:build | 59 | 1,899 | 1 | 855.001 | - | us-am:other-shards | 48 | 1,000 | 2 | 139.672 | - -- Final groups total 4,352 passed, 11 existing skips, no failures/errors across - 141 files. Skips cover eight optional Axiom cases, one opt-in 3.7 GB SIPP - audit, and two existing live data-loader tests. No skip was added by this fix. -- Ruff packages/tools passed (exit 0, 0.066s); all six changed Python files - passed format checking (exit 0, 0.021s). Diff whitespace passed (0.036s). -- Spec proof passed: 42,154/42,154 configuration fields, 41/41 inventories; - exit 0, 81.683s wall. No spec pins moved. Protected-file and AST verification - passed (exit 0, 0.080s): the uncached identity body, existing validators, - authenticator, probe, and launch policy are unchanged. The full - `worker_execution` subtree remains excluded from inventory digests. -- Independent reviews found no actionable defects. Engine-free compatibility - was reviewed from source; no engine-free test run is claimed. Python 3.13, - GitHub/Linux runners, wheels, unrelated CI groups, and certification were - not run in this local fix lane. All command receipts are in the report. +- Verified worktree/branch/head; synced `uv sync --all-packages --locked --extra us` (exit 0, pandas 3.0.3). +- Read `docs/us-native-scale-transport.md` (481 lines) and + `experiments/native-scale-transport/out.md` (839 lines) in full. +- Read at this head: `survey_population_replay.py` (208 lines, whole file), + `survey_atomic_geography._population_stamp` / `_copy_population`, + `survey_population_preparation._frame_identity` / `_cell` / + `_frame_cell_encode`, `graph_context.encode_us_frame_context`, + `population._storage_parts`, `executor._observer_snapshot` and the observer + call site. ## Next -No implementation or requested local verification remains. No push. - -# Historical: Astra gate round 1 journal - -> The section below records the prior lane at `50c9232b`; its results and -> next steps are historical, not the current CI-crawl fix state. - -# F1 portable worker identity — Astra gate round 1 - -## State - -Complete on 2026-09-04 on `f1-portable-worker-identity`, starting at -`32ce6f518e8847647f23b3f6f11e4a8dc060ed01`. Both peer findings have committed -fail-before regressions and fixes. All eight requested suites passed (964 -tests); Ruff, formatting, and the spec-pin proof passed. No pins moved. -No GitHub network or push. Final report: -`/private/tmp/microcosm-pr871-astra-round1/out.md`. - -## Done - -- Read `CLAUDE.md`, the prior Sol journal, the worker identity/bootstrap, - pinned QRF launcher, stacked launch/binding, and relevant test sections. -- Confirmed the assigned branch and clean starting tree. -- Read the GitNexus debugging workflow; its tools are unavailable, so trace - execution directly from source and offline tests. -- Preserved the full `worker_execution` digest exclusion and all hard-boundary - files, including the seed-attested `puf_qrf_chain.py` launcher. -- No concrete `-o` path was supplied. Asked asynchronously and wrote the - report outside the repository at the stated default path, - `/private/tmp/microcosm-pr871-astra-round1/out.md`. - -- Reproduced finding 1 before any production edit: the fresh interpreter at - the real stacked/chain launch observes Torch autoload `1` before the worker - module loads (expected `0`); one test failed, direct exit 1. -- Reproduced finding 2 before any production edit: both source-tree and - inherited-prefix valid-header stale caches execute altered code in the - production identity probe; two cases failed, direct exit 1. Control children - prove that disabling bytecode writes alone still executes those caches. -- The recovered pytest script uses an obsolete interpreter shebang. Use - `uv run --no-sync python -m pytest` with a writable `UV_CACHE_DIR`; direct - pytest script attempts exited 4 before collection. No sync was needed. -- Recorded both failing commands and observations in the external report. - -- Fixed finding 1 by passing the semantic binding's forced overrides into - the stacked launch environment before Python starts. Retained the worker - bootstrap guard and corrected its comment; the pinned chain is untouched. -- The fresh-interpreter first-Torch-import regression now passes: 1 passed - in 58.11s, direct exit 0. Its child sees `0` while the parent retains `1`. - -- Fixed finding 2 using a shared launch context that forces an empty fresh - `PYTHONPYCACHEPREFIX` and `PYTHONDONTWRITEBYTECODE=1` in both probe and stacked - worker. The execution binding records the stable `{empty_pycache_dir}` - placeholder, and traces refuse unexpected namespace/stdlib bytecode paths. -- The cache reproduction now passes in both identity-probe and worker `-m` - modes across both cache locations: 4 passed in 22.62s, direct exit 0. - Probe/refusal/exception-cleanup checks also passed (5 passed in 23.95s). -- Extended the real stacked launch regression to check child Python cache - flags, replacement of an inherited prefix, and cleanup without parent - environment mutation. Added semantic tamper coverage for both cache controls. - -- Extended only the test validation fixtures to reuse pristine real worker - identities by lock/fit controls and return independent deep copies. The - stacked mutation helper compares against an independently obtained baseline; - H5 tests mutate artifacts only. Production factories and source/cache - mutation regressions remain uncached. Focused H5 checks: 2 passed, exit 0. -- Independent read-only review found no blocking issue in the startup/cache - fixes, regressions, hard boundaries, or fixture mutation separation. -- Final repository Ruff passed; changed-file format check passed (5 files). - Required release-preflight (42), fiscal-refresh (224), and source-blindness - (497) suites passed with direct exit 0 each. - -- Extended the same test-only reuse to the 11-case tail-control matrix: it - mutates parent constants, while the fresh worker's installed source and - startup environment stay identical. Independent review confirmed the real - resource extraction and digest comparison remain active. -- Interrupted the earlier focused worker run after discovering that matrix's - 22 redundant identities: direct exit 130, 58 passes before interruption; - this is not final evidence. Restarted the complete requested worker selector. -- Launch/binding integration passed (2 tests, exit 0). Required H5 (98), - inventory (15), coverage-tool (7), and imputation (4) suites passed, exit 0 - each. Ruff and changed-file formatting passed after the last test edit. -- Verified every forbidden path, docs/tools, and uv.lock is unchanged from - `32ce6f51`. - -- Standalone spec proof passed, direct exit 0: 42,154/42,154 configuration - fields and 41/41 inventory checks. No pins moved; the unchanged spec SHA is - `9db29b4d33424fbb21a83c63927c7de55ba9a333d631f6323935f67a496eee46`. -- The focused worker rerun completed with direct exit 0: 77 passed, 244 - deselected, 141 warnings in 866.48s. The real transfer-bank integration also - passed. All eight required suites total 964 passes, with no failures or skips. -- Finished the external report with both fail-before commands/observations, - fix SHAs (`b131afb7`, `4a575d7d`), cache-isolation rationale, exact final - verification commands/counts, no-pin proof, and deliberate scope exclusions. - -## Next - -- Local review of the committed fixes and external report. No implementation - or verification work remains for these two findings; no push was performed. - -# Historical: Sol gate round 1 journal - -> The section below is the prior round's handoff at `32ce6f51`. Its pending -> state and verification claims are preserved as historical evidence. - -# F1 portable worker identity — Sol gate round 1 - -## State - -In progress on 2026-09-04 on `f1-portable-worker-identity`, starting from -`b26708a1`. All four Sol findings have fail-before reproductions. The schema-9 -envelope, Torch backend-autoload, loaded-runtime/stdlib, and real-resource -fixes are implemented and focused green. Pin proof and full verification remain. - -## Done - -- Read `CLAUDE.md` and the F1 PR body/progress brief. -- Confirmed the requested branch and clean starting tree at `b26708a1`. -- Read the GitNexus debugging workflow. This workspace exposes no GitNexus - query/resource tools, so call-path analysis is being performed directly from - source and tests. -- Recorded the hard boundaries: keep the complete `worker_execution` subtree - out of spec-engine digests; do not edit graph interface/acceptance lock files; - refuse before side effects; keep tests offline; commit each coherent step. -- Recorded the required focused and final verification suites and the - requirement to report only commands actually run in `out.md`. -- Synced the locked all-package US/UK environment after directing uv's cache to - a sandbox-writable path; the unmodified command's two environment-specific - refusals and the successful command are recorded in `out.md`. -- Added fail-before coverage proving that schema 9 accepts missing/wrong - pipelines and routes other missing envelope sections around the common - validator; the focused result was 1 passed and 1 failed, exit 1. -- Added fail-before identity coverage proving that loaded-runtime and stdlib - mutations are unbound, a synthetic unapproved `torch.backends` provider is - accepted, and the real SOI interest-components resource is absent. The four - focused nodes failed as intended, exit 1. -- Added a fail-before launcher regression proving that an inherited/caller - `TORCH_DEVICE_BACKEND_AUTOLOAD=1` reaches the child unchanged. The focused - node failed as intended, exit 1. -- Replaced the prior unrelated `out.md` with the current round's reproduction - report; fix/pin/final-verification sections remain explicitly pending. -- Fixed the schema-9 bypass: schema 9 now has an explicit complete stacked - field set, traverses the same envelope classifier as schema 10, and does so - before any compatibility attestation is read or authenticated. -- The valid schema-9 metadata-restoration case and the seven-case malformed - envelope regression pass together (2 passed, exit 0). -- Forced `TORCH_DEVICE_BACKEND_AUTOLOAD=0` in both the authenticated semantic - environment and the worker module bootstrap, before its QRF/Torch import. -- Enumerated and bound selected `torch.backends` entry-point metadata, refused - provider distributions outside the installed-code closure before clean - worker import, and refused duplicate canonical distribution identities so a - colliding provider cannot evade RECORD hashing. -- Focused provider-refusal, duplicate-provider, launch-override, semantic - tamper, and legacy relocated-worker acceptance checks pass (exit 0 each). -- Replaced the two-file resource list with a fresh worker-import audit trace - using the same inherited startup search path as the real worker. The semantic - transitive-import digest now includes every opened - Microcosm namespace file, with bytecode canonicalized to source, portable - locators only, and ambiguous duplicate locators refused. -- The trace freezes and revalidates namespace roots, captures transient import - origins and successful pre-open file paths, refuses disappeared namespace - files, and rejects an empty or displaced worker trace. Its opened stdlib - paths supplement final `sys.modules` so transient stdlib imports stay bound. -- Resolved the loaded Python image through platform mapping with static and - sysconfig fallbacks, and bound its kind and byte digest without serializing - its path. The interpreter identity also binds the clean import's file-backed - stdlib source and extension bytes while excluding site packages. -- The mocked runtime-byte and stdlib-source mutation tests pass together; the - real SOI interest-components resource is observed and changes the resource - closure digest; and a real full identity constructs and validates with the - mapped `libpython3.14.dylib` (all exit 0). -- Kept worker identity schema v1 because it is the still-unreleased exact - schema introduced by this branch. Bumping it would churn authored spec - templates despite the requirement that spec-engine pins remain fixed; - structural validation now requires the added v1 fields. -- Restored `puf_qrf_chain.py` byte-for-byte after the first pin run proved that - editing its operational launcher also moves the QRF seed-kernel source - attestation. The worker bootstrap now owns the override, keeping the existing - seed protocol and compiled seed-map pins intact without re-pinning. -- Added bound-environment-keyed, deep-copied caches only to the stacked-spine - and H5 canonical test fixtures so parameterized/tiny-pool cases do not rebuild - one identical production identity apiece; production identity generation - remains uncached. The interrupted pre-cache final run had 9 passes before - exit 130 and is not treated as final evidence. -- Added the same narrowly scoped reuse to the inventory and coverage-tool test - modules: each module constructs one real binding and deep-copies it for - repeated report builds whose digests deliberately strip `worker_execution`. - This does not cache the production resolver or replace its first real check. - -## Next - -- Prove spec-engine pins remain fixed, run the complete requested verification - block, and finish `out.md` with exact commands, counts, and exit codes. - -# Historical: ACS predictor release join - -> **Historical note (2026-08-28).** This journal describes the -> `acs-predictor-release-join` lane as of 2026-08-27. The branch has since -> been merged into the `stacked-release-fix-train` integration branch -> together with the #794 gate-alignment and #798/#799 pregnancy/prior-year -> fixes, with spec envelope digests and coverage evidence regenerated over -> the union tree. Treat the "State"/"Next" sections below as history; -> check git/GitHub for current truth. - -## State - -Complete on 2026-08-27. The owner-approved release-time join from stacked-pool -ACS source lineage to the SHA-pinned 2024 one-year ACS person/household zips is -implemented, receipted, real-pool exercised, and fully verified. It populates -the six archived donor models' CPS-named predictors through reviewed -native-ACS crosswalks with strict hash, lineage, collision, totality, universe, -and clone-fan-out contracts. Model selection logic and every gate threshold -remain unchanged. The completed evidence and handoff are in `out.md`. No -network access, pool build, release build, publication, push, retraining, -threshold change, or launcher-contract edit occurred. - -## Done - -- Read `CLAUDE.md` and the prior weeksgate report's six owner-ruling items with - their release-call and model-consumer evidence. -- Confirmed the requested branch `acs-predictor-release-join` is clean at - `606cbd69`, based on `stacked-release-gate-alignment`. -- Read the GitNexus exploration and impact-analysis workflows. This workspace - exposes neither GitNexus repository resources nor query tools, so the same - call/dependency analysis will be performed directly from source and tests. -- Recorded the required source zips and SHA-256 pins, strict exact/total join - contract, explicit crosswalk and receipt requirements, and verification - boundary. -- Proved that `person_source_id` is not a reversible ACS key: ACS people are - sorted by `(SERIALNO, SPORDER)`, receive a zero-based raw spine ID, and then - receive a collision-dependent assembly offset. The pool retains the raw - spine ID, `source_row_id`, `source_person_id`, household `SERIALNO`, and clone - metadata, so the release join will use the retained semantic - `(SERIALNO, integral SPORDER)` key and treat `person_source_id` only as the - one-to-many clone fan-out identity. -- Audited the supplied candidate pool read-only: 856,626 distinct ACS source - people expand to 1,736,840 rows (856,626 clone 0, 856,626 clone 1, and - 23,588 clone 2), with no duplicate `(person_source_id, clone_index)` pair. - Every ACS row agrees with its raw spine/source lineage, and all selected - people match the pinned raw person archive exactly. -- Verified both local archives against the charter pins. The person archive - has 3,422,888 unique `(SERIALNO, SPORDER)` rows and no household orphans; - the household archive has 1,631,969 unique serials, including 1,531,614 - occupied records. Both contain every requested native predictor. -- Established the disability universes from the pinned archive and the - archived repository mapping: DEAR/DEYE are complete at every age; - DREM/DPHY/DDRS are asked from age 5; DOUT from age 15; native code 1 is the - consumer's difficulty bin and code 2 (plus an age-valid universe blank) is - its non-difficulty bin. -- Established the consumed race/Hispanic bins: both SCF models distinguish - White, Black, Asian, Hispanic, and Other; ORG distinguishes Hispanic, - non-Hispanic White, non-Hispanic Black, and Other. `RAC1P`/`HISP` can map - exactly to those bins without inventing detailed CPS combinations. -- Recovered the complete 2024 Census detailed-occupation-to-`POCCU2` consumed - grouping from the native ASEC relationship and confirmed that ACS `OCCP` - uses the same detailed codes. `PEIOOCC` is therefore a direct carry, while - `POCCU2` will use an explicit reviewed 53-bin table; blank out-of-universe - occupation maps to code 0, military to 52, and code 9920 to 53. -- Confirmed ACS `TEN` maps to the SPM vehicle model's three consumed tenure - bins (mortgaged owner, outright owner, non-owner); no-cash-rent and verified - group-quarters blanks belong to the non-owner bin. Confirmed the SSI model's - `SSI_VAL` use is only the `> 0` reporter anchor and that native ACS `SSIP` is - already carried as harmonized `ssi_reported`, observed exactly from age 15. -- Added the dedicated `acs_release_predictors` release boundary. It verifies - the two canonical archive pins before opening either zip, streams only - selected households, validates exact archive members and headers, rejects - raw/person/clone collisions, binds retained pool lineage to - `(SERIALNO, SPORDER)`, requires total one-to-one source-person matching, and - fans mapped values to clones only through `person_source_id`. -- Added explicit disability, race/Hispanic, 530-code occupation, and tenure - tables. A canonical crosswalk payload is pinned at SHA-256 - `1d4906242e9c73e31b3283659e5cad8242b8cbc42914ab6fa59547a10c8770e9` - and rides the JSON-ready join receipt with per-model/per-predictor - ASEC-native, ACS-joined, and still-null counts. -- Preserved CPS disability universe semantics (`-1` below the question age) - and the ACS occupation universe. Blank `PEIOOCC` uses the CPS NIU sentinel - `-1`; blank `POCCU2` remains 0 through age 15 and maps to the consumed - no-occupation code 53 only from age 16. This explicitly preserves the - one-year ACS/CPS source-universe gap instead of assigning every ACS - 15-year-old a never-worked status without source evidence. The explicit - occupation table covers every one of the 530 codes in the pinned ACS person - archive and every consumed POCCU2 bin. -- Changed the SSI-disability reporter read, without source routing, to - row-wise coalesce measured ASEC `SSI_VAL` with harmonized native ACS - `ssi_reported`. Adult blanks and conflicting dual reporters fail; genuine - below-age-15 ACS blanks remain null in the frame and become false only for - the consumer's `> 0` predicate. -- Hardened the join after independent crosswalk review: raw ACS `SSIP` and - `ADJINC` now travel through the pinned join and must agree exactly with every - native clone-0 `ssi_reported` value under the established adjusted-dollar - formula. Raw `ESR`/`OCCP` must obey their exact age-16 universes, and all - ASEC predictor receipt cells must be numeric and finite with complete, - nonnegative `SSI_VAL`. -- Updated the SSI signal diagnostic to use the same row-wise reporter coalesce - as the model consumer, while retaining the archived native-role anchor - scope. A lost positive ACS-native reporter can therefore no longer evade the - release gate merely because `SSI_VAL` is null on physical ACS rows. -- Added focused tests for crosswalk identity/all consumed bins, exact join and - clone invariance, ASEC byte preservation, receipt contents, missing joins, - raw and source-identity collisions, hash refusal, no-ACS identity, and SSI - coalescing/universe refusal. Coverage now also fixes the age-15 occupation - gap, malformed ESR refusal, malformed ASEC SSI refusal, raw SSI attestation, - and gate-side ACS reporter preservation. The complete join, SSI, and - source-blindness test files pass together, and focused Ruff is green. - -- The release CLI now accepts the person/household zip and lowercase 64-hex - SHA-256 options as an all-or-none set. It invokes the authenticated join - after the last unrelated native-input gate and before SCF wealth, therefore - before all six archived donor-model stages, then carries the complete join - receipt into both `build_manifest.json` and `release_manifest.json`. -- Added parser refusal tests, a source-order contract over all six model calls, - an end-to-end mocked main corridor that verifies the exact four join - arguments and runtime ordering, an AST contract that binds the saved receipt - to the sole manifest call, and JSON round-trip assertions for both manifests. - Focused Ruff, five parser/order/manifest cases, and all six parametrized main - corridor cases pass. -- Exercised the hardened join read-only on the complete supplied candidate. - The 3,239,263,147-byte H5 matches its frozen manifest SHA-256 - `871b7e6467675a1e9475b54fd1baf64c53c0f75a3258b8357303a8df0d53642d`. - The current official loader refuses that older candidate before H5 loading - because its archived primary-QRF worker binding predates this branch's - execution identity; this is an existing candidate/code-version mismatch. - Loading those independently manifest-hash-verified bytes with their frozen - assembly receipt allowed the join boundary itself to be tested without - writing an artifact. -- The real join passed every source, raw-key, universe, SSI-attestation, - totality, collision, and clone-fan-out check: 856,626 unique ACS source - people matched 856,626 raw people in 382,903 households and populated - 1,736,840 support rows (856,626 each at clone indices 0 and 1, plus 23,588 - at clone index 2). Every CPS-named predictor consumed by the six models has - 234,133 valid ASEC-native and 1,736,840 ACS-joined cells with zero nulls. - The logical SSI reporter anchor has 234,133 ASEC cells, 1,475,235 observed - ACS cells, and exactly 261,605 preserved child-universe null support rows. -- The first real-data attempt exposed fixed-format HDF's expected object dtype - for mixed-source columns. Tightened the ASEC validator to inspect each cell, - accepting object-wrapped real numbers while still refusing strings, - nonfinite values, nulls, and negative SSI. A focused H5-shape regression and - all 15 join tests pass before the successful full-pool rerun. -- Added `changelog.d/acs-release-predictor-join.fixed.md`, describing the - pinned release join, reviewed mappings, fail-closed lineage, dual-manifest - receipts, and unchanged model/gate behavior. -- Repository Ruff passes, and the CI inventory verifier reports 310 tracked - tests with `verification=ok`. Four complete pytest shards pass in separate - processes: frame 295 passed/36 skipped, fit 93 passed, calibrate 203 passed, - and data 318 passed/2 skipped. -- The first complete build-shard process reached 100% with 6,575 passed and 45 - skipped, plus five failures and six fixture errors. All eleven were the same - expected source-attestation drift: `ssi_disability_criteria.py` belongs to - both the direct and QRF seed-kernel inventories, so this task's runtime edit - moved the seed protocol, compiled US seed map, every country spec identity, - the minimal loader golden, and the generated coverage evidence. No ACS join, - release CLI, manifest, archived-model behavior, or gate test failed. -- Applied the repository's established five-file source-identity repin only: - seed protocol `59a098f9...31d8b`, US seed map `ce3850d8...e42ab`, US spec - `16b7d5e6...dca38`, UK spec `2f921e4c...33a62`, BE spec - `c87a0012...34ba`, and minimal-loader golden `b4946105...f2af`; regenerated - `docs/evidence/spec-engine/us-f0-coverage.json`. All 25 affected cases and - focused Ruff pass. The US bundle generator `--check` passes at the new spec - identity, and coverage `--check` passes at 42,122/42,122 fields and 41/41 - inventory checks. -- Re-ran the complete build shard after the reviewed repin: 6,586 passed and - 45 skipped, with exit code 0. Re-ran final repository Ruff, the 310-file CI - inventory verifier, both retained spec `--check` commands, and - `git diff --check`; all pass. Wrote the required final report to `out.md`. - -## Next - -- No work remains in this lane. The dispatcher owns rebasing and the launcher - contract update. A future authorized build must produce a pool whose current - source-attested worker identity passes the official release loader; the - supplied older candidate is useful join evidence but cannot be promoted. - -# Weeksgate: stacked release gates and integer-week provenance - -## State - -Complete on 2026-08-27. Real-pool provenance has refuted the proposed -post-transfer amount-mapping mechanism: every fractional week is an ACS-origin -non-native clone prediction outside the calibration's clone-0 recipient scope. -The source codec, weeks-gate architecture, source-scope, clone-layout, and -stable-identity repairs are implemented and focused-tested. The complete -release-call roster is classified; six archived-model input assumptions require -owner rulings and are deliberately reported instead of guessed. Repository-wide -Ruff, the CI inventory verifier, and all five full pytest shards pass in their -required independent processes. The completed provenance, audit, verification, -judgment calls, and host-owned checkpoint-rerun consequence are in `out.md`. -No network access, artifact build, publication, push, pool build, or release -build is in scope. - -## Done - -- Read `CLAUDE.md` and the GitNexus debugging workflow. -- Confirmed branch `stacked-release-gate-alignment` is clean at `4f453746`. -- Confirmed the local GitNexus CLI is installed but the repository is not yet - indexed. Its offline analyzer parsed the repository but could not register - the index because the sandbox forbids writes to `~/.gitnexus`; the generated - local index was moved out of the worktree to `/private/tmp`. -- Recorded the four requested workstreams: fractional-week provenance and PUF - misclassification; integer-support calibration repair; stacked/legacy weeks - gate alignment; and the full release-side gate archaeology sweep. -- Recorded the required verification boundary: repository Ruff plus one pytest - process per shard, with no pool/release builds. -- Read the fixed-format HDF5 blocks directly and classified all 369 noninteger - `weeks_unemployed` rows: 360 are ACS clone 1 (355 UC=0, 5 UC>0) and 9 are ACS - clone 2 (all UC=0); all are positive, all 369 values are distinct, and the - exact range is 1.0003521955067698--37.796501228614694. -- Confirmed zero nonintegers on ASEC rows and ACS clone 0. The receipted - calibration covers exactly the 856,626 ACS clone-0 rows, maps 8,419 carrier - amounts onto observed ASEC support with zero donor-support violations, and - records QED 0.5882352941176471 to 0.0. -- Reproduced the 5,218-row false "PUF" classification: the legacy role helper - calls every clone index above zero `puf_tax_detail`, regardless of raw source - channel. The rows are all ACS-origin clones: 4,733 integer clone-1 rows, 355 - fractional clone-1 rows, 121 integer clone-2 rows, and 9 fractional clone-2 - rows with nonzero weeks while UC is nonpositive. -- Traced the actual fractional mechanism to the ACS transfer target codec: - PolicyEngine-US declares `weeks_unemployed` as physical `float`, so the - generic QRF path treats it as continuous even though its reviewed source - contract is integer-supported. The later calibration repairs clone 0 only. -- Bound every final fractional value bit-for-bit to the late-transfer target - bank. Its raw QRF draw contains 711 nonintegers: 342 ACS clone 0, 360 clone 1, - and 9 clone 2. Post-transfer calibration changes 13,417 clone-0 rows and - eliminates all 342 clone-0 fractions; the 369 out-of-scope clone fractions - pass through unchanged. -- Added `weeks_unemployed` to the ACS transfer's authority-bound discrete - numeric target registry. The existing codec now snaps every prediction to - actual observed ASEC donor support, and the execution-contract identity - receipts the policy alongside the two mortgage-year targets. -- Added a focused ACS-transfer regression proving weeks predictions retain - observed integer support and the execution contract declares the target. -- Regenerated the checked-in US imputation authority so - `transfer_execution.discrete_numeric_targets` includes `weeks_unemployed`. - The generator's compile and byte-staleness checks pass at bundle spec SHA - `821d5838da3ac368170e61e017f1a72648f93e8a011aa40e33b8c2b4b14511f3` - at that source-fix step; the later runtime/seed identity repin below - supersedes this intermediate digest. The spec-bundle and imputation- - semantics suites pass. -- Passed the complete ACS-transfer test file (65 tests), the complete - post-transfer calibration receipt-contract file (47 tests), focused Ruff, - and `git diff --check` using the prebuilt `.venv` directly. A task-local - `UV_CACHE_DIR` later allowed the required `uv run --no-sync` commands to run - against that same prebuilt environment without accessing `~/.cache/uv`. -- Modernized the weeks summary/gate to derive its roster from actual assembled - source channels, while retaining the legacy ASEC/PUF role path. The ASEC - source-validity scope, direct native-clone reconciliation scope, and reviewed - UC-constraint scope are now distinct and explicitly receipted in details. -- Kept all four plausibility-band tuples and every numeric threshold unchanged; - non-ASEC assembled channels use the unchanged legacy recipient band. -- Added stacked ASEC+ACS and legacy ASEC+PUF fixtures covering roster detection, - raw-source scoping, native reconciliation, and UC constraint ownership. The - complete weeks file passes (26 passed, 1 skipped) with focused Ruff. -- Replayed the updated gate over the supplied pool's exact live arrays and - weights. It now reports 234,133 valid ASEC source rows, 108,073 exact native - reconciliation rows, 982,686 UC-constrained rows, zero source/UC failures, - both unchanged channel bands passing, and only the genuine 369 nonintegers. -- Added a centralized provenance-owner API that distinguishes validated - physical source channels from legacy clone-operator roles. The weeks gate - now consumes that API rather than reading provenance columns directly, and - both repository source-blindness tripwires pass alongside the stacked and - legacy provenance/weekly-signal suites. -- Made WIC's deterministic draw key prefer the assembly-unique - `person_source_id` on multispine frames, before the source-local raw identity - triple. Distinct ASEC/ACS records can no longer collide when their vintages - align, while all support clones of one assembled person retain the same draw - and the legacy key order remains unchanged. -- Completed the release-tool call-roster audit. It found unambiguous physical - ASEC scoping repairs in SSI take-up, workers' compensation, alimony, - retirement contributions/distributions, and Medicare; clone-2 layout fixes - in Head Start, voluntary filing, and prior-year income; and a stable-key fix - in WIC. Archived ASEC-only predictor assumptions in SSI disability, SCF - wealth, SCF auto loans, and SIPP vehicles require explicit model-owner - rulings and will be reported rather than guessed. -- Reworked the assembled Head Start and voluntary-filing receiver layouts to - key by assembly-unique source ID plus explicit clone index, accept clone 2 - and later, reject duplicate source/clone rows, choose clone 0 (or the lowest - surviving clone) deterministically, and fan one source-level decision to all - clones. Their legacy role-only duplicate contract remains unchanged. -- Replaced occurrence-pair clone diagnostics in voluntary filing and - prior-year income with all-clone grouping on assembled source IDs, so a - clone-2-only divergence is now detected. Focused tests and Ruff passed for - all three module/test pairs. -- Scoped Medicare and retirement release diagnostics to physical ASEC source - rows, separating raw-source validity from native direct-carry reconciliation - where transferred clones intentionally differ. Kept the producer kernels - origin-blind: the authenticated-pool release path skips those producers, and - indirect physical-source routing would violate the repository's population- - operator boundary. Their 60 focused module/source-blindness tests and Ruff - pass after that review correction. -- Scoped alimony and workers' compensation raw validity to every physical ASEC - clone and exact source-carry checks to physical ASEC native rows, leaving all - clone-operator plausibility bands unchanged. Stacked ASEC+ACS and legacy - ASEC+PUF fixtures pass (28 alimony and 21 workers' compensation tests). -- Changed SSI reporter-lineage capture to validate `SSI_VAL` only on physical - ASEC rows and accept null ACS raw-source cells. Assignment remains source- - blind: it consumes the source-ID set captured before L0, or obtains that set - through the reporter helper when no explicit set is supplied. The complete - SSI take-up file passes (71 tests). -- Fixed SSI-disability's non-fatal clone-divergence diagnostic to group every - assembled clone by source person, so clone-2-only divergence is reported. - The existing decision not to make divergence gate-fatal remains unchanged - for an owner ruling; the complete focused file and source-blindness checks - pass. -- Narrowed physical-channel resolution to a gate/reporter-only provenance API, - removed its general runtime/PUF-support re-exports, and added a static exact- - caller contract (including internal mask-helper callers). A future derive, - impute, or wrapper use now fails the source-blindness suite instead of - passing through indirection. -- Replayed the repaired release gates read-only against the supplied pool. - Alimony, Medicare, retirement contributions/distributions, workers' - compensation, and SSI reporter capture pass; the weeks gate now fails only - on the genuine 369 fractional values. Prior-year income remains outside its - unchanged availability band and WIC finds pregnant nonfemale rows, both - genuine data/spec outcomes rather than stacked-layout archaeology. -- Completed an adversarial review of the repaired code and focused tests with - no additional implementation defect found. It confirmed six owner-ruling - items: SSI disability criteria, SCF wealth, SCF auto loans, SIPP vehicles, - SIPP tips, and ORG wages/FLSA all consume ASEC-only archived predictors on a - frame whose 1,736,840 physical ACS rows carry null source cells. ORG is - guaranteed to fail its unchanged race/occupation bands; SIPP tips' unchanged - tipped-occupation band passes while the ACS channel is dead. -- Passed repository-wide Ruff and the CI test-group inventory verifier. The - calibrate, data, fit, and frame shards pass in four independent pytest - processes. -- Corrected the reviewed WIC seed protocol to match the implemented assembled- - multispine key precedence: assembly-unique `person_source_id` first, then the - unchanged legacy raw/support/person fallbacks. An exact seed-grammar test now - binds that order. -- Re-pinned the fail-closed spec-engine proof after adding one authored - transfer-execution field and one resolved seed-protocol field: 42,122 total - fields (32,352 authored and 9,770 resolved), complete exact-pointer claims, - and all 41 inventory checks. Regenerated the committed coverage report and - validated the final US spec SHA - `5f44d96d45e9aabcea2d565ef063d68bfc0652df1b38b08aa31ce6896d15f371`. -- Verified in a detached `origin/main` worktree, using the same prebuilt venv, - that the old BE, UK, and minimal-spec golden vectors still pass there. Their - current repins therefore reflect this branch's attested runtime and seed- - protocol changes rather than environment drift. All 102 tests in the eight - directly affected spec-engine files pass; generated-bundle and coverage- - report byte checks, focused Ruff, and `git diff --check` also pass. -- Ran the full build shard after that coherent spec repin. It reached 100% with - exactly one failure and no errors: the multispine constants-adapter fixture - still expected the former live US spec SHA. Updated only that live-binding - expectation to the regenerated final SHA; the separate arbitrary checkpoint - identity fixture remains deliberately unchanged. -- Passed the complete multispine-pool-tool file after that correction and - committed the coherent fixture repin as `12a918ed`. -- Reran the entire build shard from zero in one process: 6,608 tests collected, - 100% reached, and pytest exited 0 with expected skips only. All five full - package shards, repository-wide Ruff, generated-artifact checks, the spec - coverage proof, the CI test inventory, and `git diff --check` are green. -- Wrote the final provenance tables, mechanism verdict, per-file rationale, - exhaustive release-gate audit, owner-ruling list, verification evidence, and - judgment calls to `out.md`. - -## Next - -- Host session: rerun `late_transfer -> simulated -> terminal-gates` from the - candidate checkpoints because the discrete weeks codec changes pool content. -- Review the six archived-model owner rulings in `out.md`; do not reinterpret - their missing ACS predictors through a gate-only threshold/scope change. - -# Historical: gate-failed base-pool release lane - -## State - -Complete on 2026-08-26. Containment, opt-in carriage, and preflight surfacing -are implemented and fully verified. The implementation rejects both redundant -green-pool waivers and any release-manifest receipt that does not exactly match -the pool authenticated by preflight; the preflight's historically required -base/selection inputs and exit semantics are unchanged. No pool or release was -built, no artifact was published, and nothing was pushed. - -## Done - -- Confirmed the assigned branch and worktree. -- Recorded the v2 charter: close the legacy bare-H5 multispine bypass, add an - explicit release-build opt-in, carry the authenticated red verdict, and - surface it in publication preflight without making it an automatic - publication failure. -- Confirmed that no network, artifact builds, publishing, or pushes are in - scope. -- Traced the strict manifest loader, current stacked-only terminal-failure - exception, H5 identity stamp, pool sidecar naming, legacy release arm, - release manifests, and both preflight output modes. -- Reviewed the salvage branch's final source and test diff line by line. Its - shared classifier/path-binding/receipt approach closes the bypass without - changing either loader's contract and was retained with the subsequent - coherence corrections recorded below. -- Completed the `simulation_ready` / `gate_failed` / loader consumer audit. - Exact-k remains deliberately strict and head-to-head scoring remains the - existing authenticated evidence exception. -- Identified report-only downstream caveats: stacked producer metadata still - names only the k-ladder readiness consumer; red pool publication returns - status 1 and stops shell chains; ACS-local derivatives keep a donor revision - but do not project the nested red verdict; generic release consumers tolerate - and ignore the additive receipt. -- Added a release/preflight-specific authenticated pool loader over the shared - `require_simulation_ready` seam. The strict simulation-ready and existing - scoring-only loader contracts remain unchanged. -- Closed the bare-H5 path by detecting either the canonical sibling manifest - or the H5's stamped pool identity, requiring the sidecar, authenticating the - publication triple, and binding it to the exact requested H5 path. -- Added `--allow-gate-failed-base-pool` only to the legacy `--base-h5` arm. - It admits only a current authenticated stacked `gate_failed` pool, rejects a - green or non-pool use, and never affects the exact-k arm. -- Added the self-contained `base_pool` receipt to both manifests, including - status/readiness, immutable pool identities, flag use, gates JSON SHA-256, - failure count/list, and the complete terminal verdict. -- Kept the static preflight inputs mandatory, authenticated its base identically, - displayed red evidence prominently without changing its exit calculation, - and required optional release-manifest carriage to match the authenticated - receipt exactly. -- Hardened carried verdict normalization so nested pass/failure pairs and the - aggregate verdict must be coherent and a red battery cannot report zero - failures. -- Passed focused Ruff and the complete builder/H5/preflight test files after - the containment changes. -- Passed the broader exact-k, launcher, release-contract, and publish-guard - regression suites. -- Passed repository-wide Ruff and the CI test-group inventory verifier. -- Passed every pytest shard in its own process: build 6,545 passed / 45 - skipped; calibrate 203 passed; data 318 passed / 2 skipped; fit 93 passed; - frame 295 passed / 36 skipped. Aggregate: 7,454 passed, 83 skipped. -- Confirmed `git diff --check` is clean and that no battery bounds, - tolerances, plans, or terminal gate logic changed. -- Wrote the complete handoff, consumer audit, manifest schema, verification - receipts, and commit inventory to `out.md`. - -## Next - -- Human review and merge of `release-from-gate-failed-pool`. -- Any later artifact operation remains separate: an operator must deliberately - choose the red-pool flag, then run publication preflight and make the human - publication decision. This lane performed none of those operations. - -## Historical prior lane - -The stacked-pool-to-release CD-vintage provenance lane previously maintained -this journal and completed before this work began. It authenticated and -applied household geography after source assembly, carried that authority -through checkpoint and publication identities, published verified CD-vintage -H5 attributes, and reached the unchanged release guard through the shared -fixed/table-aware reader. Its final verification was 7,241 passed, 77 skipped, -with repository-wide Ruff and anti-rot checks green. Full details remain at -commit `2263df36` (the parent of this lane's first journal commit). - -The still-earlier PolicyEngine-US 1.819.0 lock-bump lane merged into -`origin/main` at `7b90bb18` on 2026-08-24; its final state remains at commit -`05d254aa` and its detailed receipts remain in the historical section of -`_LANE-NOTES.md`. - -## US launch integration staging — 2026-09-09 - -### State -Source-only staging in progress. Execution and source/data admission remain root-owned. - -### Done -Verified requested clean branch, base HEAD, main ancestry and preservation pins. - -### Next -Apply thirteen explicit source layers, commit each, then separately review isolated ordinary execution. Existing journal history above is retained. - -Layer 1: SAFE-ADDITIVE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 2: GRAPH-RESTORE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 3: ACCEPTED-SHARED-RESTORE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 4: PUF-SUPPORT-MERGE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 5: SOLVE-MERGE-PROPOSAL.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 6: J-GRAPH-COMPATIBILITY.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 7: F-CATALOGUE-OPTIMIZATION.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 8: GRAPH-ATTACHMENT-METADATA.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 9: F-JOINT-GEOGRAPHY-GATE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 10: SOURCE-CLOSURE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 11: PLACEMENT-ADDITIONS.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 12: ORDINARY-CLOSURE.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -Layer 13: INTEGRATION-REGRESSIONS.patch applied; all declared postimages and preservation hashes verified. No tests executed. - -### State -All thirteen source layers staged and committed; behavioral qualification pending. - -### Done -Per-layer pins and actual commit messages checked; store/current-main preservation retained. - -### Next -Root reviews exact ordinary guard/source/resource admissions before execution. Full65 findings and survey/SCF lanes remain separately owned. No remote action is authorized. - -### Integration source-resource closure — 2026-09-09 - -Root preflight found seven JSON source definitions declared by the ordinary guard but omitted by the staged patch delivery. Added the exact previously reviewed resource bytes from the accepted full65 source projection; no new resource admission or genuine payload. Preserved source staging and earlier evidence. - -## Layer14 full65 replay correction — 2026-09-09 - -State: exact accepted two-file correction staged; successor57 integration execution pending. -Done: verified clean84243 preimages, exact r2 postimages, fixed store/current-main sources and frozen36 evidence. -Next: root admits the separate exact-allowlist57-case guard and final source identities before execution; no new resources. - -## Source review publication — 2026-09-09 - -The user explicitly authorized pushing the current source work and creating PRs for Anthony to review. This supersedes earlier source-only local restrictions for source publication; it does not authorize a data release, merge or deployment. - -Integration controls passed 36 cases; the exact replay correction subsequently passed all 57 cases at dfa7f872cd3eba3c42adf5758cde8b3ca38f3d17. Source/control, model-declaration and resource hashes matched externally after both runs. A final formatting/import cleanup and explicit test-observer loop binding are included for CI; no release result is claimed. See docs/us-launch-review.md for current scope, evidence and related PRs. - - -# US completion host integration — 2026-09-13 - -State: source implementation and bounded tests authored for root review; all runtime checks and tests UNRUN. -The task-specific no-commit instruction overrides the standing commit order. -Base: 6aa745c8762c3e4a140b8168470de5827e12c5cb. - -Done: inspected status, branch, HEAD, AGENTS.md, CLAUDE.md, frozen R2 plan and -independent source-plan review. Concurrent completion/tax seam files are excluded -from this work's ownership. - -Done: added private completion custody, retained issued property parent, registry -copies, actual union artifact/state checks, role/child reconstruction and separate -tax gate wiring. Authored 10 test functions / 20 source-expanded cases and docs. -AST, direct Ruff lint/format, and tracked whitespace checks pass. - -Next: root review and reconciliation, inventories/test-spine updates, bounded -resource/guard proposal, and actual fixture/cold/required/adversarial validation. -45/49 counts remain source-derived; combined 51 and fixture assumptions are UNRUN. -Final report: FINAL_REPORT.md. No acceptance or publication claim. - -## Bounded completion-host source corrections — 2026-09-13 - -State: bounded source corrections complete; all runtime checks remain UNRUN. -This correction pass used the 20-minute source-only window beginning 22:09 UTC. -The later explicit no-commit/no-staging instruction governs this correction pass. -Ownership is limited to graph_survey_completion_host.py, -test_us_graph_atomic_completion_host.py, docs/us-survey-completion-host.md, -and this exact PROGRESS.md / FINAL_REPORT.md update. - -Done: read the full independent source review and full approved R2 plan from -the supplied recovered absolute paths. The R2 SHA-256 matches -b402d6be9d374774e27da7bf42fb83a216305d4cddd2ad8347c8149536269a29. -Confirmed intentional dirty worktree and HEAD -6aa745c8762c3e4a140b8168470de5827e12c5cb. Earlier journal content is preserved. - -Done: exact receiving1/roles0-or-2/child6/tax3 ordered rosters are bound before -compilation; child document count and knownness derive from retained nodes and -reconstructed/materialized-verified evidence. Authored separate 49/45 cold+required -acceptance, equal-values/replaced-Population refusal, real observed base-hit/union -and final-I/O identity controls, actual tax-rebased parent refusal, exact child -adversary reasons, full-spine writer ordering, both FILTER conservation ledgers, -dimensions and owner/version transitions. Source proves table copying; immediate -base preservation assertions remain alongside permanent revocation. Fixture-only -composition wrappers now restore before issuance to avoid cross-fixture chaining. - -Done: AST parsing, direct Ruff lint/format and git diff --check pass. AST reports -16 test functions / 32 source-expanded cases. All tests, collection and project -imports remain UNRUN. Only the five explicitly owned files were edited; no -commits, staging, agents, dependency/network/browser access or payload execution. - -Next: root reviews exact source/hashes and budgets separately selected execution. -T4 financial-only-parent coverage, remaining T5 initial/final/reconciliation -adversaries, T6 actual fixture validation and T7 GQ/zero-weight scientific fixtures -remain open with concrete reasons in FINAL_REPORT.md. The independently confirmed -old status/tax42 owner repair, inventories, CI/source closure, scientific/native -capacity and release/default/PR893 decisions remain separately owned. - ---- - -# Lane: verify native sources once per run (`native-verify-once`) - -Branch `native-verify-once`, worktree `~/PolicyEngine/_worktrees/microcosm-verify-once`, -base `f7bb88525a78786f91bc3ebe2083ef4b1c85de18` (PR #893 head). Started -2026-09-15. - -**State (2026-09-16, third session):** all five mechanisms landed with proofs -and the epoch wired into both atomic capsules; draft PR #935 open; the -CI-shaped test battery is green (no test group failed; `ruff format --check .` -is red on 77 pre-existing files that this branch does not touch, and CI does -not run it); the before/after probe and the nineteen-node harness have run; the -lane report `experiments/native-verify-once/out.md` is complete, with the -measurement, the pins, the verbatim summaries and the open questions. The open -work is the main-only split (its commits sit on -`graph-verify-once-main-stale-20260916-0057`, unpushed) and Max's answers to the -report's open questions. -*(Historical as of 2026-09-17: the split is now draft PR #938, and an -adversarial verification of this branch has since been resolved by ten further -commits on a rebased branch — see "Done (2026-09-17, fourth session)" below.)* - -**Goal.** The 9/15 pilot v5 measurement -(`~/PolicyEngine/_recovered/pilot-runs/native45-v5/out.md` §2) attributes ~79 % -of a native 1/1000 run to repeated admission and verification of 3.48 GiB of -staged source: node execution is 1,143 s of a 5,362 s run. Make source -authentication and identity verification happen once per run instead of once -per accessor use and once per executed node, without weakening any refusal and -without moving any digest value. - -**Done:** read the evidence base; located all five mechanisms at base HEAD. - -**Done (2026-09-16):** - -- `docs/us-native-verification-once.md` — the design note, written before any - capsule changed. -- **Mechanism 5**: `_object_stream` builds each plain float/int/bool column's - context-digest bytes with numpy. Byte-identical against a verbatim copy of the - pre-change body over every column kind, the float specials, non-canonical NaN - payloads, both int64 endpoints and a 200-frame random sweep. 0.47 s -> 0.09 s - on a 6,928 x 240 frame. -- **Mechanism 4**: the ACS record fence now uses `bytes.find`/`bytes.count` with - cached terminator cursors. Proven against the byte loop on 46,655 exhaustive - short strings across five ceiling settings, 4,500 random strings, and the real - staged `csv_pus.zip`: 3,422,890 records, identical digests, 226.9 s -> 4.6 s - (49x). Re-pins `_ACCEPTED["acs_person_coverage_authentication.py"]`. -- **Mechanism 3**: `run_graph` carries a `_SourceIdentities` cache keyed on stat - signatures, and re-derives every source in full before building the manifest. - `RunManifest.source_identities` records it without moving any existing value. - The refusal it protects had no test at all; `test_graph_executor_source_identity` - now pins it. Graph suite 108 s -> 72 s. -- **Mechanisms 1 and 2**: `survey_population_preparation.verification_epoch()`, - an opt-in scoped memo. Cheap tier every borrow (live authority, attached - payloads, producer encoding); expensive tier skipped only while a signature - over every path it reads — the roster stat identities among them — and every - live buffer it digests is unchanged; unconditional full re-validation on - leaving the epoch. The two moved inventory contracts are re-derived (below). - (The roster stat identities were compared in the cheap tier when this line - was first written; the fourth session moved them into the signature so a - memo miss raises the code an unmemoised borrow raises. See below.) - -**Done (2026-09-16, second session):** - -- Draft PR [#935](https://github.com/PolicyEngine/microcosm/pull/935), base - `microcosm-us-launch-integration-20260909`. -- Main-only split branch `graph-verify-once-main`, worktree - `~/PolicyEngine/_worktrees/microcosm-graph-verify-once-main`, branched from - `origin/main` (51c314382). It carries the graph-shard change only: the three - commits re-applied, plus a changelog fragment. Its one deliberate difference - from this branch is that `_update_scalar` has no exact-float `struct.pack` - shortcut on main -- that belongs to PR #893 -- so the parity test's reference - copy was aligned to main's actual pre-change body, which makes it a literal - verbatim copy there too. -- Before/after measurement staging in `.measure/` (gitignored): a parameterised - copy of the v5 pilot probe that takes the source tree, the staged run inputs - and the output directory from the environment, asserts every imported - `microcosm` module resolves inside the measured tree, and enforces a 16 GiB - resident ceiling alongside the CPU one. Baseline worktree - `~/PolicyEngine/_worktrees/microcosm-verify-once-baseline` is detached at - `f7bb88525`. - -**Done (2026-09-17, fourth session) — the verification findings:** an -adversarial static verification of the branch returned four medium and seven low -findings and no high one -(`~/PolicyEngine/_recovered/scratch-backup/893/lanes/verify-once-findings-20260916.json`). -Ten commits answer them, on a branch rebased onto the base tip `363a9033b`: - -- rebased onto `363a9033b` and force-pushed, so two-dot and three-dot diffs - agree again (26 files, +5,422 / −819); -- a mutation test per memoised capsule that changes file **content** while the - cheap tier's view of it stays byte-identical, so the memo-miss branch itself - refuses rather than a stat comparison in front of it; -- the roster stat identities moved out of the refusing cheap tier into the - signature, so an in-epoch refusal carries the code an unmemoised borrow - carries (behaviour, not documentation: the design note promised it and the - fix is smaller than the divergence, and it deletes a redundant `_file_stats` - walk per borrow); -- `verification_epoch()`'s record is bound by both runners and attached to the - manifest as `RunManifest.verification_epoch` — outside the key, the JSON, - every node receipt and every cache record. A real nine-node run over invented - sources reports `{"capsules": 1, "hits": 20, "misses": 3, - "final_validations": 1}`; -- `_source_stat_signature` follows a member symlink, as `_directory_identity` - does, so a target rewritten mid-run refuses at the next node rather than at - run end; `_object_stream` packs floats in native order, as the loop it - replaces does; -- the dead `_stat_or_absent` is deleted; the four `_path_stat` copies and the - two `_stat_identity` definitions stay, with the pin cost of consolidating - them re-derived through the modules' own generator and stated in the report; -- the design note's five inaccuracies are corrected, the nested finalizer - chains its refusal instead of dropping it, and the report's stale items are - historicized. - -Re-run afterwards, all rc 0: `769 passed, 1 skipped in 44.56s` -(`packages/microcosm-graph/tests`), `128 passed in 188.39s` (the four new test -files), `1811 passed, 1 skipped, 38 warnings in 11200.54s (3:06:40)` (the 46 -build test files that reach a changed module), plus `ruff check` clean, -`17 files already formatted`, `uv lock --check`, both `ci_test_groups` checks. - -**Done (2026-09-17, second fix pass) — the two residuals:** a re-verification of -the ten commits above left two low findings, and two commits answer them. (1) -Only the nine-node runner had a test that read `RunManifest.verification_epoch` -back off a real run, so -`test_nineteen_node_financial_cold_and_required_replay` now reads it off both -manifests the financial runner returns — its own epoch's and the nested -population epoch's — for the cold run and the required replay, asserting the -protocol label, `final_validations == capsules >= 1`, and that `to_json` still -cannot see it; and the committed probe -(`experiments/native-verify-once/probe_verify_once.py`) now copies the record -into its output as `verification_epoch`, so the next measured run is the first -that will carry counts. Nothing was re-measured, and the three measurement files -the report quotes predate that line, which the report, the README and the design -note all state. (2) This journal's "Next" and the report's open question 4 -restated #938 at its current head and size, with the mirror recorded as landed. -No `packages/*/src` file changed in either commit. - -Re-run, both rc 0: `7 passed in 119.98s (0:01:59)` -(`test_us_graph_atomic_survey_financial.py`) and -`24 passed in 157.61s (0:02:37)` (`test_us_native_verify_once_epoch.py`), plus -`ruff check` and `ruff format --check` clean on the two changed Python files. - -**Done (2026-09-17, third fix pass) — one medium and two low:** a second -re-verification found that the 2026-09-16 refusal-code fix had weakened one -at-borrow refusal, and three commits answer it and the two low findings. -(1) `_finalize_epoch` recorded its memo signature *after* validating, so at an -inner nested close — whose memo survives into the outer epoch — a roster file -whose stat moved during the trailing part of `_validate` was absorbed as the -new normal and the outer borrows were hits until the outermost close. Each -close now takes the signature before validating and again after and records -none when they differ, so the next borrow is a miss that refuses; -`asec_2024_native_population._epoch_exit` had the same ordering and is fixed -the same way. Two new tests move a source from a profile hook as that close's -own validation returns and fail with `DID NOT RAISE` against the previous -ordering. `_memoized_validate`'s signature handler also narrowed from -`BaseException` to `Exception`, which is what its own comment describes. -(2) `RunManifest.verification_epoch` and `run_graph(_verification_epoch=)` had -no graph-shard test on either branch: `test_graph_verification_epoch.py` adds -nine, and two build-shard tests cover the signature-cannot-be-taken handler and -the `raise own from nested` branch. (3) The stale cheap-tier descriptions the -2026-09-16 fix left behind — test-file docstrings, the module block comment, the -design note and this report — now say what the code does, and `executor.py:2100` -is corrected to `:2105`. Two `packages/*/src` files changed and no pin moved: -`graph_implementation._dependency_contract` was re-run over both edited modules -and `imports`, `resource_accesses_sha256` and `unbound_uses_sha256` are all -unchanged. - -Re-run, all rc 0: `778 passed, 1 skipped in 37.72s` -(`packages/microcosm-graph/tests`, up from 769 — the new -`test_graph_verification_epoch.py` adds nine), `28 passed in 190.65s (0:03:10)` -(`test_us_native_verify_once_epoch.py`, up from 24), -`1607 passed, 38 warnings in 8311.33s (2:18:31)` (the 34 build test files naming -either changed module), `8 passed in 0.68s` -(`test_us_asec_prepared_resources.py`, which builds the stage implementation -manifests), and `597 passed, 2 skipped in 17.75s` in the -`graph-verify-once-main` worktree. Plus `ruff check .`, `ruff format --check` on -all five changed Python files, `uv lock --check`, both `ci_test_groups` checks. - -**Next:** Max's rulings on the report's remaining open questions. Open question -4 is answered and needs nothing further: the split is draft PR -[#938](https://github.com/PolicyEngine/microcosm/pull/938), opened 2026-09-16 -and now at head `33f3150bb` (base `main`, 7 files +1,430/−8, MERGEABLE, still -draft; `gh pr view 938`, 2026-09-17 07:43 UTC). The graph-shard commits made -after the verification findings were mirrored into it on 2026-09-17 at -06:45–06:46 UTC — the verification epoch on the manifest, the member-symlink -follow, the native-order float cast and the docstring fix — and the third fix -pass's new `test_graph_verification_epoch.py` followed as `33f3150bb`. Diffed -the same morning, its five earlier graph files differ from this branch's only by -the documented `import struct` and exact-`float` hunks that belong to #893's -base. -#935's body already carries the measurement table. Measured: nine-node prefix -before 1,803.87 CPU s without completing (ceiling) against after 1,444.78 CPU s -completing; nineteen-node 5,278.61 -> 2,010.07 CPU s (2.63x) against the v4 cold -receipt, peak RSS 9.31 -> 13.31 GB. - -**Pins re-derived so far:** +1. `docs/us-native-retention-seal.md` — the four required answers. +2. Discrimination battery (pre-change), then implement. +3. 1/1000 cold + required replay; then the 1/15 run. -| pin | old | new | -|---|---|---| -| `acs_native_coverage_binding._ACCEPTED["acs_person_coverage_authentication.py"]` | `475aa795…fe85bcff` | `9ec68721…d88e8e49f` | -| `graph_implementation_inventory.json` `survey_population_preparation.py` `unbound_uses_sha256` | `29c09f6f…d296ef91` | `d114117c…dd4005910` | -| `graph_implementation_inventory.json` `asec_2024_native_population.py` `unbound_uses_sha256` | `71463df4…d7f287608` | `7bb20439…40d58cdf4` | +## Open findings (provisional, to be proved) -**Lane notes.** Root `out.md` is a tracked file holding the Amendment 19 lane's -committed report; this lane's report goes to -`experiments/native-verify-once/out.md`. The v5 cold run (`run_v5.py`, pid -81194) is live on this machine, so probe runs wait for the window the lane -brief allows (`pgrep -f run_v5.py` empty, or > 40 GB free). +- `_population_stamp` is documented as **not** comparable across + reconstructions (`survey_atomic_geography.py:230-237`): it folds masked + `_data` under nulls. `same_replayed_population` deliberately *permits* the + store's zeroed null backing (`survey_population_replay.py:79-82`, + `NONCANONICAL_NULL_BACKING`). So a stamp-equality seal would refuse runs the + object comparison accepts. The seal this lane needs is a **replay seal**, not + the stamp. From 64b6533e960426ad21675d865f0d6e0663be2178 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 17:42:30 -0400 Subject: [PATCH 02/75] Prove neither existing seal is the seal, with the scripts that show it _population_stamp refuses a store round trip that same_replayed_population accepts; _frame_identity misses NaN payload bits and DataFrame.flags. The transport report's proposed mechanism for option (b) does not hold as written. Co-Authored-By: Claude Opus 5 --- PROGRESS.md | 50 +++++++++++++++---- .../probe_frame_identity_gaps.py | 45 +++++++++++++++++ .../probe_stamp_vs_comparison.py | 48 ++++++++++++++++++ 3 files changed, 134 insertions(+), 9 deletions(-) create mode 100644 experiments/native-retention-seal/probe_frame_identity_gaps.py create mode 100644 experiments/native-retention-seal/probe_stamp_vs_comparison.py diff --git a/PROGRESS.md b/PROGRESS.md index 90c30e851..4672132c0 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -13,7 +13,9 @@ afterwards — check git/GitHub for current truth. ## State -Reading the authorities and the code at this head. Nothing implemented. +Design established and proved by running code at this head. The transport +report's proposed mechanism for option (b) does not hold; the seal has to be +purpose-built. Nothing implemented yet. ## Done @@ -33,12 +35,42 @@ Reading the authorities and the code at this head. Nothing implemented. 2. Discrimination battery (pre-change), then implement. 3. 1/1000 cold + required replay; then the 1/15 run. -## Open findings (provisional, to be proved) +## Findings, proved by running code at this head -- `_population_stamp` is documented as **not** comparable across - reconstructions (`survey_atomic_geography.py:230-237`): it folds masked - `_data` under nulls. `same_replayed_population` deliberately *permits* the - store's zeroed null backing (`survey_population_replay.py:79-82`, - `NONCANONICAL_NULL_BACKING`). So a stamp-equality seal would refuse runs the - object comparison accepts. The seal this lane needs is a **replay seal**, not - the stamp. +**Neither existing seal is the seal.** The transport report's §10 question 2(b) +says "`_population_stamp` already folds everything `same_replayed_population` +compares except the *type* assertions". That sentence is wrong in **both** +directions, and both halves are proved by scripts committed under +`experiments/native-retention-seal/`: + +1. **`_population_stamp` is too strict.** + `probe_stamp_vs_comparison.py` builds a US_SCHEMA population, round-trips its + frame through `ContentStore.put_frame`/`load_frame`, and gets: + `same_replayed_population` **ACCEPTS** (the store zeroed `_data` under the + null mask, which `NONCANONICAL_NULL_BACKING`, + `survey_population_replay.py:79-82`, deliberately permits) while + `_population_stamp(expected) != _population_stamp(actual)`. So a + stamp-equality seal would **refuse a required replay**. Its own docstring + says so (`survey_atomic_geography.py:231-237`). +2. **`_frame_identity` is too weak.** `probe_frame_identity_gaps.py` finds three + discriminations `same_replayed_frame` makes that `_frame_identity` does not: + float64 NaN **payload bits** (two quiet NaNs; `NATIVE_BITS`), quiet versus + signalling NaN (`NATIVE_BITS`), and + `DataFrame.flags.allows_duplicate_labels` (`TABLE_TYPE_OR_FLAGS`). `_cell` + maps every NaN to `None`, so `_frame_identity` spells all of them `null`. + It also cannot be applied to a non-US_SCHEMA frame at all (`FRAME_TYPE`). + +**Therefore the seal is purpose-built and lives in `survey_population_replay.py`, +beside the comparison it replaces**, folding exactly the bytes each comparison +compares — no more, no less. + +**A fourth gap, from pandas rather than from this repo.** +`pd.DatetimeIndex._comparables == ['name', 'freq']`, and two DatetimeIndexes with +equal values, dtype and name but different `freq` are **not** `identical()` while +their bytes are equal. Any seal that folds only (class, dtype, name, bytes) +misses it. The seal folds `type(index)._comparables` generically. + +**Admitted dtypes are narrow**, which bounds the problem: `_series` refuses +`CategoricalDtype`, `Float64Dtype` and `DatetimeTZDtype` with +`UNSUPPORTED_EXTENSION_DTYPE`; only masked integer/boolean, `StringDtype`, +`object` and plain numpy dtypes get through. diff --git a/experiments/native-retention-seal/probe_frame_identity_gaps.py b/experiments/native-retention-seal/probe_frame_identity_gaps.py new file mode 100644 index 000000000..351529ba9 --- /dev/null +++ b/experiments/native-retention-seal/probe_frame_identity_gaps.py @@ -0,0 +1,45 @@ +"""Is _frame_identity WEAKER than same_replayed_frame anywhere? NaN payloads, flags, index freq.""" +import numpy as np, pandas as pd +from dataclasses import replace +from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights +from microcosm.graph.population import Population +from microcosm.build.us_runtime.survey_population_replay import same_replayed_frame +from microcosm.build.us_runtime import survey_population_preparation as prep + +def build(nan_bits=0x7FF8000000000011, flags=True, extra=None): + person = pd.DataFrame({ + "person_id": np.arange(3, dtype=np.int64), + "person_household_id": np.array([0,0,1], dtype=np.int64), + "person_tax_unit_id": np.array([0,0,1], dtype=np.int64), + "person_spm_unit_id": np.array([0,0,1], dtype=np.int64), + "person_family_id": np.array([0,0,1], dtype=np.int64), + "person_marital_unit_id": np.array([0,1,2], dtype=np.int64), + "native_float": np.array([nan_bits, 0x3FF4000000000000, 0x8000000000000000], + dtype=np.uint64).view(np.float64), + }) + person.flags.allows_duplicate_labels = flags + groups = {e: pd.DataFrame({f"{e}_id": np.array([0,1], dtype=np.int64)}) + for e in ("household","tax_unit","spm_unit","family")} + groups["marital_unit"] = pd.DataFrame({"marital_unit_id": np.array([0,1,2], dtype=np.int64)}) + return Frame({"person": person, **groups}, US_SCHEMA, + {"household": Weights(np.array([1.5,2.5]), WeightKind.IMPORTANCE)}, + pd.Series(["urban","urban","rural"], dtype=object)) + +def report(label, a, b): + try: + same_replayed_frame(a, b); cmp = "ACCEPTS" + except ValueError as e: cmp = "REFUSES " + str(e) + try: + eq = prep._frame_identity(a) == prep._frame_identity(b); ident = "same" if eq else "differs" + except Exception as e: ident = "ERR " + type(e).__name__ + ":" + str(e) + print(f"{label:34s} same_replayed_frame={cmp:52s} _frame_identity={ident}") + +# 1. NaN payload bits +report("nan payload 0x...11 vs 0x...12", + build(0x7FF8000000000011), build(0x7FF8000000000012)) +# 2. DataFrame.flags +report("flags allows_duplicate_labels", + build(flags=True), build(flags=False)) +# 3. quiet vs signalling NaN +report("quiet vs signalling NaN", + build(0x7FF8000000000000), build(0x7FF0000000000001)) diff --git a/experiments/native-retention-seal/probe_stamp_vs_comparison.py b/experiments/native-retention-seal/probe_stamp_vs_comparison.py new file mode 100644 index 000000000..d76e4a29d --- /dev/null +++ b/experiments/native-retention-seal/probe_stamp_vs_comparison.py @@ -0,0 +1,48 @@ +"""US_SCHEMA version: _population_stamp vs same_replayed_population on a store round trip.""" +import pathlib, tempfile +from dataclasses import replace +import numpy as np, pandas as pd +from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights +from microcosm.graph.population import Population +from microcosm.graph.store import ContentStore +from microcosm.build.us_runtime.survey_population_replay import same_replayed_population +from microcosm.build.us_runtime import survey_atomic_geography as geo +from microcosm.build.us_runtime import survey_population_preparation as prep + +n = 3 +person = pd.DataFrame({ + "person_id": np.arange(n, dtype=np.int64), + "person_household_id": np.array([0, 0, 1], dtype=np.int64), + "person_tax_unit_id": np.array([0, 0, 1], dtype=np.int64), + "person_spm_unit_id": np.array([0, 0, 1], dtype=np.int64), + "person_family_id": np.array([0, 0, 1], dtype=np.int64), + "person_marital_unit_id": np.array([0, 1, 2], dtype=np.int64), + "nullable_integer": pd.arrays.IntegerArray( + np.array([11, 71, 73], dtype=np.int64), np.array([False, True, True])), +}) +groups = { + "household": pd.DataFrame({"household_id": np.array([0, 1], dtype=np.int64)}), + "tax_unit": pd.DataFrame({"tax_unit_id": np.array([0, 1], dtype=np.int64)}), + "spm_unit": pd.DataFrame({"spm_unit_id": np.array([0, 1], dtype=np.int64)}), + "family": pd.DataFrame({"family_id": np.array([0, 1], dtype=np.int64)}), + "marital_unit": pd.DataFrame({"marital_unit_id": np.array([0, 1, 2], dtype=np.int64)}), +} +frame = Frame({"person": person, **groups}, US_SCHEMA, + {"household": Weights(np.array([1.5, 2.5]), WeightKind.IMPORTANCE)}, + pd.Series(["urban", "urban", "rural"], dtype=object)) +tmp = pathlib.Path(tempfile.mkdtemp(prefix="stampgap2-")) +store = ContentStore(tmp / "store") +store.put_frame("a" * 64, frame) +loaded = store.load_frame("a" * 64) + +expected = Population.from_frame(frame, "allocated") +actual = replace(expected, frame=loaded) +print("expected masked _data:", expected.frame.person["nullable_integer"].array._data.tolist()) +print("actual masked _data:", actual.frame.person["nullable_integer"].array._data.tolist()) +try: + same_replayed_population(expected, actual); print("same_replayed_population: ACCEPTS") +except ValueError as e: print("same_replayed_population: REFUSES", e) +se, sa = geo._population_stamp(expected), geo._population_stamp(actual) +print("_population_stamp equal :", se == sa, se[:12], sa[:12]) +fe, fa = prep._frame_identity(expected.frame), prep._frame_identity(actual.frame) +print("_frame_identity equal :", fe == fa, fe[:12], fa[:12]) From fa2137f5b4b49c4dd77e57787ecc91ffc4e70d7d Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 17:45:33 -0400 Subject: [PATCH 03/75] Write the design note: the four answers, and why neither existing seal is it Answer (4) is 'nothing about the content'; the two things that change in character (sha256 for bytes, unary assertions firing on arrival) are stated rather than hidden, and neither narrows what a run proves about the data. Co-Authored-By: Claude Opus 5 --- docs/us-native-retention-seal.md | 383 +++++++++++++++++++++++++++++++ 1 file changed, 383 insertions(+) create mode 100644 docs/us-native-retention-seal.md diff --git a/docs/us-native-retention-seal.md b/docs/us-native-retention-seal.md new file mode 100644 index 000000000..0d052268a --- /dev/null +++ b/docs/us-native-retention-seal.md @@ -0,0 +1,383 @@ +# US native build: replacing the replayed-population object comparison with a content seal + +Design authority for the change Max chose on 2026-09-17 — option (b) of the +transport lane's report §10 question 2 +(`experiments/native-scale-transport/out.md`), whose own authority is +`docs/us-native-scale-transport.md` §3d and §4. + +The problem, restated from that authority and re-derived here: the financial +runner compares its own replay against the executor's observation with +`atomic.same_replayed_population(expected[n], observed[n])` +(`graph_atomic_survey_financial.py:1796`, `:1948`), and the replay cannot be +built until `run_graph` has returned. So nineteen detached `Population` objects +must exist between observation and comparison — about 290 GiB at full source — +unless the comparison becomes a comparison of content seals. + +Every line number is at this branch's head unless it names another. Line +numbers move; re-derive the ones you rely on. + +## 0. The report's proposed mechanism does not hold, and this is the first thing to say + +Report §10 question 2(b) says: + +> `_population_stamp` already folds everything `same_replayed_population` +> compares except the *type* assertions, which can be made on arrival without +> retaining. + +That sentence is wrong in **both** directions. Both halves are proved by +scripts committed under `experiments/native-retention-seal/`, run at this head. + +**`_population_stamp` is too strict — it would refuse a required replay.** +`probe_stamp_vs_comparison.py` builds a `US_SCHEMA` population whose +`nullable_integer` column carries `_data = [11, 71, 73]` under a mask +`[False, True, True]`, round-trips its frame through +`ContentStore.put_frame`/`load_frame` (which zeroes values beneath a null mask, +`store.py:456`), and reports: + +``` +expected masked _data: [11, 71, 73] +actual masked _data: [11, 0, 0] +same_replayed_population: ACCEPTS +_population_stamp equal : False eb6879921de1 0400e853a2dd +_frame_identity equal : True 774ed8184472 774ed8184472 +``` + +`same_replayed_population` **accepts** that pair, because +`NONCANONICAL_NULL_BACKING` (`survey_population_replay.py:79-82`) deliberately +permits the actual side's null backing to be canonically zeroed. The stamp +refuses it. `_population_stamp`'s own docstring says exactly this +(`survey_atomic_geography.py:231-237`): "Never persist it or compare it across +reconstructions: it folds physical storage parts (masked `_data` under nulls)". +A stamp-equality seal would turn every `resume="require"` replay red. + +**`_frame_identity` is too weak — it would miss real defects.** +`probe_frame_identity_gaps.py` finds three discriminations `same_replayed_frame` +makes that `_frame_identity` does not: + +``` +nan payload 0x...11 vs 0x...12 same_replayed_frame=REFUSES NATIVE_BITS _frame_identity=same +flags allows_duplicate_labels same_replayed_frame=REFUSES TABLE_TYPE_OR_FLAGS _frame_identity=same +quiet vs signalling NaN same_replayed_frame=REFUSES NATIVE_BITS _frame_identity=same +``` + +`_cell` (`survey_population_preparation.py:732-745`) maps **every** NaN to +`None`, so `_frame_identity` spells all NaN payloads `null`; and nothing in it +folds `DataFrame.flags`. It is also inapplicable to a non-`US_SCHEMA` frame at +all (`FRAME_TYPE`, `:977-981`). + +**So the seal is purpose-built and lives beside the comparison it replaces**, in +`survey_population_replay.py`, folding exactly the bytes each comparison +compares — no more, no less. It reuses neither existing seal. + +## 1. Every field and assertion `same_replayed_population` compares today + +Read off `survey_population_replay.py` at this head. **Kind** is `unary` for a +predicate about one side that can be asserted on arrival, `binary` for a +comparison between the two sides that a seal must fold, and `directional` for +the one predicate that is not symmetric. + +### 1a. Population level (`:179-208`) + +| # | predicate | code | kind | +|---|---|---|---| +| P1 | `type(expected) is Population`, `type(actual) is Population` | `POPULATION_TYPE` | unary, both sides | +| P2 | `type(version) is str` (both) and `expected.version == actual.version` | `POPULATION_CONTEXT` | unary + binary | +| P3 | `tuple(sorted(owners.items()))` equal — order-**insensitive** | `POPULATION_CONTEXT` | binary | +| P4 | `all(type(v) is str for v in actual.owners.values())` | `POPULATION_CONTEXT` | unary, actual only | +| P5 | `tuple(weight_kind.items())` equal — order-**sensitive** | `POPULATION_CONTEXT` | binary | +| P6 | `all(type(v) is WeightKind for v in actual.weight_kind.values())` | `POPULATION_CONTEXT` | unary, actual only | +| P7 | `type(mass_ledger) is tuple` (both) | `POPULATION_CONTEXT` | unary | +| P8 | `all(type(r) is MassRecord ...)` over both ledgers | `POPULATION_CONTEXT` | unary | +| P9 | `canonical_json([asdict(r) for r in mass_ledger])` equal | `POPULATION_CONTEXT` | binary | +| P10 | `tuple(design_weights)` equal — key **order** | `POPULATION_CONTEXT` | binary | +| P11 | per entity `_array_bytes_equal(design_weights[e], …)`: `type is np.ndarray`, dtype equal, `not hasobject`, shape equal, `tobytes()` equal | `DESIGN_BYTES` | unary + binary | + +### 1b. Frame level (`:132-176`) + +| # | predicate | code | kind | +|---|---|---|---| +| F1 | `isinstance(expected, Frame)`, `isinstance(actual, Frame)` | `FRAME_TYPE` | unary | +| F2 | `schema` equal | `FRAME_CONTEXT` | binary | +| F3 | `entities` equal (ordered tuple) | `FRAME_CONTEXT` | binary | +| F4 | `links == () ` on **both** | `FRAME_CONTEXT` | unary | +| F5 | `canonical_json(_encode_frame_metadata(metadata))` equal | `FRAME_CONTEXT` | binary | +| F6 | `all(type(r) is MassChangeRecord ...)` over both mass logs | `FRAME_CONTEXT` | unary | +| F7 | `canonical_json([asdict(r) for r in mass_log])` equal | `FRAME_CONTEXT` | binary | +| F8 | `weighted_entities` equal | `FRAME_CONTEXT` | binary | +| F9 | `type(table) is pd.DataFrame` (both) | `TABLE_TYPE_OR_FLAGS` | unary | +| F10 | `left.flags == right.flags` | `TABLE_TYPE_OR_FLAGS` | binary | +| F11 | strata name: `_name_bytes` equal | `STRATA_NAME` | binary | +| F12 | per weighted entity `left.kind is right.kind` | `WEIGHT_BYTES` | binary | +| F13 | per weighted entity `_array_bytes_equal(values)` | `WEIGHT_BYTES` | unary + binary | + +### 1c. Axis (`:121-129`), applied to every table index, every table columns axis, and the strata index + +| # | predicate | code | kind | +|---|---|---|---| +| A1 | `type(expected) is type(actual)` | `AXIS` | binary | +| A2 | `expected.identical(actual)` — pandas: `equals()` **and** every name in `type(index)._comparables` **and** exact class **and** dtype | `AXIS` | binary | +| A3 | `not isinstance(expected, pd.MultiIndex)` | `AXIS` | unary (expected; A1 carries it to actual) | +| A4 | `_name_bytes(name)` equal — `canonical_json(_axis_name_payload(name))` | `AXIS_NAME` | binary, plus unary encodability (`UNSUPPORTED_AXIS_NAME`) | +| A5 | `_series` over the axis array | the S codes | | + +### 1d. Series (`:48-111`), applied to every table column, every axis array and the strata + +| # | predicate | code | kind | +|---|---|---|---| +| S1 | `type(x) is pd.Series` (both) | `SERIES_DTYPE_OR_LENGTH` | unary | +| S2 | `type(expected.dtype) is type(actual.dtype)` | `SERIES_DTYPE_OR_LENGTH` | binary | +| S3 | `expected.dtype == actual.dtype` | `SERIES_DTYPE_OR_LENGTH` | binary | +| S4 | `len(expected) == len(actual)` | `SERIES_DTYPE_OR_LENGTH` | binary | +| **masked branch** — `BooleanDtype`, or `ExtensionDtype` and integer | | | | +| S5 | `_array_bytes_equal(lm, rm)` — the null masks | `MASKED_STORAGE` | binary | +| S6 | `lm.dtype is bool`; `lm.shape == (n,)`; `type(ld) is type(rd) is np.ndarray`; `not ld.dtype.hasobject`; `ld.shape == rd.shape == lm.shape` | `MASKED_STORAGE` | unary + shape | +| S7 | `ld.dtype == rd.dtype` | `MASKED_STORAGE` | binary | +| S8 | `_array_bytes_equal(ld[~lm], rd[~rm])` — the **present** values | `PRESENT_BITS` | binary | +| S9 | `_array_bytes_equal(ld, rd) or bool(np.all(rd[rm] == 0))` | `NONCANONICAL_NULL_BACKING` | **directional** | +| **string branch** — `StringDtype` | | | | +| S10 | `storage` equal and `(na_value is pd.NA)` parity | `STRING_POLICY` | binary | +| S11 | `_array_bytes_equal(isna(l), isna(r))` | `STRING_MASK` | binary | +| S12 | present cells: `type(a) is str and type(b) is str and a == b` | `STRING_VALUE` | unary + binary | +| **object branch** — `is_object_dtype` | | | | +| S13 | per cell `_encode_object_scalar(a) == _encode_object_scalar(b)` | `OBJECT_VALUE` | binary, plus unary encodability (`UNSUPPORTED_OBJECT`) | +| **native branch** | | | | +| S14 | `not isinstance(dtype, pd.api.extensions.ExtensionDtype)` | `UNSUPPORTED_EXTENSION_DTYPE` | unary | +| S15 | `_array_bytes_equal(to_numpy(l), to_numpy(r))` — type, dtype, `not hasobject`, shape, bytes | `NATIVE_BITS` | unary + binary | + +Forty-four discriminations, eleven refusal codes. The admitted dtype set is +narrow, which bounds the problem: `_series` refuses `CategoricalDtype`, +`Float64Dtype` and `DatetimeTZDtype` with `UNSUPPORTED_EXTENSION_DTYPE` +(measured), so only masked integer/boolean, `StringDtype`, `object` and plain +numpy dtypes get through. + +## 2. Which of them `_population_stamp` already folds + +`_population_stamp` (`survey_atomic_geography.py:230-268`) folds, in order: +`canonical_json({"frame": _frame_identity(frame), "version", sorted owners, +weight_kind items, mass_ledger asdict})`; then, for every entity column in +order and then the strata, a length-prefixed `_storage_parts(series, +slice(None))` pair (physical values, null bitmap); then, per design-weight +entity, `canonical_json((entity, str(dtype), shape))` and `values.tobytes()`. + +| group | folded? | note | +|---|---|---| +| P2 version, P3 owners, P5 weight_kind, P9 mass ledger | yes | in the header `canonical_json` | +| P10 design key order, P11 design bytes | yes | the design loop | +| F2 schema, F3 entities, F8 weighted entities | yes | through `_frame_identity` / `encode_us_frame_context` | +| F5 metadata, F7 mass log | yes, but through `_normative_metadata` and `_json_data`, not `_encode_frame_metadata` | a **different** normalisation from the one the comparison uses | +| F12/F13 weights | yes | `_frame_identity` folds kind, dtype, shape and `tobytes()` | +| S5/S8 masks and present values | yes | `_storage_parts` | +| S15 native bytes | yes | `_storage_parts` | +| A4 axis names | partly | `_frame_identity` folds index `names` and the strata name; not through `_axis_name_payload` | +| **S9 directional null backing** | **folded the wrong way** | it folds the masked `_data`, so it **refuses** what the comparison accepts — §0 | +| **F10 `DataFrame.flags`** | **no** | nothing folds it | +| **A1/A2 exact index class and `_comparables`** | partly | `type(index).__name__` only; not the qualified class, not `freq` | +| all unary type assertions (P1, P4, P6, P7, P8, F1, F4, F6, F9, S1, S6, S12, S14) | no | they are assertions, not content | + +So the honest summary is: `_population_stamp` folds most of the *content*, gets +the one directional predicate backwards, misses `DataFrame.flags` outright, and +— unavoidably — folds none of the unary assertions. + +## 3. Which it does not, and how each is preserved + +Three mechanisms, applied in this order. + +**(a) Every unary assertion is made on arrival, in the seal constructor, with +the same refusal code.** `replayed_population_seal(population)` runs P1, P4, +P6, P7, P8, F1, F4, F6, F9, S1, S6, S12, S14, A3 and the two encodability +assertions (`UNSUPPORTED_OBJECT`, `UNSUPPORTED_AXIS_NAME`) against the one +population it is given, raising the identical +`SURVEY_POPULATION_REPLAY_` string. Four of them (P4, P6, and the +`actual`-side halves of the rest) are written today as assertions about +`actual` only; sealing both sides makes them fire on `expected` too. That is +strictly more closed, it fires earlier, and no code changes. + +Two are re-expressed rather than dropped, because they are not properties of +one side: + +* **S2/S3, the dtype pair.** Instead of `type(a.dtype) is type(b.dtype) and + a.dtype == b.dtype`, the seal folds a **dtype token** + `(type(dtype).__module__, type(dtype).__qualname__, str(dtype), + getattr(dtype, "storage", None), getattr(dtype, "na_value", …) is pd.NA)`. + Token equality must be equivalent to the pair for every admitted dtype; the + battery proves it by sweeping a dtype census and asserting the equivalence + both ways. `str()` alone is not enough — `StringDtype("python")` and + `StringDtype("pyarrow")` both spell `string` — which is why `storage` is in + the token; and `CategoricalDtype`, whose `__eq__` is not an equivalence + relation, never reaches here because S14 refuses it. +* **A1/A2, `type(...)` and `Index.identical`.** The seal folds the **qualified + class** (`__module__` + `__qualname__`, rather than `_frame_identity`'s bare + `__name__`), the dtype token, and a digest over every name in + `type(index)._comparables`. That last part closes a gap neither existing + seal has: `pd.DatetimeIndex._comparables == ['name', 'freq']` (measured at + pandas 3.0.3), so two `DatetimeIndex`es with identical values, dtype and + name but different `freq` are **not** `identical()` while their bytes are + equal. `name` is folded through `_name_bytes`; any other comparable is + folded through `_name_bytes` when the store codec accepts it and through + `repr()` otherwise. §6 states the one residual risk that `repr` fallback + carries. + +**(b) Every binary comparison folds exactly the bytes that comparison +compares.** Not an approximation of them, and not a different normalisation: +F5 folds `canonical_json(_encode_frame_metadata(metadata))` because that is +what `same_replayed_frame` compares — *not* `encode_us_frame_context`'s +`_normative_metadata`, which is what `_frame_identity` folds. S15 folds +`to_numpy(copy=False).tobytes()` with its dtype and shape, so NaN payload bits +and `-0.0` survive, which `_cell`'s `None` mapping destroys. F10 folds +`allows_duplicate_labels` — generically, over `type(flags)._keys` (measured: +`{'allows_duplicate_labels'}`), so a pandas release that adds a flag is folded +without an edit here. + +**(c) The one directional predicate, S9, is folded as three values rather than +one digest**, which is what makes an exact reproduction possible at all. Per +series the masked branch records + +* `mask_digest` — the null mask bytes (S5), +* `present_digest` — the present values' bytes with dtype and shape (S7, S8), +* `full_digest` — the whole `_data` buffer's bytes, +* `null_backing_canonical` — `bool(np.all(data[mask] == 0))` for **this** side, + +and the seal comparison requires +`present_digest` equal **and** +(`actual.null_backing_canonical` **or** `full_digest` equal), +which is `_array_bytes_equal(ld, rd) or np.all(rd[rm] == 0)` term for term, per +series, with the same direction and the same `NONCANONICAL_NULL_BACKING` code. +A single population-wide flag would **not** reproduce it: today's disjunction is +evaluated per column, and one non-canonical column must not force whole-frame +byte equality on the others. + +The seal record is therefore a nested structure of 32-byte digests and small +tokens whose size is **O(columns), not O(rows)** — for a US frame of six +entities and a few hundred columns, tens of kilobytes per node against about +1.5 GiB for the retained population at 1/10. + +## 4. What is no longer proved + +**Nothing about the content.** Every one of the forty-four discriminations in +§1 still refuses the same defect with the same code; §5's battery is the proof +obligation and it is discharged mutation by mutation, in both directions. + +Two things change in character rather than in coverage, and this note states +them rather than leaving them to be found: + +1. **Byte equality becomes sha256 equality.** Today `same_replayed_population` + compares the bytes; the seal compares a 256-bit digest of those bytes. This + is the same substitution the whole runtime already rests on — + `_population_stamp`, `_frame_identity`, `preparation_sha256`, every node key + and every store address are sha256 digests of content — so it introduces no + assumption the build does not already make. It is not nothing, and it is + written here so the owner can see it. +2. **Unary assertions fire earlier.** A `MultiIndex`, an unencodable object + cell, a `Float64` column or a non-`str` owner value is refused when the + population is observed rather than when it is compared, with the same code. + For `expected`-side defects that is unchanged in timing; for `actual`-side + defects the refusal moves from after `run_graph` returns to inside the + observer callback, which `run_graph` documents as refusing the run + (`executor.py:2453-2454`). + +Neither is a narrowing of what a run proves about the data, so this note does +not stop and ask. The report §10's *stated mechanism* was wrong and this note +says so in §0 and replaces it; the *decision* — compare content seals, not +objects — is implemented as chosen. + +## 5. The proof obligation + +For every row of §1, a test that mutates exactly that property in one of two +otherwise-equal populations and asserts: + +* `same_replayed_population` refuses, with its code — this half already exists + for most rows in `packages/microcosm-build/tests/test_us_survey_population_replay.py` + and is extended to cover the rest; +* the **seal** comparison refuses the same mutation with the same code. + +Plus, in both directions, the pairs today's comparison **accepts** — the store +round trip of §0, the object-scalar equivalences of +`test_object_scalar_equivalence_uses_actual_store_types` — which the seal must +also accept. A mutation the seal misses is a finding, not a test to delete. + +## 6. Fail-closed after the change + +**Every refusal code that exists today still fires on the same defect.** The +eleven codes of §1 are raised by the seal constructor (unary) or the seal +comparison (binary), with the same `SURVEY_POPULATION_REPLAY_` prefix and the +same suffix. No code is retired and none is added. + +**`FINANCIAL_NODE_POPULATION_CHANGED` after the change** means: *for a node +whose `Population` object the run still retains — the declared-consumer roster — +the object's in-process `_population_stamp` differs from the one recorded when +the run was issued; and for every other node, the seal record the run retains +is not the one recorded at issuance.* Both arms refuse the same class of defect +the one arm refuses today — something the run retained about a node's +population changed after issuance — and neither arm is vacuous, because the +seal record is re-digested rather than merely re-read. + +**The declared-consumer roster.** `result.financial_population is +observed[final_node]` (`graph_atomic_survey_financial.py:1897`) is an identity +check, so the objects a caller holds must be the objects the observer saw. The +base financial run declares three: the final node (`financial.ATTACH_NODE`, or +`_tax_module().GATE_NODE` when the tax rebase is enabled), +`financial.ATTACH_NODE` itself (`legacy_population`, `:1821`), and the property +population (`observed[final_node]` before the rebase moves `final_node`, +`:1818`). Those three are retained as objects and stamped exactly as today. +Every other observation is sealed on arrival and dropped. + +**The completion host is out of scope and does not move.** +`graph_survey_completion_host` runs its own `run_graph` with its own observer +(`:781-794`), retains every observation, and pins their object ids in its own +identity fence (`:543-545`, `:572`). It neither passes the new executor mode +nor takes the new roster, so its two identity checks stay true by not changing. +`_node_population_seals` therefore accepts both shapes. + +**The residual risk this note will not hide.** The `_comparables` fallback in +§3(a) uses `repr()` for a comparable the store's axis-name codec will not +encode. `repr` is not guaranteed injective with respect to `==`, so a pair of +`==`-equal, differently-`repr`'d comparables would make the seal refuse where +today's `identical()` accepts. No index class in this runtime reaches it: US +frames carry `Index` and `RangeIndex`, whose `_comparables` is `['name']` +(measured), and `graph_context.encode_us_frame_context` requires group entities +to carry a default unnamed `RangeIndex` (`:130-143`). The seal folds the +comparables tuple itself, so a future index class with a wider one cannot pass +unnoticed. + +## 7. The `microcosm-graph` half + +`_observer_snapshot` (`executor.py:356-408`) detaches by round-tripping every +entity and link table plus the strata through `pickle.loads(pickle.dumps(..., +protocol=5))` — one full independent copy per reached node, cache hits included +(`:2773-2774`). An observer that only seals never retains and never mutates, so +it needs no detached copy. + +`run_graph` gains `_population_observer_detach: bool = True`, private and +paired with `_population_observer`. When it is `False` the observer receives the +live admitted population and no snapshot is allocated. Default behaviour is +unchanged, the keyword enters no key, no receipt and no cache record, and the +change imports nothing from the US runtime. + +**It is deliberately not called `population_retention`.** That keyword existed +on PR #893's original branch (commit `6c0f24c77`, layer 08) and was dropped; it +made the **manifest's attached population views** lazy against the content +store, which is a different target from the observer's per-node detached +snapshots (`docs/us-native-scale-transport.md` §4). Reusing the name for this +would invite exactly the confusion that document warns about. + +Invariants that do not move, each with the test that pins it +(`packages/microcosm-graph/tests/test_graph_executor.py`): + +| invariant | test | +|---|---| +| exactly once per reached node, in `compiled.order`, cold and on hits | `:4130-4166` | +| an observer's exception refuses the run with the observer's type | `:4172-4179` | +| a mutating, retaining observer cannot move a key, a manifest or a store object — **in the default detaching mode** | `:4183-4248` | +| with no observer, no snapshot is allocated | `:4368-4374` | + +The new tests are the shape of the last: with an observer **and** +`_population_observer_detach=False`, a monkeypatched `_observer_snapshot` that +raises must never be called; the run's manifest key and every store object must +equal a no-observer run's; and the observer must receive the same object the +executor holds, which is the mode's whole point and is what the caller is +declaring it will not mutate. + +That declaration is the mode's one honest cost: in this mode the executor no +longer *enforces* that the observer cannot reach execution state, it *trusts* +the caller's declaration. The default mode still enforces it, and this mode is +opt-in and private. From 5cf620e2f47d33aa04d6a72a0172771fb9d0815f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 17:48:25 -0400 Subject: [PATCH 04/75] Add the replay content seal beside the comparison it replaces Every byte-string comparison becomes a sha256 comparison; every other predicate keeps the object it applies to and the identical is/== operator; every one-sided predicate is asserted when the seal is built, with the same code. The directional NONCANONICAL_NULL_BACKING disjunction stays per series. Co-Authored-By: Claude Opus 5 --- .../us_runtime/survey_population_replay.py | 357 ++++++++++++++++++ 1 file changed, 357 insertions(+) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py b/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py index dd55a5e65..8c5f59d56 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py @@ -7,6 +7,7 @@ from __future__ import annotations +import hashlib from dataclasses import asdict import numpy as np @@ -206,3 +207,359 @@ def same_replayed_population(expected: Population, actual: Population) -> None: ), "DESIGN_BYTES", ) + + +# --- Content seals for the same comparisons, without retaining the objects --- +# +# ``same_replayed_population`` needs both Populations alive at once. A runner +# that observes nineteen node populations before it can build its own replay +# must therefore keep nineteen detached copies (about 290 GiB at full US +# source). The seal below carries, per node, what those comparisons actually +# compare, in space proportional to the number of columns rather than rows. +# +# Two existing seals were measured and are not this one, which is why this is +# a third: ``survey_atomic_geography._population_stamp`` folds masked ``_data`` +# beneath nulls, so it refuses the store round trip ``NONCANONICAL_NULL_BACKING`` +# deliberately accepts; and ``survey_population_preparation._frame_identity`` +# spells every NaN ``null`` and folds no ``DataFrame.flags``, so it misses +# ``NATIVE_BITS`` and ``TABLE_TYPE_OR_FLAGS`` defects. See +# ``docs/us-native-retention-seal.md`` section 0. +# +# The rule the record follows, and the only substitution it makes: +# +# * every comparison of BYTE STRINGS becomes a comparison of their sha256; +# * every other predicate is preserved verbatim, by retaining the small +# object it is applied to (a dtype, a class, an axis name, a WeightKind, +# a flags value) and applying the identical ``is``/``==`` to it; +# * every predicate about ONE side is asserted when the seal is built, with +# the same refusal code, so it fires on arrival instead of at comparison. +# +# The record is nested tuples of immutable values. It is never persisted and +# never compared across processes: the retained dtype and class objects are +# compared by identity, which is exactly what the object comparison does. + +SEAL_PROTOCOL = "microcosm.us.survey-population-replay-seal.v1" + + +def _seal_digest_bytes(*parts: bytes) -> bytes: + """sha256 over length-prefixed parts, so concatenation is unambiguous.""" + digest = hashlib.sha256() + for part in parts: + digest.update(len(part).to_bytes(8, "little")) + digest.update(part) + return digest.digest() + + +def _array_seal(values): + """The retained half of ``_array_bytes_equal``: its type/dtype/shape + bytes.""" + _require(type(values) is np.ndarray and not values.dtype.hasobject, "ARRAY_STORAGE") + return (values.dtype, values.shape, _seal_digest_bytes(values.tobytes())) + + +def _same_array_seal(expected, actual, reason): + _require( + expected[0] == actual[0] + and expected[1] == actual[1] + and expected[2] == actual[2], + reason, + ) + + +def _name_seal(value): + return _seal_digest_bytes(_name_bytes(value)) + + +def _series_seal(series): + """Seal one Series, asserting every one-sided predicate ``_series`` makes. + + The branches, their order and their refusal codes mirror ``_series`` line + for line; only the byte comparisons become digests. + """ + _require(type(series) is pd.Series, "SERIES_DTYPE_OR_LENGTH") + dtype = series.dtype + length = len(series) + head = (type(dtype), dtype, length) + array = series.array + if isinstance(dtype, pd.BooleanDtype) or ( + isinstance(dtype, pd.api.extensions.ExtensionDtype) + and pd.api.types.is_integer_dtype(dtype) + ): + data, mask = getattr(array, "_data", None), getattr(array, "_mask", None) + _require( + type(mask) is np.ndarray + and mask.dtype == np.dtype(np.bool_) + and mask.shape == (length,) + and type(data) is np.ndarray + and not data.dtype.hasobject + and data.shape == mask.shape, + "MASKED_STORAGE", + ) + # ``np.all(rd[rm] == 0)`` for this side. An unmasked column is + # canonical by construction, and the gather is skipped for it. + canonical = not mask.any() or not data[mask].any() + return ( + "masked", + head, + data.dtype, + _seal_digest_bytes(np.ascontiguousarray(mask).tobytes()), + _seal_digest_bytes(np.ascontiguousarray(data[~mask]).tobytes()), + _seal_digest_bytes(np.ascontiguousarray(data).tobytes()), + bool(canonical), + ) + if isinstance(dtype, pd.StringDtype): + nulls = series.isna().to_numpy(dtype=np.bool_, copy=False) + payload = bytearray() + for missing, value in zip(nulls, series, strict=True): + if missing: + continue + _require(type(value) is str, "STRING_VALUE") + encoded = value.encode("utf-8", "surrogatepass") + payload += len(encoded).to_bytes(8, "little") + encoded + return ( + "string", + head, + dtype.storage, + dtype.na_value is pd.NA, + _seal_digest_bytes(np.ascontiguousarray(nulls).tobytes()), + _seal_digest_bytes(bytes(payload)), + ) + if pd.api.types.is_object_dtype(dtype): + payload = bytearray() + for value in series: + encoded = _object_bytes(value) + payload += len(encoded).to_bytes(8, "little") + encoded + return ("object", head, _seal_digest_bytes(bytes(payload))) + _require( + not isinstance(dtype, pd.api.extensions.ExtensionDtype), + "UNSUPPORTED_EXTENSION_DTYPE", + ) + return ("native", head, _array_seal(series.to_numpy(copy=False))) + + +def _same_series_seal(expected, actual): + """Refuse exactly what ``_series`` refuses, with the same codes.""" + _require(expected[0] == actual[0], "SERIES_DTYPE_OR_LENGTH") + left, right = expected[1], actual[1] + _require( + left[0] is right[0] and left[1] == right[1] and left[2] == right[2], + "SERIES_DTYPE_OR_LENGTH", + ) + kind = expected[0] + if kind == "masked": + _require( + expected[2] == actual[2] and expected[3] == actual[3], "MASKED_STORAGE" + ) + _require(expected[4] == actual[4], "PRESENT_BITS") + # Directional, and per series: the ACTUAL side may carry canonically + # zeroed backing beneath its nulls, which is what a store round trip + # produces. One non-canonical column must not force byte equality on + # the others, so this disjunction stays inside the per-series walk. + _require(actual[6] or expected[5] == actual[5], "NONCANONICAL_NULL_BACKING") + return + if kind == "string": + _require(expected[2] == actual[2] and expected[3] == actual[3], "STRING_POLICY") + _require(expected[4] == actual[4], "STRING_MASK") + _require(expected[5] == actual[5], "STRING_VALUE") + return + if kind == "object": + _require(expected[2] == actual[2], "OBJECT_VALUE") + return + _same_array_seal(expected[2], actual[2], "NATIVE_BITS") + + +def _axis_seal(index): + """Seal one axis: ``_axis``'s class, ``identical`` parts, name and values.""" + _require(not isinstance(index, pd.MultiIndex), "AXIS") + # ``Index.identical`` is ``equals`` and every name in ``_comparables`` and + # the exact class and the dtype. ``_comparables`` is ``['name']`` for + # ``Index`` and ``RangeIndex`` and ``['name', 'freq']`` for + # ``DatetimeIndex``, so folding it generically closes a gap that folding + # only (class, dtype, name, values) would leave open. + comparables = tuple(type(index)._comparables) + return ( + type(index), + comparables, + tuple(getattr(index, name, None) for name in comparables), + _name_seal(index.name), + _series_seal(pd.Series(index.array, copy=False)), + ) + + +def _same_axis_seal(expected, actual): + _require( + expected[0] is actual[0] + and expected[1] == actual[1] + and expected[2] == actual[2], + "AXIS", + ) + _require(expected[3] == actual[3], "AXIS_NAME") + _same_series_seal(expected[4], actual[4]) + + +def _flags_seal(table): + """Every documented flag, read off ``type(flags)._keys`` rather than named.""" + flags = table.flags + keys = getattr(type(flags), "_keys", None) + _require( + type(keys) is set and keys and all(type(k) is str for k in keys), + "TABLE_TYPE_OR_FLAGS", + ) + return tuple((key, getattr(flags, key)) for key in sorted(keys)) + + +def replayed_frame_seal(frame: Frame) -> tuple: + """Seal everything ``same_replayed_frame`` compares, in O(columns) space.""" + _require(isinstance(frame, Frame), "FRAME_TYPE") + _require(frame.links == (), "FRAME_CONTEXT") + _require(all(type(r) is MassChangeRecord for r in frame.mass_log), "FRAME_CONTEXT") + entities = [] + for entity in frame.entities: + table = frame.table(entity) + _require(type(table) is pd.DataFrame, "TABLE_TYPE_OR_FLAGS") + entities.append( + ( + entity, + _flags_seal(table), + _axis_seal(table.index), + _axis_seal(table.columns), + tuple((column, _series_seal(table[column])) for column in table), + ) + ) + weights = [] + for entity in frame.weighted_entities: + held = frame.weights_for(entity) + weights.append((entity, held.kind, _array_seal(held.values))) + return ( + SEAL_PROTOCOL, + frame.schema, + tuple(frame.entities), + _seal_digest_bytes(canonical_json(_encode_frame_metadata(frame.metadata))), + _seal_digest_bytes( + canonical_json([asdict(record) for record in frame.mass_log]) + ), + tuple(frame.weighted_entities), + tuple(entities), + _axis_seal(frame.strata.index), + _name_seal(frame.strata.name), + _series_seal(frame.strata), + tuple(weights), + ) + + +def same_replayed_frame_seals(expected: tuple, actual: tuple) -> None: + """Refuse exactly what ``same_replayed_frame`` refuses, with the same codes.""" + _require( + type(expected) is tuple + and type(actual) is tuple + and len(expected) == len(actual) == 11 + and expected[0] == actual[0] == SEAL_PROTOCOL, + "FRAME_SEAL_PROTOCOL", + ) + _require( + expected[1] == actual[1] + and expected[2] == actual[2] + and expected[3] == actual[3] + and expected[4] == actual[4] + and expected[5] == actual[5], + "FRAME_CONTEXT", + ) + _require( + tuple(entity[0] for entity in expected[6]) + == tuple(entity[0] for entity in actual[6]), + "FRAME_CONTEXT", + ) + for left, right in zip(expected[6], actual[6], strict=True): + _require(left[1] == right[1], "TABLE_TYPE_OR_FLAGS") + _same_axis_seal(left[2], right[2]) + _same_axis_seal(left[3], right[3]) + _require( + tuple(column for column, _ in left[4]) + == tuple(column for column, _ in right[4]), + "AXIS", + ) + for (_, one), (_, other) in zip(left[4], right[4], strict=True): + _same_series_seal(one, other) + _same_axis_seal(expected[7], actual[7]) + _require(expected[8] == actual[8], "STRATA_NAME") + _same_series_seal(expected[9], actual[9]) + _require( + tuple(entity for entity, _, _ in expected[10]) + == tuple(entity for entity, _, _ in actual[10]), + "FRAME_CONTEXT", + ) + for (_, kind, values), (_, other_kind, other_values) in zip( + expected[10], actual[10], strict=True + ): + _require(kind is other_kind, "WEIGHT_BYTES") + _same_array_seal(values, other_values, "WEIGHT_BYTES") + + +def replayed_population_seal(population: Population) -> tuple: + """Seal everything ``same_replayed_population`` compares. + + Every one-sided predicate that comparison makes is asserted here, on this + population, with the same code -- including the four it writes against the + ``actual`` operand alone, which sealing both operands makes fire on both. + """ + _require(type(population) is Population, "POPULATION_TYPE") + _require(type(population.version) is str, "POPULATION_CONTEXT") + _require( + all(type(v) is str for v in population.owners.values()), "POPULATION_CONTEXT" + ) + _require( + all(type(v) is WeightKind for v in population.weight_kind.values()), + "POPULATION_CONTEXT", + ) + _require(type(population.mass_ledger) is tuple, "POPULATION_CONTEXT") + _require( + all(type(r) is MassRecord for r in population.mass_ledger), "POPULATION_CONTEXT" + ) + return ( + SEAL_PROTOCOL, + replayed_frame_seal(population.frame), + population.version, + tuple(sorted(population.owners.items())), + tuple(population.weight_kind.items()), + _seal_digest_bytes( + canonical_json([asdict(record) for record in population.mass_ledger]) + ), + tuple(population.design_weights), + tuple( + (name, _array_seal(values)) + for name, values in population.design_weights.items() + ), + ) + + +def same_replayed_population_seals(expected: tuple, actual: tuple) -> None: + """Refuse exactly what ``same_replayed_population`` refuses, same codes.""" + _require( + type(expected) is tuple + and type(actual) is tuple + and len(expected) == len(actual) == 8 + and expected[0] == actual[0] == SEAL_PROTOCOL, + "POPULATION_SEAL_PROTOCOL", + ) + same_replayed_frame_seals(expected[1], actual[1]) + _require( + expected[2] == actual[2] + and expected[3] == actual[3] + and expected[4] == actual[4] + and expected[5] == actual[5] + and expected[6] == actual[6], + "POPULATION_CONTEXT", + ) + for (_, values), (_, other) in zip(expected[7], actual[7], strict=True): + _same_array_seal(values, other, "DESIGN_BYTES") + + +def seal_identity(seal: tuple) -> str: + """A stable hex digest of a seal record, for retaining it under a stamp. + + ``repr`` is the encoding because a seal holds live dtype, class and + ``WeightKind`` objects whose canonical JSON does not exist; every one of + them has an unambiguous ``repr`` and the digest is only ever compared with + another digest taken in the same process. + """ + _require(type(seal) is tuple, "SEAL_TYPE") + return hashlib.sha256(repr(seal).encode("utf-8", "surrogatepass")).hexdigest() From 1b0b915b0c8cefb2faab67377bfe88fd040ce7c3 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 17:53:03 -0400 Subject: [PATCH 05/75] Drive every mutation in the replay battery through both comparison paths The agreement driver caught four real gaps in the seal's first draft: an axis needs a value-equality fold as well as a byte fold, so AXIS and the series codes each keep their own defect; and the exact index class, the columns axis dtype and the index _comparables were folded too coarsely. Three mutations the battery asked for are not constructible through Population's own validation and are written past it the way test_graph_executor writes its own. Co-Authored-By: Claude Opus 5 --- .../us_runtime/survey_population_replay.py | 48 +- .../tests/test_us_survey_population_replay.py | 423 ++++++++++++++++-- 2 files changed, 437 insertions(+), 34 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py b/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py index 8c5f59d56..c18eabe5a 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/survey_population_replay.py @@ -367,6 +367,39 @@ def _same_series_seal(expected, actual): _same_array_seal(expected[2], actual[2], "NATIVE_BITS") +def _equals_class_seal(series): + """A digest whose equality matches ``Index.equals``, not raw bytes. + + ``_axis`` refuses ``AXIS`` when ``identical`` fails and only then falls + through to the byte-exact ``_series`` codes, so an axis needs both a + value-equality fold and a byte fold to keep each refusal on its own code. + ``array_equivalent`` treats two NaNs as equal and ``-0.0`` as ``0.0``, so + the float fold collapses both before hashing; every other admitted dtype's + byte fold already is its value fold, because byte equality implies + ``equals`` for all of them. + """ + dtype = series.dtype + if ( + not isinstance(dtype, pd.api.extensions.ExtensionDtype) + and dtype.kind == "f" + and not dtype.hasobject + ): + values = np.array(series.to_numpy(copy=False), dtype=dtype) + values[np.isnan(values)] = np.nan # one spelling for every payload + values += 0.0 # -0.0 + 0.0 is +0.0 + return ( + "float", + dtype, + _seal_digest_bytes(np.ascontiguousarray(values).tobytes()), + ) + seal = _series_seal(series) + if seal[0] == "masked": + return ("masked", seal[1], seal[3], seal[4]) + if seal[0] == "string": + return ("string", seal[1], seal[4], seal[5]) + return seal + + def _axis_seal(index): """Seal one axis: ``_axis``'s class, ``identical`` parts, name and values.""" _require(not isinstance(index, pd.MultiIndex), "AXIS") @@ -376,12 +409,16 @@ def _axis_seal(index): # ``DatetimeIndex``, so folding it generically closes a gap that folding # only (class, dtype, name, values) would leave open. comparables = tuple(type(index)._comparables) + values = pd.Series(index.array, copy=False) return ( type(index), comparables, tuple(getattr(index, name, None) for name in comparables), + type(index.dtype), + index.dtype, + _equals_class_seal(values), _name_seal(index.name), - _series_seal(pd.Series(index.array, copy=False)), + _series_seal(values), ) @@ -389,11 +426,14 @@ def _same_axis_seal(expected, actual): _require( expected[0] is actual[0] and expected[1] == actual[1] - and expected[2] == actual[2], + and expected[2] == actual[2] + and expected[3] is actual[3] + and expected[4] == actual[4] + and expected[5] == actual[5], "AXIS", ) - _require(expected[3] == actual[3], "AXIS_NAME") - _same_series_seal(expected[4], actual[4]) + _require(expected[6] == actual[6], "AXIS_NAME") + _same_series_seal(expected[7], actual[7]) def _flags_seal(table): diff --git a/packages/microcosm-build/tests/test_us_survey_population_replay.py b/packages/microcosm-build/tests/test_us_survey_population_replay.py index 86d025ddd..6b84a5a7f 100644 --- a/packages/microcosm-build/tests/test_us_survey_population_replay.py +++ b/packages/microcosm-build/tests/test_us_survey_population_replay.py @@ -9,8 +9,13 @@ import pytest from microcosm.build.us_runtime.survey_population_replay import ( + replayed_frame_seal, + replayed_population_seal, same_replayed_frame, + same_replayed_frame_seals, same_replayed_population, + same_replayed_population_seals, + seal_identity, ) from microcosm.frame import EntitySchema, Frame, MassChangeRecord, WeightKind, Weights from microcosm.graph.population import MassRecord, Population @@ -19,6 +24,74 @@ _ERROR = "^SURVEY_POPULATION_REPLAY_" +def _verdict(call, *arguments): + """The refusal code, or ``None`` for acceptance.""" + try: + call(*arguments) + except ValueError as error: + return str(error) + return None + + +def _sealed_verdict(seal, compare, expected, actual): + """The seal path's verdict: a refusal from either constructor counts. + + Every predicate ``same_replayed_*`` makes about one operand alone is + asserted when that operand's seal is built, so the battery below must + treat a construction refusal and a comparison refusal as the same event. + """ + try: + expected_seal, actual_seal = seal(expected), seal(actual) + except ValueError as error: + return str(error) + return _verdict(compare, expected_seal, actual_seal) + + +def _agree_frames(expected, actual): + """Both paths must reach the same verdict, with the same refusal code. + + This is the discrimination battery of ``docs/us-native-retention-seal.md`` + section 5: every mutation below is driven through the object comparison + that exists today and through the content seal that replaces it, and a + mutation the seal misses fails here rather than being quietly dropped. + """ + direct = _verdict(same_replayed_frame, expected, actual) + sealed = _sealed_verdict( + replayed_frame_seal, same_replayed_frame_seals, expected, actual + ) + assert direct == sealed, f"object path {direct!r}, seal path {sealed!r}" + return direct + + +def _agree_populations(expected, actual): + direct = _verdict(same_replayed_population, expected, actual) + sealed = _sealed_verdict( + replayed_population_seal, same_replayed_population_seals, expected, actual + ) + assert direct == sealed, f"object path {direct!r}, seal path {sealed!r}" + return direct + + +def _refuses_frames(expected, actual): + code = _agree_frames(expected, actual) + assert code is not None and code.startswith("SURVEY_POPULATION_REPLAY_"), code + return code + + +def _refuses_populations(expected, actual): + code = _agree_populations(expected, actual) + assert code is not None and code.startswith("SURVEY_POPULATION_REPLAY_"), code + return code + + +def _accepts_frames(expected, actual): + assert _agree_frames(expected, actual) is None + + +def _accepts_populations(expected, actual): + assert _agree_populations(expected, actual) is None + + def _frame(): index = pd.Index([101, 103, 107], dtype="int64", name="source_row") person = pd.DataFrame( @@ -122,13 +195,10 @@ def test_actual_store_roundtrip_accepts_canonical_nulls_and_typed_objects(tmp_pa actual.person["object_value"].iloc[0] is not expected.person["object_value"].iloc[0] ) - assert same_replayed_frame(expected, actual) is None + _accepts_frames(expected, actual) original_population = _population(expected) - assert ( - same_replayed_population( - original_population, replace(original_population, frame=actual) - ) - is None + _accepts_populations( + original_population, replace(original_population, frame=actual) ) # Acceptance is read-only and must not canonicalize the original in place. assert expected.person["nullable_integer"].array._data.tobytes() == before_integer @@ -136,7 +206,7 @@ def test_actual_store_roundtrip_accepts_canonical_nulls_and_typed_objects(tmp_pa def test_unchanged_noncanonical_hidden_storage_is_accepted(): - assert same_replayed_frame(_frame(), _frame()) is None + _accepts_frames(_frame(), _frame()) @pytest.mark.parametrize("column", ["nullable_integer", "nullable_boolean"]) @@ -152,8 +222,7 @@ def test_hidden_storage_exception_is_directional_and_whole_column(column, change else: expected.person[column].array._data[1:] = [False, True] actual.person[column].array._data[1:] = [True, False] - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) @pytest.mark.parametrize("change", ["value", "mask", "dtype", "membership", "text"]) @@ -180,8 +249,7 @@ def test_observed_values_masks_and_dtypes_are_exact(change): "text", pd.array(["changed", "invented\x00label", pd.NA], dtype="string"), ) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) @pytest.mark.parametrize( @@ -222,8 +290,7 @@ def test_axes_schema_and_strata_are_exact(change): else: arguments["schema"] = EntitySchema(group_entities=("tax_unit", "household")) actual = _rebuild(expected, **arguments) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) @pytest.mark.parametrize( @@ -276,8 +343,7 @@ def test_frame_context_and_exact_weight_contract_are_preserved(change): else: kind = WeightKind.CALIBRATED actual = _rebuild(expected, weights={"household": Weights(values, kind)}) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) @pytest.mark.parametrize( @@ -295,8 +361,7 @@ def test_native_signed_zero_and_nan_payload_bits_cannot_change(dtype, bits, slot changed = values.copy() changed[slot] = 0 if slot == 0 else int(changed[slot]) + 1 actual = _with_column(expected, "native_float", changed.view(dtype)) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) @pytest.mark.parametrize( @@ -320,7 +385,7 @@ def test_object_scalar_equivalence_uses_actual_store_types(before, after): "object_value", pd.Series([after, None, None], index=expected.person.index, dtype=object), ) - assert same_replayed_frame(expected, actual) is None + _accepts_frames(expected, actual) @pytest.mark.parametrize( @@ -339,8 +404,7 @@ def test_object_scalar_types_null_sentinels_and_float_bits_cannot_change(before, "object_value", pd.Series([after, None, None], index=frame.person.index, dtype=object), ) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) def test_object_nan_payload_bits_cannot_change(): @@ -358,8 +422,7 @@ def test_object_nan_payload_bits_cannot_change(): "object_value", pd.Series([payloads[1], None, None], index=frame.person.index, dtype=object), ) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) @pytest.mark.parametrize("kind", ["object", "category", "nullable_float"]) @@ -376,8 +439,7 @@ def test_equal_unsupported_store_values_still_refuse(kind, tmp_path): unsupported = _with_column(frame, "object_value", values) with pytest.raises(TypeError): ContentStore(tmp_path / "unsupported-store").put_frame("b" * 64, unsupported) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(unsupported, unsupported) + _refuses_frames(unsupported, unsupported) @pytest.mark.parametrize("axis", ["index", "columns"]) @@ -390,8 +452,7 @@ def test_axis_names_preserve_store_scalar_types(axis, before, after): assert getattr(expected.table("household"), axis).identical( getattr(actual.table("household"), axis) ) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(expected, actual) + _refuses_frames(expected, actual) def test_multiindex_outside_reviewed_store_profile_refuses(): @@ -399,8 +460,7 @@ def test_multiindex_outside_reviewed_store_profile_refuses(): tables = {entity: frame.table(entity).copy() for entity in frame.entities} tables["household"].index = pd.MultiIndex.from_tuples([("a", 0), ("b", 1)]) unsupported = _rebuild(frame, tables=tables) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_frame(unsupported, unsupported) + _refuses_frames(unsupported, unsupported) @pytest.mark.parametrize( @@ -450,5 +510,308 @@ def test_population_provenance_and_original_design_bytes_are_exact(change): actual = replace( expected, frame=frame, weight_kind={"household": WeightKind.CALIBRATED} ) - with pytest.raises(ValueError, match=_ERROR): - same_replayed_population(expected, actual) + _refuses_populations(expected, actual) + + +# --- Discriminations the battery above did not reach --- +# +# Every test below drives one property through ``_refuses_*``/``_accepts_*``, +# so each one proves the object comparison and the content seal agree on it. +# The three marked "seal gap" are discriminations that +# ``survey_population_preparation._frame_identity`` does NOT make, which is why +# the seal is purpose-built rather than reusing it +# (``docs/us-native-retention-seal.md`` section 0). + + +def test_table_flags_are_part_of_the_comparison(): # seal gap + expected, actual = _frame(), _frame() + actual.person.flags.allows_duplicate_labels = False + assert ( + _refuses_frames(expected, actual) + == "SURVEY_POPULATION_REPLAY_TABLE_TYPE_OR_FLAGS" + ) + + +def test_every_documented_flag_is_folded_not_only_the_named_one(): + """The seal reads ``type(flags)._keys`` so a new pandas flag is covered.""" + keys = type(_frame().person.flags)._keys + assert type(keys) is set and keys and all(type(key) is str for key in keys) + sealed = replayed_frame_seal(_frame()) + person = next(entity for entity in sealed[6] if entity[0] == "person") + assert tuple(key for key, _ in person[1]) == tuple(sorted(keys)) + + +def test_the_exact_index_class_is_part_of_the_comparison(): + """A ``RangeIndex`` and an ``Index`` of the same values are not identical.""" + expected, actual = _frame(), _frame() + table = actual.table("household") + assert type(table.index) is pd.RangeIndex + table.index = pd.Index(list(table.index), dtype=table.index.dtype) + assert type(table.index) is pd.Index + assert table.index.equals(expected.table("household").index) + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_AXIS" + + +def test_the_columns_axis_dtype_is_part_of_the_comparison(): + expected, actual = _frame(), _frame() + table = actual.table("household") + table.columns = pd.Index(list(table.columns), dtype=object) + assert table.columns.dtype != expected.table("household").columns.dtype + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_AXIS" + + +def test_an_axis_byte_difference_equals_does_not_see_keeps_the_series_code(): + """``identical`` accepts ``-0.0`` for ``0.0``; the byte walk still refuses.""" + expected, actual = _frame(), _frame() + for frame, first in ((expected, -0.0), (actual, 0.0)): + table = frame.table("household") + table.index = pd.Index([first, 1.0], dtype="float64") + assert expected.table("household").index.identical(actual.table("household").index) + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_NATIVE_BITS" + + +def test_an_axis_nan_payload_difference_keeps_the_series_code(): + payloads = np.array([0x7FF8000000000011, 0x7FF8000000000012], dtype=np.uint64).view( + np.float64 + ) + expected, actual = _frame(), _frame() + for frame, value in ((expected, payloads[0]), (actual, payloads[1])): + frame.table("household").index = pd.Index([value, 1.0], dtype="float64") + assert expected.table("household").index.identical(actual.table("household").index) + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_NATIVE_BITS" + + +def test_index_comparables_beyond_the_name_are_part_of_the_comparison(): + """``DatetimeIndex._comparables`` carries ``freq``; the bytes do not.""" + stamps = ["2020-01-01", "2020-01-02"] + with_freq = pd.DatetimeIndex(stamps, freq="D") + without = pd.DatetimeIndex(stamps) + assert type(with_freq)._comparables == ["name", "freq"] + assert with_freq.to_numpy().tobytes() == without.to_numpy().tobytes() + assert not with_freq.identical(without) + expected, actual = _frame(), _frame() + expected.table("household").index = with_freq + actual.table("household").index = without + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_AXIS" + + +def test_native_nan_payload_bits_are_part_of_the_comparison(): # seal gap + payloads = (0x7FF8000000000011, 0x7FF8000000000012) + frames = [ + _with_column( + _frame(), + "native_float", + np.array( + [bits, 0x3FF4000000000000, 0x8000000000000000], dtype=np.uint64 + ).view(np.float64), + ) + for bits in payloads + ] + assert _refuses_frames(*frames) == "SURVEY_POPULATION_REPLAY_NATIVE_BITS" + + +def test_quiet_and_signalling_nan_are_distinguished(): # seal gap + frames = [ + _with_column( + _frame(), + "native_float", + np.array( + [bits, 0x3FF4000000000000, 0x8000000000000000], dtype=np.uint64 + ).view(np.float64), + ) + for bits in (0x7FF8000000000000, 0x7FF0000000000001) + ] + assert _refuses_frames(*frames) == "SURVEY_POPULATION_REPLAY_NATIVE_BITS" + + +def test_a_native_dtype_change_at_equal_values_still_refuses(): + expected = _with_column(_frame(), "native_int", np.array([1, 2, 3], dtype=np.int64)) + actual = _with_column(expected, "native_int", np.array([1, 2, 3], dtype=np.int32)) + assert ( + _refuses_frames(expected, actual) + == "SURVEY_POPULATION_REPLAY_SERIES_DTYPE_OR_LENGTH" + ) + + +@pytest.mark.parametrize("storage", ["python", "pyarrow"]) +def test_string_storage_and_null_policy_are_part_of_the_comparison(storage): + other = "pyarrow" if storage == "python" else "python" + values = ["a", "b", None] + expected = _with_column( + _frame(), "text", pd.array(values, dtype=pd.StringDtype(storage)) + ) + actual = _with_column( + _frame(), "text", pd.array(values, dtype=pd.StringDtype(other)) + ) + assert ( + _refuses_frames(expected, actual) + == "SURVEY_POPULATION_REPLAY_SERIES_DTYPE_OR_LENGTH" + ) + + +def test_a_declared_link_table_refuses_on_both_paths(): + from microcosm.frame import LinkSpec + + frame = _frame() + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["relations"] = pd.DataFrame( + {"person_id": [1, 2, 3], "household_id": [10, 10, 20]} + ) + linked = _rebuild( + frame, + tables=tables, + schema=EntitySchema( + group_entities=("household", "tax_unit"), + links=(LinkSpec("relations", "person", "household"),), + ), + ) + assert linked.links == ("relations",) + assert _refuses_frames(linked, linked) == "SURVEY_POPULATION_REPLAY_FRAME_CONTEXT" + + +def test_strata_index_name_is_part_of_the_comparison(): + expected, actual = _frame(), _frame() + strata = actual.strata.copy() + strata.index = strata.index.rename("other_strata_axis") + assert _refuses_frames(expected, _rebuild(actual, strata=strata)) == ( + "SURVEY_POPULATION_REPLAY_AXIS" + ) + + +@pytest.mark.parametrize( + "field,value", + [ + ("owners", {("person", "person_id"): 7}), + ("weight_kind", {"household": "importance"}), + ("mass_ledger", [MassRecord("a", "reweight", "declared", 1.0, 1.0, (), ())]), + ("mass_ledger", ({"node_id": "allocation"},)), + ("version", 7), + ], +) +def test_one_sided_population_type_assertions_refuse_on_both_paths(field, value): + """These fire when the seal is built rather than when it is compared. + + ``Population`` validates its own construction, so each defect is written + past that validation the way ``test_graph_executor`` writes its own + (``object.__setattr__`` on the frozen record), which is the only way a + caller could actually present one. + """ + expected = _population(_frame()) + actual = _population(_frame()) + object.__setattr__(actual, field, value) + assert ( + _refuses_populations(expected, actual) + == "SURVEY_POPULATION_REPLAY_POPULATION_CONTEXT" + ) + + +@pytest.mark.parametrize("field", ["weight_kind", "design_weights"]) +def test_mapping_order_is_part_of_the_comparison(field): + frame = _rebuild( + _frame(), + weights={ + "household": Weights(np.array([1.5, 2.5]), WeightKind.IMPORTANCE), + "tax_unit": Weights(np.array([3.5, 4.5]), WeightKind.IMPORTANCE), + }, + ) + expected = Population.from_frame( + frame, + "allocated", + design_weights={ + "household": np.array([5.0, 7.0]), + "tax_unit": np.array([9.0, 11.0]), + }, + ) + reversed_mapping = dict(reversed(list(getattr(expected, field).items()))) + actual = replace(expected, **{field: reversed_mapping}) + assert tuple(getattr(actual, field)) != tuple(getattr(expected, field)) + assert ( + _refuses_populations(expected, actual) + == "SURVEY_POPULATION_REPLAY_POPULATION_CONTEXT" + ) + + +def test_owner_mapping_order_alone_is_accepted_by_both_paths(): + """``owners`` is compared sorted, so its insertion order must not matter.""" + expected = _population(_frame()) + actual = replace(expected, owners=dict(reversed(list(expected.owners.items())))) + assert tuple(actual.owners) != tuple(expected.owners) + _accepts_populations(expected, actual) + + +_DTYPE_CENSUS = ( + "int8", + "int64", + "uint64", + "float32", + "float64", + "bool", + "object", + " Date: Thu, 17 Sep 2026 17:57:01 -0400 Subject: [PATCH 06/75] Let a seal-only observer skip the executor's pickle detachment run_graph(_population_observer_detach=False) passes the live admitted population instead of allocating a detached snapshot per reached node. Opt-in, default unchanged, no US import, and the keyword enters no key, receipt or cache record. Four tests: no snapshot is allocated, the run exports byte- identical store objects and an identical manifest key, the observer really does receive the executor's own object, and the keyword is a bool that does nothing without an observer. Co-Authored-By: Claude Opus 5 --- .../src/microcosm/graph/executor.py | 19 ++- .../tests/test_graph_executor.py | 144 ++++++++++++++++++ 2 files changed, 162 insertions(+), 1 deletion(-) diff --git a/packages/microcosm-graph/src/microcosm/graph/executor.py b/packages/microcosm-graph/src/microcosm/graph/executor.py index ac8a37bba..fb4110eb5 100644 --- a/packages/microcosm-graph/src/microcosm/graph/executor.py +++ b/packages/microcosm-graph/src/microcosm/graph/executor.py @@ -2442,6 +2442,7 @@ def run_graph( resume: ResumePolicy = "auto", decisions: tuple[Decision, ...] = (), _population_observer: Callable[[str, Population], None] | None = None, + _population_observer_detach: bool = True, _verification_epoch: Mapping[str, object] | None = None, ) -> RunManifest: """Execute a compiled graph with content-addressed reuse and receipts. @@ -2454,6 +2455,17 @@ def run_graph( kernel capability, enters no key or receipt, and an unreached node has no population to observe. + The detachment costs one full independent population and a temporary + serialized table buffer per reached node, which is the whole cost for an + observer that only reads. ``_population_observer_detach=False`` is that + observer's declaration that it will neither retain nor mutate what it is + given: no snapshot is allocated and the live admitted population is passed + instead. The declaration is the caller's, not the executor's -- in this + mode the executor no longer enforces that an observer cannot reach + execution state, so a mutating observer in it would corrupt the run. The + default is unchanged and still enforces it. The keyword enters no key, no + receipt and no cache record, and is meaningless without an observer. + The private verification-epoch record is a caller's own counts mapping -- a country runtime that scopes source verification around the whole run hands in the record that scope yields. It is attached to the manifest @@ -2464,6 +2476,8 @@ def run_graph( if resume not in ("auto", "require", "forbid"): raise ValueError("resume must be 'auto', 'require', or 'forbid'.") + if type(_population_observer_detach) is not bool: + raise TypeError("_population_observer_detach must be a bool.") normalized_decisions: list[Decision] = [] for decision in decisions: if isinstance(decision, Decision): @@ -2771,7 +2785,10 @@ def run_graph( populations[node.id] = updated if _population_observer is not None: - _population_observer(node_id, _observer_snapshot(updated)) + _population_observer( + node_id, + _observer_snapshot(updated) if _population_observer_detach else updated, + ) if not hit: manifest_artifacts, record = _write_node( diff --git a/packages/microcosm-graph/tests/test_graph_executor.py b/packages/microcosm-graph/tests/test_graph_executor.py index 1ceddbc72..b222a5817 100644 --- a/packages/microcosm-graph/tests/test_graph_executor.py +++ b/packages/microcosm-graph/tests/test_graph_executor.py @@ -4374,6 +4374,150 @@ def forbidden(population): _run(_graph(), source, ContentStore(tmp_path / "store"), _registry()) +def test_a_seal_only_observer_allocates_no_snapshot( + tmp_path: Path, monkeypatch +) -> None: + """``_population_observer_detach=False`` skips the pickle round trip. + + The detachment is one full independent population plus a temporary + serialized table buffer per reached node. An observer that only reads -- + one that seals each population and keeps the seal -- needs none of it, and + this is the shape of ``test_absent_observer_allocates_no_snapshot`` with an + observer actually present. + """ + + def forbidden(population): + raise AssertionError("snapshot allocated for a seal-only observer") + + monkeypatch.setattr(graph_executor, "_observer_snapshot", forbidden) + source = _source_path(tmp_path / "source") + seen: list[tuple[str, int]] = [] + manifest = run_graph( + compile_graph(_graph()), + sources={"survey": source}, + store=ContentStore(tmp_path / "store"), + kernels=_registry(), + _population_observer=lambda node_id, population: seen.append( + (node_id, population.frame.n("person")) + ), + _population_observer_detach=False, + ) + assert [node_id for node_id, _ in seen] == list(manifest.nodes) + assert {n for _, n in seen} == {3} + + +@pytest.mark.parametrize("warm", (False, True)) +def test_a_seal_only_observer_moves_no_key_receipt_or_store_object( + tmp_path: Path, warm: bool +) -> None: + """A read-only observer in either mode is invisible to what a run exports.""" + source = _source_path(tmp_path / "source") + compiled = compile_graph(_graph(leaf=False)) + plain_store = ContentStore(tmp_path / "plain") + plain = run_graph( + compiled, + sources={"survey": source}, + store=plain_store, + kernels=_registry(), + ) + plain_objects = _object_bytes(plain_store) + store = ContentStore(tmp_path / "sealed") + if warm: + run_graph( + compiled, sources={"survey": source}, store=store, kernels=_registry() + ) + seals: list[tuple[str, tuple]] = [] + + def observe(node_id, population): + seals.append( + ( + node_id, + tuple( + (entity, population.frame.n(entity)) + for entity in population.frame.entities + ), + ) + ) + + sealed = run_graph( + compiled, + sources={"survey": source}, + store=store, + kernels=_registry(), + _population_observer=observe, + _population_observer_detach=False, + ) + assert [node_id for node_id, _ in seals] == list(sealed.nodes) + assert sealed.key == plain.key + assert {name: item.key for name, item in sealed.nodes.items()} == { + name: item.key for name, item in plain.nodes.items() + } + assert _object_bytes(store) == plain_objects + + +def test_a_seal_only_observer_receives_the_live_population(tmp_path: Path) -> None: + """The mode's whole point, and the contract the caller is declaring. + + In the default mode the observer receives an independent copy. Here it + receives the object the executor holds -- which is why the keyword is a + declaration that the observer will not retain or mutate it, and why the + default is unchanged. + """ + source = _source_path(tmp_path / "source") + compiled = compile_graph(_graph(leaf=False)) + detached: list[Population] = [] + run_graph( + compiled, + sources={"survey": source}, + store=ContentStore(tmp_path / "detached"), + kernels=_registry(), + _population_observer=lambda node_id, population: detached.append(population), + ) + live: list[Population] = [] + manifest = run_graph( + compiled, + sources={"survey": source}, + store=ContentStore(tmp_path / "live"), + kernels=_registry(), + _population_observer=lambda node_id, population: live.append(population), + _population_observer_detach=False, + ) + attached = manifest.populations["survey"] + assert any( + seen.frame.table(entity) is attached.table(entity) + for seen in live + for entity in attached.entities + ) + assert not any( + seen.frame.table(entity) is attached.table(entity) + for seen in detached + for entity in attached.entities + ) + + +def test_the_detach_keyword_is_a_bool_and_does_nothing_without_an_observer( + tmp_path: Path, +) -> None: + source = _source_path(tmp_path / "source") + with pytest.raises(TypeError, match="_population_observer_detach must be a bool"): + run_graph( + compile_graph(_graph()), + sources={"survey": source}, + store=ContentStore(tmp_path / "typed"), + kernels=_registry(), + _population_observer_detach=0, + ) + plain = _run(_graph(), source, ContentStore(tmp_path / "plain"), _registry()) + alone = run_graph( + compile_graph(_graph()), + sources={"survey": source}, + store=ContentStore(tmp_path / "alone"), + kernels=_registry(), + _population_observer_detach=False, + ) + assert alone.key == plain.key + + def test_a_gate_reached_only_through_bytes_still_derives_the_tier( tmp_path: Path, ) -> None: From 43fb3927090d4eae3d3e5f9af1fae8fc7bf1a417 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 18:03:18 -0400 Subject: [PATCH 07/75] Repair a stale implementation contract this branch inherited, and guard it survey_population_preparation._spill_roster gained a Path.read_bytes on native-scale-transport at b6081efcb, which added a resource_accesses entry and left graph_implementation_inventory.json's declared resource_accesses_sha256 stale. implementation_manifest therefore REFUSED authenticated_survey_population_v1 -- the stage whose kernels' own implementation_hash calls it -- at every commit from b6081efcb to the branch tip a64f7b733, so no 19-node graph run was possible there. The transport report's section 4 'No committed pin moves' was recomputed before that commit and is stale at its own tip. The repository had no test over the whole inventory: the one caller built three of the ten stages. test_us_implementation_inventory_contracts.py builds all ten and checks all 124 declared contracts; reverting the re-pin turns both arms red, which was verified before committing. Co-Authored-By: Claude Opus 5 --- .../implementation-identity-receipt.json | 137 ++++++++++++++++++ .../implementation_identity_receipt.py | 136 +++++++++++++++++ .../graph_implementation_inventory.json | 2 +- ...t_us_implementation_inventory_contracts.py | 68 +++++++++ 4 files changed, 342 insertions(+), 1 deletion(-) create mode 100644 experiments/native-retention-seal/implementation-identity-receipt.json create mode 100644 experiments/native-retention-seal/implementation_identity_receipt.py create mode 100644 packages/microcosm-build/tests/test_us_implementation_inventory_contracts.py diff --git a/experiments/native-retention-seal/implementation-identity-receipt.json b/experiments/native-retention-seal/implementation-identity-receipt.json new file mode 100644 index 000000000..0ba56bb27 --- /dev/null +++ b/experiments/native-retention-seal/implementation-identity-receipt.json @@ -0,0 +1,137 @@ +{ + "moved_module_digests": { + "base->head": { + "acs_codec_2024": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "acs_housing_universe_2024": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "asec_codec_v4": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "asec_prepared_v3": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "assembly_harmonize": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "assembly_prepare": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "authenticated_survey_population_v1": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "composed_asec_binding_v1": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "composed_population_v1": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + }, + "geography": { + "microcosm.graph/executor.py": { + "base": "46b2424b67c68c983b069cad463c707fab18db6320ee12d506642dbabb38beed", + "head": "feb26666af48c00113775b999066197a936cec73e7a19e921aa25c2e57697740" + } + } + }, + "base->us_only": {} + }, + "release_eligible": false, + "revisions": { + "base": "a64f7b733", + "head": "9bef866c5", + "us_only": "1b0b915b0" + }, + "scope": "diagnostic only; not a build, certification or release artifact", + "stages": [ + "acs_codec_2024", + "acs_housing_universe_2024", + "asec_codec_v4", + "asec_prepared_v3", + "assembly_harmonize", + "assembly_prepare", + "authenticated_survey_population_v1", + "composed_asec_binding_v1", + "composed_population_v1", + "geography" + ], + "working_tree_contract_acceptance": { + "acs_codec_2024": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "7a3339bbd2e236f48655def8c0f267aa7b2d71ec2ad10eb1fc2fa3c9231d3894" + }, + "acs_housing_universe_2024": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "27e8545b52a341dd8895fd0c472351798a42bd6c47e99c09d660addcb120b100" + }, + "asec_codec_v4": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "71fe9fca05e1ad5f381851e5fab5babe919b345c348aa99afad10dcee1d0385f" + }, + "asec_prepared_v3": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "478819f81fe35322ff958258f85ed8d5e3aaa6bbfc7a4f15bd08a811a2b9684a" + }, + "assembly_harmonize": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "69514bd1029e3570cbd31b6261402d507c9518f0c0815097b9fc21b49a5fc947" + }, + "assembly_prepare": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "74a73bb5295a38a2614328c2c38e46cdb3adf3529cf8a758b3495a5987d5dd42" + }, + "authenticated_survey_population_v1": { + "accepted": false, + "error": "ValueError('Unclassified US dependency/resource contract: microcosm.build/us_runtime/survey_population_preparation.py.')" + }, + "composed_asec_binding_v1": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "2b9028a71b9515d396b20273951c900a0a4d0fb8593896f73e439f15821bd187" + }, + "composed_population_v1": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "2caf3c5c46ada06f8662f838942c2439ef28786c524ea23005e44eb33873140f" + }, + "geography": { + "accepted": true, + "inventory_sha256": "58513b5b4c25b75eb6677cf42fbb7dd4e55cc713c52607ad58ebd003d9785093", + "modules_sha256": "0f72e2751b3797d47d704cf17e7d9cb3d0f28d082e4b6c02d53440ba664b8966" + } + } +} diff --git a/experiments/native-retention-seal/implementation_identity_receipt.py b/experiments/native-retention-seal/implementation_identity_receipt.py new file mode 100644 index 000000000..639902748 --- /dev/null +++ b/experiments/native-retention-seal/implementation_identity_receipt.py @@ -0,0 +1,136 @@ +"""What this branch moves in the US implementation identity, and what it does not. + +``graph_implementation.implementation_manifest`` folds ``sha256`` of the WHOLE +FILE of every module in a stage's roster (``graph_implementation.py:431-435``), +and that manifest reaches ``params["implementation"]`` and therefore +``node_key``. So a receipt of which roster files this branch changed is a +receipt of which node keys and store addresses move. + +Run from the worktree root with the workspace interpreter. Reads git objects +and the working tree; writes one JSON file and nothing else. +""" + +import hashlib +import json +import pathlib +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parents[2] +INVENTORY = ( + "packages/microcosm-build/src/microcosm/build/us_runtime/" + "graph_implementation_inventory.json" +) +PACKAGE_SOURCE = { + "microcosm.build": "packages/microcosm-build/src/microcosm/build", + "microcosm.graph": "packages/microcosm-graph/src/microcosm/graph", + "microcosm.frame": "packages/microcosm-frame/src/microcosm/frame", + "microcosm.fit": "packages/microcosm-fit/src/microcosm/fit", + "microcosm.calibrate": "packages/microcosm-calibrate/src/microcosm/calibrate", + "microcosm.data": "packages/microcosm-data/src/microcosm/data", +} +# ``microunit`` is an installed dependency, not a path in this repository, so no +# revision of this repository can move it. It is recorded as out of scope rather +# than silently skipped. +OUTSIDE_THE_REPOSITORY = ("microunit",) + + +def _git(*arguments): + return subprocess.run( + ["git", *arguments], cwd=ROOT, capture_output=True, check=True + ).stdout + + +def _blob(revision, path): + try: + return _git("show", f"{revision}:{path}") + except subprocess.CalledProcessError: + return None + + +def _digests(revision, inventory): + result = {} + for stage, spec in inventory["stages"].items(): + stage_digests = {} + for name in spec["modules"]: + package, relative = name.split("/", 1) + if package in OUTSIDE_THE_REPOSITORY: + stage_digests[name] = "outside-the-repository" + continue + path = f"{PACKAGE_SOURCE[package]}/{relative}" + payload = _blob(revision, path) + stage_digests[name] = ( + None if payload is None else hashlib.sha256(payload).hexdigest() + ) + result[stage] = stage_digests + return result + + +def main(): + revisions = dict(pair.split("=", 1) for pair in sys.argv[1:-1]) + out = pathlib.Path(sys.argv[-1]) + inventory = json.loads((ROOT / INVENTORY).read_bytes()) + per_revision = {label: _digests(rev, inventory) for label, rev in revisions.items()} + + labels = list(revisions) + base = labels[0] + moved = {} + for label in labels[1:]: + changes = {} + for stage, digests in per_revision[label].items(): + differing = { + name: {base: per_revision[base][stage][name], label: digest} + for name, digest in digests.items() + if per_revision[base][stage][name] != digest + } + if differing: + changes[stage] = differing + moved[f"{base}->{label}"] = changes + + # The contract arm: the manifest refuses an unclassified import or resource + # access, so calling it at the working tree proves this branch added none. + contracts = {} + sys.path[:0] = [str(ROOT / p) for p in sorted((ROOT / "packages").glob("*/src"))] + from microcosm.build.us_runtime.graph_implementation import implementation_manifest + + for stage in inventory["stages"]: + try: + manifest = implementation_manifest(stage) + contracts[stage] = { + "accepted": True, + "inventory_sha256": manifest["inventory_sha256"], + "modules_sha256": hashlib.sha256( + json.dumps(manifest["modules"], sort_keys=True).encode() + ).hexdigest(), + } + except Exception as error: # noqa: BLE001 - the refusal is the result + contracts[stage] = {"accepted": False, "error": repr(error)} + + out.write_text( + json.dumps( + { + "scope": "diagnostic only; not a build, certification or release " + "artifact", + "release_eligible": False, + "revisions": revisions, + "stages": sorted(inventory["stages"]), + "moved_module_digests": moved, + "working_tree_contract_acceptance": contracts, + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + for comparison, changes in moved.items(): + if not changes: + print(f"{comparison}: no roster module digest moved") + continue + names = sorted({name for stage in changes.values() for name in stage}) + print(f"{comparison}: {len(changes)} stage(s) move; files: {names}") + refused = [s for s, c in contracts.items() if not c["accepted"]] + print("contracts accepted at the working tree:", not refused, refused or "") + + +if __name__ == "__main__": + main() diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/graph_implementation_inventory.json b/packages/microcosm-build/src/microcosm/build/us_runtime/graph_implementation_inventory.json index cf1c55f03..1959dc535 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/graph_implementation_inventory.json +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/graph_implementation_inventory.json @@ -946,7 +946,7 @@ "numpy", "pandas" ], - "resource_accesses_sha256": "ccfed1c1acff5a1c50b538424dc129930f69c374d9841e233b3fb593a440c3aa", + "resource_accesses_sha256": "0071f934801d15c14e4be66af12ff73daee5f65c89f61d655a23e462afdcd061", "unbound_uses_sha256": "d114117ca19e9b866982147e5497d601ed91ca603d0568730d4051bdd4005910" }, "microcosm.frame/__init__.py": { diff --git a/packages/microcosm-build/tests/test_us_implementation_inventory_contracts.py b/packages/microcosm-build/tests/test_us_implementation_inventory_contracts.py new file mode 100644 index 000000000..60ea19615 --- /dev/null +++ b/packages/microcosm-build/tests/test_us_implementation_inventory_contracts.py @@ -0,0 +1,68 @@ +"""Every declared implementation contract still describes the tree it names. + +``graph_implementation.implementation_manifest`` refuses a stage whose module +carries an import, unbound use or resource access the packaged inventory does +not declare (``graph_implementation.py:432-434``), and a stage kernel's +``implementation_hash`` goes through it -- so a stale contract refuses every +graph run that reaches that stage, not merely a review. + +The repository had no test over the whole inventory: the one existing caller +(``test_us_asec_prepared_resources.py``) builds three of the ten stages. A +``Path.read_bytes`` added to ``survey_population_preparation._spill_roster`` +on ``native-scale-transport`` (commit ``b6081efcb``) therefore left +``authenticated_survey_population_v1`` refusing with no test red, which is the +defect this file exists to make impossible. + +Nothing here reads gated data, runs a graph or needs a country engine: it +hashes the packaged source files the inventory already names. +""" + +from __future__ import annotations + +import pytest + +from microcosm.build.us_runtime import graph_implementation + + +def _inventory(): + roots = graph_implementation._package_roots() + inventory, _ = graph_implementation._inventory(roots) + return roots, inventory + + +def _stages(): + return sorted(_inventory()[1]["stages"]) + + +def _contracts(): + return sorted(_inventory()[1]["contracts"]) + + +@pytest.mark.parametrize("name", _contracts()) +def test_every_declared_contract_matches_the_packaged_module(name): + roots, inventory = _inventory() + package, relative = name.split("/", 1) + if package not in roots: + pytest.skip(f"optional dependency package absent: {package}") + payload = (roots[package] / relative).read_bytes() + actual = graph_implementation._dependency_contract( + payload, name, graph_implementation._covered_imports(name, inventory) + ) + assert actual == inventory["contracts"][name] + + +@pytest.mark.parametrize("stage", _stages()) +def test_every_declared_stage_manifest_builds(stage): + """A stage kernel's ``implementation_hash`` is this call; it must not raise.""" + manifest = graph_implementation.implementation_manifest(stage) + assert manifest["stage"] == stage + assert manifest["schema"] == "microcosm.us.implementation-manifest.v1" + assert manifest["modules"] + + +def test_the_inventory_declares_a_contract_for_every_rostered_module(): + _, inventory = _inventory() + rostered = { + name for spec in inventory["stages"].values() for name in spec["modules"] + } + assert rostered <= set(inventory["contracts"]) From bb50765d62e63fea7b3ee31203946456951ab833 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 18:09:20 -0400 Subject: [PATCH 08/75] Seal each observation on arrival and retain only the declared consumers The base financial runner now names its declared consumers before the run -- financial.ATTACH_NODE, the property graph's attach node, and the tax gate when the rebase is enabled -- detaches those three with the executor's own snapshot function so object identity is exactly what it is today, and seals every other observation on arrival and drops it. Both same_replayed_population call sites become seal comparisons with the same codes. _node_population_stamp takes either arm: a retained Population is stamped as before, a sealed node re-derives seal_identity from its retained record, so FINANCIAL_NODE_POPULATION_CHANGED stays non-vacuous on both. The completion host keeps today's behaviour untouched: it runs its own observer, retains everything, and passes no seal. Co-Authored-By: Claude Opus 5 --- .../graph_atomic_survey_financial.py | 97 +++++++++++++++++-- 1 file changed, 87 insertions(+), 10 deletions(-) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/graph_atomic_survey_financial.py b/packages/microcosm-build/src/microcosm/build/us_runtime/graph_atomic_survey_financial.py index 8c40adba3..fd5c3edee 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/graph_atomic_survey_financial.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/graph_atomic_survey_financial.py @@ -37,6 +37,7 @@ from microcosm.graph.executor import ( _all_node_keys, _apply_result, + _observer_snapshot, _project_context, _source_paths_and_keys, ) @@ -46,6 +47,11 @@ from . import graph_atomic_survey_population as atomic from . import graph_current_survey_predictors as financial +from .survey_population_replay import ( + replayed_population_seal, + same_replayed_population_seals, + seal_identity, +) values = financial.values codec = financial.codec @@ -489,6 +495,17 @@ def require_complete_property_taxes(run): def _node_population_stamp( compiled, node_id, population, completion_boundary=None, *, manifest=False ): + if type(population) is tuple: + # A node the run sealed and dropped rather than retained. The stamp is + # re-derived from the retained seal record exactly as the Population + # arm re-derives it from the retained object, so + # FINANCIAL_NODE_POPULATION_CHANGED keeps a defined, non-vacuous + # meaning on both arms: something the run retained about this node's + # population is not what it was at issuance. + require( + completion_boundary is None and not manifest, "FINANCIAL_SEALED_NODE_SCOPE" + ) + return seal_identity(population) if completion_boundary is None: return reconstruction._population_stamp(population) # Dispatch through the maintained module, never an arbitrary boundary method. @@ -498,7 +515,12 @@ def _node_population_stamp( def _node_population_seals(compiled, node_populations, completion_boundary=None): - """Retain the existing pair format with an exact compiled-order binding.""" + """Retain the existing pair format with an exact compiled-order binding. + + A value is either a ``Population`` the run retained -- stamped as before -- + or the content seal of one it sealed on arrival and dropped. The roster + still binds to ``compiled.order`` exactly, so a gap is still refused. + """ if not node_populations and completion_boundary is None: return () # Preserve the existing optional path for non-completion runs. require( @@ -1637,12 +1659,42 @@ def require_status_current(): ) receipts[financial.PROJECTION_NODE] = qualified.evidence receipts[columns_node.id] = qualified.evidence + # Retention, declared rather than universal. Nineteen detached + # populations between the observation and the replay comparison below + # are ~290 GiB at full source; the comparison itself needs only what + # ``same_replayed_population`` compares, which the seal carries in + # space proportional to columns. What cannot be sealed is object + # IDENTITY: ``result.financial_population is observed[final_node]`` + # further down is an identity check, and the extension host has its + # own. Those consumers are named here, before the run, and only they + # are retained -- detached with the executor's own snapshot function, + # so the object a caller receives is exactly what it is today. + # See docs/us-native-retention-seal.md. + declared_consumers = frozenset( + ( + financial.ATTACH_NODE, + *(() if property_graph is None else (property_graph.ATTACH_NODE,)), + *(() if not rebase_property_taxes else (_tax_module().GATE_NODE,)), + ) + ) observed, observed_stamps = {}, {} + observed_seals, observed_seal_ids = {}, {} def observe(node_id, population): - require(node_id not in observed, "ATOMIC_OBSERVER_DUPLICATE") - observed[node_id] = population - observed_stamps[node_id] = reconstruction._population_stamp(population) + require(node_id not in observed_seals, "ATOMIC_OBSERVER_DUPLICATE") + # The executor hands the live admitted population below; sealing is + # read-only, so it needs no copy of its own. + seal = replayed_population_seal(population) + observed_seals[node_id] = seal + observed_seal_ids[node_id] = seal_identity(seal) + if node_id in declared_consumers: + retained = _observer_snapshot(population) + require( + replayed_population_seal(retained) == seal, + "ATOMIC_OBSERVER_SNAPSHOT_SEAL", + ) + observed[node_id] = retained + observed_stamps[node_id] = reconstruction._population_stamp(retained) manifest = run_graph( compiled, @@ -1651,9 +1703,21 @@ def observe(node_id, population): kernels=kernels, resume=resume, _population_observer=observe, + # This observer seals and drops; it retains only the declared + # consumers, and it detaches those itself, so the executor's own + # per-node pickle round trip is not needed. + _population_observer_detach=False, _verification_epoch=verification, ) - require(tuple(observed) == compiled.order, "ATOMIC_OBSERVER_ROSTER") + require(tuple(observed_seals) == compiled.order, "ATOMIC_OBSERVER_ROSTER") + require( + set(observed) == declared_consumers + and all( + seal_identity(observed_seals[node_id]) == recorded + for node_id, recorded in observed_seal_ids.items() + ), + "ATOMIC_OBSERVER_RETENTION", + ) loaded = _artifacts(manifest, compiled, store, kernels, keys, implementations) require( all(loaded[k] == v for k, v in prefix_artifacts.items()), @@ -1793,7 +1857,9 @@ def observe(node_id, population): else: population = current[version] expected[node_id] = current[version] = population - atomic.same_replayed_population(population, observed[node_id]) + same_replayed_population_seals( + replayed_population_seal(population), observed_seals[node_id] + ) if property_graph is not None: from .graph_property_income_receipts import verify_property_model_receipts @@ -1819,6 +1885,7 @@ def observe(node_id, population): if rebase_property_taxes: final_node = _tax_module().GATE_NODE legacy_population = observed[financial.ATTACH_NODE] + require(final_node in observed, "ATOMIC_OBSERVER_RETENTION") result = AtomicSurveyFinancialRunValues( prefix, observed[final_node], @@ -1941,11 +2008,18 @@ def observe(node_id, population): for node_id, population in expected.items(): require( reconstruction._population_stamp(population) == expected_stamps[node_id] - and reconstruction._population_stamp(observed[node_id]) - == observed_stamps[node_id], + and ( + node_id not in observed + or reconstruction._population_stamp(observed[node_id]) + == observed_stamps[node_id] + ) + and seal_identity(observed_seals[node_id]) + == observed_seal_ids[node_id], "ATOMIC_FINAL_POPULATION_MUTATION", ) - atomic.same_replayed_population(population, observed[node_id]) + same_replayed_population_seals( + replayed_population_seal(population), observed_seals[node_id] + ) for version, population in current.items(): survey._same_frame(population.frame, manifest.population(version)) require( @@ -1971,7 +2045,10 @@ def observe(node_id, population): rebase_property_taxes=rebase_property_taxes, property_population=property_population, person_status_boundary=status_boundary, - node_populations=observed, + node_populations={ + node_id: observed.get(node_id, seal) + for node_id, seal in observed_seals.items() + }, node_states=states, ) return result if return_values else manifest From 7a4d74b264d4a76a7ed884d831285c21e170bddc Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 18:18:10 -0400 Subject: [PATCH 09/75] Battery: the refusals where both operands are equal A probe fleet over the comparison surface found eleven discriminations the first battery missed, all of the same shape: the pair is byte-identical and the comparison refuses anyway, because it is asserting something about one operand. A seal cannot carry those as content, so it must assert them when it is built, and the agreement driver proves it does -- a MassChangeRecord or MassRecord subclass, a Population subclass, a pyarrow or sparse masked carrier, a uint8 mask, an ndarray-subclass backing, a numpy str_ cell, metadata key order, object NaN sign, and non-finite metadata, which the store codec spells in hex rather than refusing. Co-Authored-By: Claude Opus 5 --- .../tests/test_us_survey_population_replay.py | 143 +++++++++++++++++- 1 file changed, 142 insertions(+), 1 deletion(-) diff --git a/packages/microcosm-build/tests/test_us_survey_population_replay.py b/packages/microcosm-build/tests/test_us_survey_population_replay.py index 6b84a5a7f..3a996e25d 100644 --- a/packages/microcosm-build/tests/test_us_survey_population_replay.py +++ b/packages/microcosm-build/tests/test_us_survey_population_replay.py @@ -2,7 +2,7 @@ from __future__ import annotations -from dataclasses import replace +from dataclasses import fields, replace import numpy as np import pandas as pd @@ -24,6 +24,10 @@ _ERROR = "^SURVEY_POPULATION_REPLAY_" +def _fields(record): + return {field.name: getattr(record, field.name) for field in fields(record)} + + def _verdict(call, *arguments): """The refusal code, or ``None`` for acceptance.""" try: @@ -815,3 +819,140 @@ def test_seal_identity_moves_with_the_seal_and_is_stable_without_it(): assert seal_identity(replayed_population_seal(expected)) != seal_identity( replayed_population_seal(actual) ) + + +# --- Refusals where BOTH operands are equal --- +# +# These are the hardest half of the seal's obligation and the easiest to miss: +# the comparison refuses a pair that is byte-identical, because it is asserting +# something about ONE operand. A seal cannot carry them as content, so it must +# assert them when it is built -- which is what ``_sealed_verdict`` above lets +# this battery check. Each case below was found by probing the comparison +# rather than by reading it. + + +def test_a_mass_change_record_subclass_refuses_though_the_values_match(): + class _Subclass(MassChangeRecord): + pass + + frame = _frame() + record = frame.mass_log[0] + substituted = _rebuild(frame, mass_log=(_Subclass(**_fields(record)),)) + assert ( + _refuses_frames(substituted, substituted) + == "SURVEY_POPULATION_REPLAY_FRAME_CONTEXT" + ) + + +def test_a_mass_record_subclass_refuses_though_the_values_match(): + class _Subclass(MassRecord): + pass + + substituted = _population(_frame()) + record = substituted.mass_ledger[0] + object.__setattr__(substituted, "mass_ledger", (_Subclass(**_fields(record)),)) + assert ( + _refuses_populations(substituted, substituted) + == "SURVEY_POPULATION_REPLAY_POPULATION_CONTEXT" + ) + + +def test_a_population_subclass_refuses_though_the_content_matches(): + class _Subclass(Population): + pass + + expected = _population(_frame()) + actual = _Subclass( + expected.frame, + expected.version, + dict(expected.owners), + dict(expected.weight_kind), + mass_ledger=expected.mass_ledger, + design_weights=dict(expected.design_weights), + ) + assert ( + _refuses_populations(expected, actual) + == "SURVEY_POPULATION_REPLAY_POPULATION_TYPE" + ) + + +@pytest.mark.parametrize("carrier", ["pyarrow", "sparse"]) +def test_a_non_pandas_masked_carrier_refuses_though_the_bytes_match(carrier): + """Only pandas' own ``_data``/``_mask`` storage is a reviewed profile.""" + if carrier == "pyarrow": + pytest.importorskip("pyarrow") + values = pd.array([1, 2, 3], dtype="int64[pyarrow]") + else: + values = pd.arrays.SparseArray(np.array([1, 2, 3], dtype=np.int64)) + frame = _frame() + unsupported = _with_column(frame, "object_value", values) + code = _refuses_frames(unsupported, unsupported) + assert code in ( + "SURVEY_POPULATION_REPLAY_MASKED_STORAGE", + "SURVEY_POPULATION_REPLAY_UNSUPPORTED_EXTENSION_DTYPE", + ), code + + +def test_a_degenerate_mask_dtype_refuses_though_both_sides_carry_it(): + frame = _frame() + array = frame.person["nullable_integer"].array + object.__setattr__(array, "_mask", array._mask.astype(np.uint8)) + assert _refuses_frames(frame, frame) == "SURVEY_POPULATION_REPLAY_MASKED_STORAGE" + + +def test_an_ndarray_subclass_backing_refuses_though_both_sides_carry_it(): + class _Subclass(np.ndarray): + pass + + frame = _frame() + array = frame.person["nullable_integer"].array + object.__setattr__(array, "_data", array._data.view(_Subclass)) + assert _refuses_frames(frame, frame) == "SURVEY_POPULATION_REPLAY_MASKED_STORAGE" + + +def test_a_str_subclass_cell_refuses_though_the_characters_match(): + """``np.str_`` arrives for free from a numpy array and is not ``str``.""" + frame = _frame() + values = pd.array(["alpha", "beta", pd.NA], dtype=pd.StringDtype("python")) + substituted = _with_column(frame, "text", values) + backing = substituted.person["text"].array._ndarray + backing[0] = np.array(["alpha"])[0] + assert type(backing[0]) is not str and backing[0] == "alpha" + assert ( + _refuses_frames(substituted, substituted) + == "SURVEY_POPULATION_REPLAY_STRING_VALUE" + ) + + +def test_metadata_key_order_is_part_of_the_comparison(): + expected = _rebuild(_frame(), metadata={"first": 1, "second": 2}) + actual = _rebuild(_frame(), metadata={"second": 2, "first": 1}) + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_FRAME_CONTEXT" + + +def test_object_nan_sign_is_part_of_the_comparison(): + bits = np.array([0x7FF8000000000000, 0xFFF8000000000000], dtype=np.uint64).view( + np.float64 + ) + frame = _frame() + frames = [ + _with_column( + frame, + "object_value", + pd.Series([value, None, None], index=frame.person.index, dtype=object), + ) + for value in bits + ] + assert _refuses_frames(*frames) == "SURVEY_POPULATION_REPLAY_OBJECT_VALUE" + + +def test_non_finite_metadata_keeps_its_sign_on_both_paths(): + """``_encode_frame_metadata`` spells a float64 in hex, infinities included. + + The seal folds the store codec's bytes, not ``graph_context``'s normative + JSON, which is the reason it can carry a value canonical JSON refuses. + """ + expected = _rebuild(_frame(), metadata={"value": float("inf")}) + _accepts_frames(expected, _rebuild(_frame(), metadata={"value": float("inf")})) + actual = _rebuild(_frame(), metadata={"value": float("-inf")}) + assert _refuses_frames(expected, actual) == "SURVEY_POPULATION_REPLAY_FRAME_CONTEXT" From 1e16d6bd2bb6a1a006d605271d59334b70f1eaac Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Thu, 17 Sep 2026 18:29:36 -0400 Subject: [PATCH 10/75] Arm both runs: the 1/1000 after-run and the 1/15, with their gates The 1/1000 launcher runs the transport lane's committed replay harness unedited, byte-identical, at NEED_GB 40. The 1/15 harness is that lane's committed tenth harness with exactly three changes -- the fraction, the RSS ceiling the retention brief names, and the label -- verified by diffing the bodies. Co-Authored-By: Claude Opus 5 --- .../harness19_fifteenth.py | 692 ++++++++++++++++++ .../native-retention-seal/launch_after.sh | 43 ++ .../native-retention-seal/launch_fifteenth.sh | 48 ++ .../native-retention-seal/setsid_exec.py | 17 + 4 files changed, 800 insertions(+) create mode 100644 experiments/native-retention-seal/harness19_fifteenth.py create mode 100755 experiments/native-retention-seal/launch_after.sh create mode 100755 experiments/native-retention-seal/launch_fifteenth.sh create mode 100644 experiments/native-retention-seal/setsid_exec.py diff --git a/experiments/native-retention-seal/harness19_fifteenth.py b/experiments/native-retention-seal/harness19_fifteenth.py new file mode 100644 index 000000000..3647d9947 --- /dev/null +++ b/experiments/native-retention-seal/harness19_fifteenth.py @@ -0,0 +1,692 @@ +"""The retention-seal lane's 1/15 run: above the transport ceiling that was. + +A copy of the native-scale lane's committed ``harness19_tenth.py`` with exactly +three changes, each named here and nowhere else in the file: + +* the sample is 1/15 -- 105,825 of the 1,587,376 source households, where the + single bounded encode refused at 96,860 -- rather than 1/10; +* ``RSS_CEILING`` is the retention brief's 48 GiB rather than 64 GiB; +* the label and the docstring say which lane and which fraction. + +Everything else, including the CPU and wall ceilings, the sampler, the flush +cadence, the receipt capture and every refusal, is byte-for-byte the tenth +harness's. Max chose 1/15 now and 1/10 when the machine is quiet (2026-09-17), +so this file exists beside the tenth harness rather than replacing it. + +Original docstring follows. + +The native-scale lane's 1/10 run: above the transport ceiling that was. + +Identical to the lane's baseline harness except for the sample, the ceilings and +one addition. The sample is 1/10 -- 158,737 of the 1,587,376 source households, +where the single bounded encode refused at 96,860 -- so it runs against its own +source tree, whose ``selection-request.json`` carries ``fraction: [1,10]`` +because the staged pilot request pins 1/1000 and ``_request`` requires the +argument to match it. Every other source file in that tree is an APFS clone of +the staged original and hashes identically; the recovered originals are not +written, linked or moved. + +The addition: every flush captures the roster transport's own on-disk receipt +(``snapshots/preparation/header.json``) and the allocation receipt from the +manifest if one exists yet. The ceiling question is answered by the preparation +and allocation nodes, which run in the first minutes, so a run the CPU ceiling +truncates still carries the evidence for it. + +Original docstring follows. + +Baseline for the native-scale lane: the 19-node graph, uncapped by node. + +This is ``harness19_verify_once.py`` from the verify-once lane with four +additions and no change to what is measured -- same call, same arguments, same +sample, same seed, same sampler, same interval, same refusals: + +* the ceilings are the native-scale brief's (9,000 CPU-s, 48 GiB, 12,000 wall-s) + rather than the verify-once lane's, so no node is cut short; +* a current-resident-size series is sampled every second through ``libproc``'s + ``proc_pidinfo``, because ``ru_maxrss`` is a high-water mark and a retention + slope needs the trace, not the peak; +* the ``run_atomic_survey_financial`` call is timed on its own, so the runner + time outside the node loop is a subtraction between measured quantities + rather than an inference; +* the returned manifest's per-node ``wall_time_s`` is recorded, which is what + the subtraction subtracts. + +Nothing is wrapped: the runtime still refuses instrumented producers, so +attribution is still the stack sampler's. + +Original docstring follows. + +Nineteen-node measurement for the verify-once lane, on this branch only. + +The instrumentation is ``probe_verify_once.py`` unchanged -- the same stack +sampler at the same interval, the same ceilings, the same refusals, the same +output shape. The only difference is the call: this runs the base financial +graph with the exact arguments the accepted v4 cold pilot ran +(``~/PolicyEngine/_recovered/pilot-runs/native19-v2/run/test_native_nineteen_node_financial.py``: +fraction 1/1000, seed 20260908, geography seed 20260908, +``demographic_conditioning=True``, ``n_estimators=2``, ``resume="auto"``), so +its wall, CPU and peak RSS are comparable with that receipt's +5,362.05 s / 5,278.61 s / 8.67 GiB. + +Two differences from that receipt are known and are not this change: it ran on +source snapshot ``2ca11c85a``, and its ``person-income-attachment.h5`` was the +pre-restoration ``5996dcdd...`` rather than the staged ``9ebc0ef2...``. Both +files are exactly 666,333,922 B, so the bytes the source tree hashes are the +same volume either way. + +Original docstring follows. + +Before/after measurement for the verify-once lane: the nine-node prefix. + +A parameterised copy of the v5 pilot probe +(``~/PolicyEngine/_recovered/pilot-runs/native45-v5/probe_repeated_verification.py``). +Everything that decides *what is measured* is verbatim: the same nine-node +population prefix, the same 1/1000 sample, the same seed, the same stack +sampler at the same interval, the same ceilings, the same output shape. The +only changes are the ones a before/after comparison needs: + +* the source tree, the staged run inputs and the output directory come from + the environment, so the identical file measures both trees; +* every imported ``microcosm`` module is asserted to live under that tree, + so a stray editable install cannot silently measure the wrong source; +* a resident-memory ceiling is enforced alongside the CPU ceiling; +* the load average and the interpreter's third-party versions are recorded + with each run, because this machine carries other lanes' work. + +Sources are read by path and never written. Nothing here is a build, a +certification or a release artifact. + + MEASURE_BASE= MEASURE_RUN= \ + MEASURE_PROBE= MEASURE_LABEL=