{_escape(node.id)}
" f'{_escape(node.kernel)} · '
- f"{_escape(transition.entity)} → {_escape(transition.to_kind)}
diff --git a/FABLE-REVIEW.md b/FABLE-REVIEW.md new file mode 100644 index 000000000..bebe6cef2 --- /dev/null +++ b/FABLE-REVIEW.md @@ -0,0 +1,53 @@ +> Historical review of the initial proposal, preserved verbatim. The findings +> were subsequently resolved or adjudicated. See the +> [13 September acceptance record](experiments/uk-shared-graph-contract-acceptance-20260913.md) +> for the tested revision and remaining scope. + + + +**Verdict: REQUEST_CHANGES** for the source proposal. Two findings are actionable design defects in the new contracts; the rest are hardening and test-strength items. I could not save FABLE-REVIEW.md: this session is in plan mode with no write tool exposed, so the full review is below for root to file verbatim. + +## Reviewed identity + +| Item | Value | +| --- | --- | +| HEAD | `6f4ba4ec989eba93786b5d88033631ad253fdc5c` (resolved from `.git/refs/heads/uk-shared-graph-contracts-20260913`) | +| Base | `15ebde806cd1a262363f7217fe535c7234ff757f` (as given; not independently resolved) | +| Method | Read-only. No git diff was available without a shell, so I reviewed the current contents of every file in the receipts' identity table plus both new test files, the B2 edit, the kernel-contract unit test, and the amendment 25/26 text. No imports, pytest, engine, or data. | + +## Ranked findings + +**1. HIGH. `frame_mass_log` leaks non-declared same-version sibling output into kernel inputs with no key binding.** +`_project_context` passes `population.frame.mass_log` at `packages/microcosm-graph/src/microcosm/graph/executor.py:678`. For an ordinary node that population is the cumulative version state, updated after every ordinary node at `executor.py:2569-2570`, and `_append_frame_mass_log` runs for ordinary nodes too at `population.py:1157`. So node X sees records appended by any earlier ordinary node A in the same version, whether or not A is an ancestor. X's key binds only `frame_key(version)` plus declared input owners (`keys.py:187-209`), never A. Result: adding, removing, or re-parameterising A changes X's visible input while X's key is unchanged, so a cache hit replays output computed against a different log. This is a new charter-A hole; before amendment 26 kernels could not see the log at all. The doc sentence "incidental node order is not authority" describes the hazard but nothing enforces it. +Exact fix, minimal: in `run_graph` record `boundary_logs[node.id] = updated.frame.mass_log` when a structural node is admitted, and pass `frame_mass_log=boundary_logs[compiled.versions[node_id]]` for ordinary nodes; structural nodes keep the full incumbent log, which their key already binds through `members` (`keys.py:221-225`). Alternative that keeps sibling visibility: track which node appended each record and reject projection when a record's author is not in `_transitive_ancestors(compiled, node_id)`. Either way `test_mass_log_is_the_incoming_log` at `test_graph_frame_context.py:346-354` must change: the successor is an ordinary node reading the appender's column, and the fix makes it see `()` unless the ancestor rule is used. + +**2. HIGH. A `WeightUpdate` on kind `design` leaves design anchors stale and, after an EXPAND, mixed.** +Anchors are captured once at CREATE (`population.py:333-338`) and only carried afterwards (`population.py:1159`, `_carry_design_weights` at `2125-2181`). `_apply_weight_update` replaces the frame's design-kind values (`population.py:2038`) without re-anchoring. Consequences: the calibrated cap at `population.py:2323-2351` and `realized_max_weight_ratio` at `2354-2375` compare against pre-update design weights, so a normalisation by factor k makes every later cap ratio off by k; an EXPAND after the update anchors entrants from the updated values (`2166-2179`) while retained rows keep original anchors. The #901 consumer, `uk.full.normalize`, is exactly a design-weight normalisation. The decl docstring at `decl.py:381-382` states "ancestry is untouched" as if deliberate, but the mixed-anchor case is not a coherent contract. +Exact fix: in `patch` after line 1159, when `isinstance(node.weights, WeightUpdate)` and the kind is `design`, set `design_weights[entity] = frame.weights_for(entity).values`, and say so in amendment 25. If root prefers the current semantics, refuse `kind="design"` in `WeightUpdate.__post_init__` instead; leaving it silent is the one option I would not accept. + +**3. MEDIUM. `_context_digest` omission is defensible but the fields share live objects.** +Actual behaviour checked: `frame_metadata` is a `MappingProxyType` over a copy whose leaves are `Frame`-frozen tuples, frozensets and `_FrozenMapping` (`bundle.py:1346-1362`); `frame_column_order` values are `tuple[str]`; `frame_mass_log` is the population's own tuple of frozen `MassChangeRecord` objects, shared by reference (`executor.py:678`). Through the public API nothing is writable, so the omission at `executor.py:459-488` does not produce false passes. But `object.__setattr__` on a shared record silently rewrites the live version's log, and unlike `tables` the executor would not notice. This matches the existing treatment of `context.node`, so it is not blocking. Cheap hardening: digest `canonical_json(column_order)`, the mass-log record fields, and `store._encode_frame_metadata(frame_metadata)` inside `_context_digest`. + +**4. MEDIUM. Two new tests are weaker than their names.** +`test_cold_then_required_replay_revalidates_the_axis` (`test_graph_weight_update.py:311-337`) only proves a hit succeeds; it never shows the axis check executes on replay. `test_a_retained_mutating_observer_changes_nothing` (`test_graph_frame_context.py:410-435`) says it rewrites the metadata view but only rewrites tables; the three new fields are untouched. + +**5. LOW. The motivating "re-solve an existing calibration" case is unreachable with the shared kernel.** `calibrate.adam@1` returns no `receipt['weight_update']` and its error text at `packages/microcosm-calibrate/src/microcosm/calibrate/kernels.py:205-209` still demands a `WeightTransition`. Declaring it as a `WeightUpdate` rejects at `population.py:2025-2030`. Not this lane's file, but amendment 25's text should not claim the case is covered. + +**6. LOW. Cosmetic.** `decl.py:568-572` says "weight transition's" for an update mismatch. `test_graph_kernel_contract.py:339-345` was also edited, so the receipts' "isolated to one test file" claim applies to the acceptance suite only; that is acceptable under the charter. + +## Reproducing tests to invent + +- **T1 leak:** probe kernel writes `float(len(context.frame_mass_log))` into `person.x`, reading only `person.age`. Run `{survey, X}` cold: x = 0. Run `{survey, A, X}` into the same store where A appends a record: X hits and reports 0 while a cold run gives 1. Same key, different truth. +- **T2 anchors:** `WeightUpdate("household","design",…)` with factor 2, then a `calibrated` transition returning weights equal to the updated design weights with `max_weight_ratio=1.5`. Currently rejected as ratio 2.0; the receipt reports realized ratio 2.0 against a frame whose own design weights it equals. +- **T3 replay:** monkeypatch `microcosm.graph.population.weight_update_receipt` to return a foreign digest during the `resume="require"` run and assert `NodeRejectedError` matching "different .household. axis". +- **T4 tamper:** a kernel that calls `object.__setattr__(context.frame_mass_log[0], "reason", "x")`; assert the next node sees the original reason. Currently the live log changes and no rejection fires. + +## Residual runtime risks + +- Nothing here was executed. Pandas copy-on-write behaviour in the observer test and the fixture column order at `fixtures/toy_country/person.csv` line 1 (age before income) are the two places a source-only read can be wrong. +- In `resume="require"`, a mismatched axis receipt fails mid-run at apply time rather than in `_preflight_require`, since preflight validates record shape only. +- Column order is bound to the key by construction (owners are ancestors, order follows depth then id, rewrites keep position); I found no leak there. diff --git a/FIX-RESULT.md b/FIX-RESULT.md new file mode 100644 index 000000000..022a0ffdf --- /dev/null +++ b/FIX-RESULT.md @@ -0,0 +1,356 @@ +# Fix round result — UK-enabling shared graph contracts (amendments 25, 26) + +> Historical source-only report. Later source corrections and the passing +> 101-test run are recorded in the +> [13 September acceptance record](experiments/uk-shared-graph-contract-acceptance-20260913.md). + +Source-only. **Runtime is UNTESTED.** No pytest, no import of the +production package, no engine, no country model, no native source, no +install, no gated data, no network beyond `git fetch`, no publication. The +checks that were run are stdlib `ast` parses, `ruff check` / `ruff format +--check`, `tools/ci_test_groups.py --verify`, and reading source. Every +behavioural claim below is a claim about what the source says, not about +an observed run; the bounded runtime plan root needs is at the end. + +## Identity + +| Thing | Value | +| --- | --- | +| Worktree | `_worktrees/microcosm-uk-shared-graph-contracts-20260913` | +| Branch | `uk-shared-graph-contracts-20260913` | +| Base | `origin/main` `15ebde806cd1a262363f7217fe535c7234ff757f` (re-fetched on resume; unchanged, 0 behind) | +| HEAD | `03b5cb8e2` | +| Reviewed head the adjudication ran on | `6f4ba4ec989eba93786b5d88033631ad253fdc5c` | + +### Commits, in order + +| Commit | Kind | What | +| --- | --- | --- | +| `97428cd9f` | journal | Lane baseline: absence proof, the #901 consumer read | +| `ef2dc69c3` | source+tests+docs | Amendment 25: `WeightUpdate`, `weight_update_receipt` | +| `f33d47cda` | source+tests+docs | Amendment 26: the three `KernelContext` frame fields | +| `895aabf19` | acceptance suite | B2's field set, isolated (as amendment 19's was) | +| `6f4ba4ec9` | journal+receipts | Lane receipts and the first runtime plan | +| `6104459f6` | journal | File `FABLE-REVIEW.md`; open the fix round | +| `c0e275543` | source+tests+docs | **F1** — a node sees the mass log its own key binds | +| `6ec46d62c` | source+tests+docs | **F3/F4** — detach the frame view; digest all three fields | +| `09827aec8` | journal | Record the resume and what it re-verified | +| `f5aa65d40` | source+tests+docs | **F2/F4/F5/F6** — design ancestry stated and proven; cached-axis replay properties; the `calibrate.adam` claim withdrawn; `decl.py` re-locked | +| `c1a7920f7` | tests | The clone fixture copies its household's members, as a copied group requires | +| `ecc98107e` | receipts | Fix-round contract deltas; refreshed identity; full test-file disclosure | +| `03b5cb8e2` | journal | Fix-round outcome, the F2 source reading, the runtime plan | + +The two commits the previous process had already landed (`c0e275543`, +`6ec46d62c`) were preserved and re-verified rather than trusted; what that +verification consisted of is in `PROGRESS-uk-shared-graph-contracts.md` +under "Verified on resume, not assumed from the commit messages". + +### File hashes at `03b5cb8e2` + +| File | SHA-256 | +| --- | --- | +| `graph/decl.py` | `11c2abd77f50389c6cd0b51e26cb3ebd5cbd0770ba96e9de1b53c71e9725eaa4` | +| `graph/kernel.py` | `2df6cc5b5b396adae578f885aedbd038d2c0e51f56b4a31b64c2f6b69e1b918d` | +| `graph/executor.py` | `5ab918e495fe4f8dd32c16155fe8c7a911e60e171cdbc8edb790626ce2d58c19` | +| `graph/population.py` | `33d1bb7bacea22870940288bf1907fb9eb24df7c245a216ff802e7fb41f5208f` | +| `graph/weight_update.py` | `0ccfe6fcd257ef62b1f771b12eecc8ac5d447f5aa7d0403102e0ae290de16720` | +| `graph/serialize.py` | `e5bf83c1082154f148626b6a36676614c6ff6c3fe0721aed94a1501da3021b1f` | +| `graph/explain.py` | `734a7b0e31692c31a99528cd83d9e74d3e508a317d913c1c169724a42d69d0de` | +| `graph/__init__.py` | `697c59a37989a36124e6d43c7b07dd3b0582d965f97303c1fb02c88b41db2d48` | +| `docs/graph-interface.lock` | `b857403be2206156b844958cbd6abcb25ef951d05c0cc11e22554169a9343d2e` | + +The lock file records `decl.py` and `kernel.py` at exactly the hashes +above; `test_graph_interface_lock.py` is what enforces that. The lock moved +twice in this lane and both times for a numbered amendment: +`kernel.py` in amendment 26's commits (`f33d47cda`, then the F1 and F3/F4 +docstring edits), `decl.py` in amendment 25's (`ef2dc69c3`, then +`f5aa65d40`'s semantics correction). It was never refreshed to make a test +green. + +### Test files touched, with their roles + +| File | Acceptance suite? | Change | +| --- | --- | --- | +| `test_graph_weight_update.py` | no | new (amendment 25); fix round adds three cached-axis replay properties and renames the weak one | +| `test_graph_frame_context.py` | no | new (amendment 26); fix round adds the boundary-log and isolation properties | +| `test_graph_population.py` | no | fix round adds three design-anchor properties beside the existing ones | +| `test_graph_kernel_contract.py` | no | one assertion of field *adjacency* relaxed to the ordering amendment 19 actually claims | +| `test_acceptance_b_ownership.py` | **yes** | B2's `KernelContext` field set, in its own commit `895aabf19` | + +The lane's earlier "isolated to one test file" claim was true of the +acceptance suite only. That is now stated accurately in the receipts and +historicized in the journal. + +--- + +## Point-by-point response to the adjudication + +### F1 — `frame_mass_log` leaked a same-version sibling's record. **Accepted; fixed.** (`c0e275543`) + +The finding was correct. `_project_context` passed +`population.frame.mass_log`, which for an ordinary node is the version's +cumulative log (`executor.py:2669` rewrites the version entry after every +ordinary member, and `_append_frame_mass_log` runs for ordinary nodes). +An ordinary node's key binds `frame_key(version_key)` and the owners of +the columns it declared (`keys.py:206-209`, `keys.py:181-201`) — never a +sibling — so a cache hit could replay output computed against a different +log. + +The fix is the reviewer's own minimal option. `run_graph` records +`boundary_mass_logs[node.id] = updated.frame.mass_log` where a structural +version is admitted (`executor.py:2674`), beside +`populations[node.id] = updated`, which both cold execution and a restored +hit reach; an ordinary node is projected from +`boundary_mass_logs[compiled.versions[node_id]]` (`executor.py:2511`), a +structural node from `incumbent.frame.mass_log`, whose key binds the base +*and* every ordinary member through `members` (`keys.py:214-225`). The two +maps are written at the same point, so the boundary lookup cannot miss a +version whose incumbent lookup (`executor.py:2413`, which runs first) +succeeded. + +`test_mass_log_is_the_incoming_log` was replaced, as the review said it +had to be. The properties now distinguish the two cases: an unread +same-version appender is invisible to an ordinary member and moves neither +its key nor its stored bytes; cold-with-sibling equals cold-without and +each replays into the other's store under `resume="require"`; the same +appender *is* visible to the structural boundary, whose key it moves. The +ancestor-tracking alternative was not taken: it would make a node's input a +function of graph topology the key does not bind either. + +### F2 — a design-kind `WeightUpdate` should re-anchor. **Not accepted.** Source-backed response. + +The premise that anchors become "stale and mixed" is not what the source +does, and the proposed fix would redefine a declared contract. + +1. **Anchors are captured once, at CREATE.** + `_create_population` calls `Population.from_frame(frame, node.id)` + (`executor.py:1461`), whose `design_weights is None` default reads the + frame's design weights (`population.py:333-338`). That default is + reached exactly once per graph. +2. **Afterwards they are only carried.** Every later version gets + `design_weights=_carry_design_weights(...)` passed **explicitly** + (`population.py:1159`, `population.py:1180`), so the re-derive default + never runs again. +3. **An update does not touch them.** `_apply_weight_update` + (`population.py:2032-2040`) returns a frame with replaced weight values + and nothing else, and it runs *before* the carry in `patch`. +4. **Clones are not "mixed".** The review's EXPAND case says entrants + anchor from updated values "while retained rows keep original anchors". + The code separates three cases, not two: + `_carry_design_weights` maps an EXPAND's **copied** rows back to their + source row's *original* anchor (`population.py:2145-2157`; the cached + twin at `941-981` does the same), retained rows keep their own, and only + a row with **no lineage at all** reads `frame.weights_for(entity)` — + because it has no earlier weight to be anchored on. The assignment's + framing is exactly the code's. +5. **The cap denominator is the original by declaration, not by accident.** + `_assert_design_weight_cap` compares `current > design * cap` against + the carried anchors (`population.py:2333-2350`) and its refusal has + always read "above N * original design weight"; + `realized_max_weight_ratio` uses the same denominator + (`population.py:2354-2375`), under a parameter the node must spell + `weight_anchor='design'`. +6. **Re-anchoring would be the silent change.** It would let a + normalization node inserted anywhere upstream widen every already- + declared `max_weight_ratio` by its own factor — a non-local + redefinition of a contract other nodes wrote against. + +The review's T2 case is real behaviour, and it is now asserted as the +intended outcome rather than left implicit: with design weights doubled by +an update, calibrated weights equal to them are refused at +`max_weight_ratio=1.5` and admitted exactly at `2.0`, reporting a realized +ratio of `2.0`. A stage that wants a cap against normalized weights states +the ratio it means. + +**No mixed-anchor defect beyond that definition was established**, so +nothing was stopped with a counterexample. One consequence is worth naming +and is now named in the amendment and the docstring: a row admitted +*after* an update is anchored on whatever design weight the `EXPAND` +installs for it. If a kernel derives an entrant's design weight from the +updated incumbent values, that derived number becomes the entrant's +anchor — because it is the weight the row entered carrying. That is the +anchor rule applied to a row with no ancestry, not a mixture of two rules. + +The review's alternative ("refuse `kind='design'` in +`WeightUpdate.__post_init__`") was also declined: the lane's recorded read +of #901 shows `uk.full.normalize` declaring +`WeightUpdate("household", weight_kind, ...)` with `weight_kind` a +variable, so refusing the design arm would refuse a consumer this +amendment exists for, to avoid a defect that is not there. + +What changed instead: `WeightUpdate`'s docstring no longer says "ancestry +is untouched" as a bare assertion but states the anchor rule and its three +cases; amendment 25 gains the same paragraph; and +`test_graph_population.py` gains three properties — +`test_a_same_kind_design_update_leaves_the_original_anchors_invariant`, +`test_a_design_update_moves_no_anchor_for_retained_clone_or_entrant_rows` +(the same EXPAND run over an updated and an un-updated population, with +the anchors asserted equal and the frames' design values asserted a factor +apart so it is not vacuous), and +`test_a_design_update_does_not_move_the_calibration_cap_denominator` (T2). + +### F3 — the three fields shared live objects. **Accepted; fixed.** (`6ec46d62c`) + +The review rated this non-blocking; root took it as blocking, because +"nothing writable through the public API" is not the property that matters +once a kernel holds the object: a frozen dataclass still yields to +`object.__setattr__`, and a shared `MassChangeRecord` is a live handle on +the version's log that, unlike a table, nothing would notice. + +`_project_context` now hands out `deepcopy(frame.metadata)` and records +rebuilt through `_detached_record`, the rule `_observer_snapshot` already +followed and now shares. `_context_digest` binds all three fields: the +metadata through `store._encode_frame_metadata` (the frame format's own +codec, so an unchanged view digests as it persists), each mass record +field by field, and the projected column order. The codec call is guarded — +a value `Frame` would never have admitted is reported as the mutation it +is rather than raising while the comparison that would report it is being +computed. + +Detachment and the digest are not redundant: the digest catches a kernel +whose output stops being a function of its declared inputs; detachment is +what stops a *retained* view from rewriting the live version after that +node's check has already passed. + +### F4 — two tests were weaker than their names. **Accepted; fixed.** (`6ec46d62c`, `f5aa65d40`) + +- The observer property now rewrites nested metadata and mass records as + well as tables, over the graph whose snapshots actually carry a mass + record, and asserts the rewrite landed on the snapshot before asserting + the live version and the next node are unchanged. +- `test_cold_then_required_replay_revalidates_the_axis` is renamed + `test_required_replay_reapplies_the_stored_update`, which is what it + proves. Three new properties carry the claim it did not: + `test_required_replay_refuses_a_cached_binding_against_another_axis` + rewrites the stored record's binding to the same household ids in + reverse order — the one case a count check cannot catch — and requires + the replay to hit that record; it is refused with `different 'household' + axis` and no kernel runs, so the refusal came from re-applying the + cached result. A second does the same with a short axis. A third re-files + the record *unchanged* and still replays, so the two refusals are about + the binding rather than about a record having been re-filed. The tests + also pin where the check fires: `resume="require"`'s preflight validates + record shape, so a foreign axis surfaces at apply time, mid-run — which + is the residual the review itself flagged. + +### F5 — the `calibrate.adam` motivating claim. **Accepted; withdrawn.** (`f5aa65d40`) + +Confirmed at source: `packages/microcosm-calibrate/src/` contains no +`weight_update` string at all, so `calibrate.adam@1` emits no +`receipt['weight_update']`, and `_apply_weight_update` would refuse the +declaration as unverifiable (`population.py:2025-2031`). Its own guard +also still asks for a `WeightTransition` by name. Amendment 25, the +`WeightUpdate` docstring and the test module docstring no longer offer +"a re-solve of an existing calibration" as a covered case; the amendment +now states plainly that re-solving through the shared kernel is a **future +consumer adaptation**. + +### F6 — cosmetic and disclosure. **Accepted.** (`f5aa65d40`, `ecc98107e`) + +The node mass-policy mismatch message said "weight transition's" on a path +both declarations reach; it now names neither. The receipts disclose every +test file touched and which one is the acceptance suite's, and the +journal's "isolated to one file" line is historicized in place rather than +edited away. + +--- + +## Bounded runtime plan for root + +Finite and invented-only: the toy country fixture under +`packages/microcosm-graph/tests/fixtures/toy_country/` and `tmp_path`. No +country model, engine, native source, gated microdata, credential or +network. Nothing below publishes, promotes or writes outside `tmp_path` +and the uv environment. + +``` +uv sync --all-packages --locked + +# 1. The new and changed properties. +uv run pytest packages/microcosm-graph/tests/test_graph_weight_update.py \ + packages/microcosm-graph/tests/test_graph_frame_context.py \ + packages/microcosm-graph/tests/test_graph_population.py + +# 2. Everything that reads the two frozen files, the executor, or replay. +uv run pytest packages/microcosm-graph/tests/test_graph_interface_lock.py \ + packages/microcosm-graph/tests/test_acceptance_b_ownership.py \ + packages/microcosm-graph/tests/test_graph_kernel_contract.py \ + packages/microcosm-graph/tests/test_graph_executor.py \ + packages/microcosm-graph/tests/test_graph_explain.py \ + packages/microcosm-graph/tests/test_graph_serialize.py \ + packages/microcosm-graph/tests/test_graph_decl.py \ + packages/microcosm-graph/tests/test_acceptance_replays.py \ + packages/microcosm-graph/tests/test_acceptance_d_weights.py + +# 3. Both shards whole. +uv run pytest packages/microcosm-graph packages/microcosm-frame + +# 4. Lint and the CI partition (both already pass here). +uv run ruff check . +uv run python tools/ci_test_groups.py --verify +``` + +### Helper dependencies each step needs + +| Step | Needs | Why | +| --- | --- | --- | +| 1 | `packages/microcosm-graph/tests/_toy.py`, `fixtures/toy_country/*.csv`, `schema.json` | every graph-level property runs the toy country; `test_graph_frame_context.py` also asserts the **person column order** from `person.csv`'s header, so a fixture column reorder is a genuine (and intended) failure there | +| 1 | `pandas`, `numpy` from the locked env | the anchor properties call `patch()` directly and build pandas lineage objects; no store, no executor | +| 1 | a writable `tmp_path` | the frame-context and weight-update properties build real `ContentStore`s under it | +| 2 | `docs/graph-interface.lock` | `test_graph_interface_lock.py` reads it from the repo, not the wheel | +| 3 | nothing further | | +| 4 | `tools/ci_test_groups.py` | partition authority; `--verify` must stay `verification=ok` | + +Expected: `test_graph_weight_update.py`'s 22 and +`test_graph_frame_context.py`'s 26 properties pass, `test_graph_population.py` +passes all 60 (57 of them pre-existing, 3 added this round), the +regression files stay green, `ruff check` and `--verify` stay clean, and no +node key moves anywhere (neither amendment adds a `Node` field or changes a canonical +projection, so no spec or seed digest should move; if one does, that is +main drift, not this lane). + +### Where a source-only round could still be wrong, worth checking in this order + +1. **`test_a_design_update_moves_no_anchor_for_retained_clone_or_entrant_rows` + builds an EXPAND by hand.** `_patch_expand`'s guards are many; one + already forced a correction in this round (`c1a7920f7` — a copied group + requires the same number of copies of every incumbent member, + `_remapped_expand_memberships`). A second guard firing would surface as + a `PopulationError` naming that guard, not as a wrong anchor. +2. **The three cached-axis properties re-file a store record** through + `ContentStore.put_json(..., verify_existing=False)`. That is the + documented replace path (`store._put` → + `_replace_write_only_collision`), and the record key is derived from the + node key rather than from content, so a rewritten receipt is not a + key/content mismatch. The third property (re-file unchanged, still + replays) is the control that separates "the binding was refused" from + "the re-filed record was unreadable". +3. **`_context_digest`'s metadata arm** calls `store._encode_frame_metadata` + twice per node. Its guard (`TypeError`, `ValueError`, `RecursionError`) + is reasoned from the codec's and `canonical_json`'s source, not + observed. It also costs one encode per node per side; metadata is small, + but that cost is real and unmeasured. +4. **`deepcopy(frame.metadata)` per node projection** is likewise + unmeasured. `Frame` metadata is stage-level, not row-level, so this is + expected to be negligible; it has not been timed. +5. **Pandas copy-on-write in the observer property.** The earlier plan + flagged that a table-only rewrite could make that property vacuous; it + now also rewrites nested metadata and mass records through + `object.__setattr__`, which copy-on-write does not affect, so the + property can no longer be vacuous in the way flagged — but the table arm + of it can still be. +6. **The mass-record probe** in `test_graph_frame_context.py` states an + unchanged household total from `context.weights["household"].values.sum()`; + if `_append_frame_mass_log`'s bracketing disagrees, it surfaces as a + `PopulationError` from the boundary-log properties. + +## What this round does not claim + +No UK build, calibration, native lane, release or publication was run, +prepared or authorized. No country model was imported. `#901` is +untouched: this lane adds no UK graph stage, kernel, target or gate, and +the two amendments remain separable in source — 25 touches `decl.py`, +`serialize.py`, `explain.py`, `population.py`, `graph/__init__.py` and the +new `weight_update.py`, leaving `kernel.py` byte-identical; 26 touches +`kernel.py` and `executor.py`, leaving `decl.py` byte-identical. Either can +be dropped without the other. diff --git a/PROGRESS-uk-shared-graph-contracts.md b/PROGRESS-uk-shared-graph-contracts.md new file mode 100644 index 000000000..028568a9f --- /dev/null +++ b/PROGRESS-uk-shared-graph-contracts.md @@ -0,0 +1,370 @@ +# UK-enabling shared graph contracts (amendments 25 and 26) + +> Historical lane journal. For the final tested revision, see the +> [13 September acceptance record](experiments/uk-shared-graph-contract-acceptance-20260913.md). + +Lane journal. Append-only within this lane; historicize rather than +overwrite once the branch merges (CLAUDE.md, "Root journals are history"). + +## State + +Bounded, source-only slice extracting the shared graph contract that +María's UK full-build graph (#901) consumes, as numbered amendments on +current `main`. No UK graph stage, calibration science or country +kernel is added here — those stay in #901. + +- Worktree: `_worktrees/microcosm-uk-shared-graph-contracts-20260913` +- Branch: `uk-shared-graph-contracts-20260913` +- Base: `origin/main` `15ebde806cd1a262363f7217fe535c7234ff757f` +- Reviewed UK head: #901 `051fb972b19d319d58277bd63306d0d0e0947ce2` + (draft, base `microcosm-us-launch-integration-20260909`, unchanged + since 2026-09-10T21:03:10Z; re-verified via `gh` on 2026-09-13) +- Source review followed: `uk-parallel-review.md` (2026-09-12), section + "Best independent implementation slice" + +**Runtime is UNTESTED in this lane.** Instructions forbid pytest, +production imports, engine, native sources and installation here. Only +stdlib `ast`/`ruff`/CI-inventory source checks were run. A finite +invented-only runtime plan is at the end of this file for root review +*before* execution. + +## Absence verified on base 15ebde806 + +| Contract | Present on main? | Evidence | +| --- | --- | --- | +| `WeightUpdate` (same-kind weight replacement) | **absent** | `grep -rni weightupdate .` over the worktree returns nothing; `decl.py:320` carries only `WeightTransition`, whose `__post_init__` requires `to_kind` strictly later in `WEIGHT_KINDS`, and `population.py:1944` rejects a non-forward move. A same-kind update is therefore unrepresentable. | +| `weight_update_receipt` / ordered-axis evidence | **absent** | no `weight_update` module under `packages/microcosm-graph/src/microcosm/graph/`. | +| `KernelContext.frame_metadata` | **absent** | `kernel.py:361-370` lists the complete field set; no metadata field. | +| `KernelContext.frame_mass_log` | **absent** (read side) | same field list. The *write* side already exists: `population._append_frame_mass_log` (`population.py:2117`) already ingests `receipt['frame_mass_log_append']`, so only the kernel's view of the incoming log is missing. | +| `KernelContext.frame_column_order` | **absent** | same field list. The executor projects `tables[entity]` in declaration order (`executor.py:600-628`), not the population version's own column order, so a consumer cannot reconstruct the frame layout. | + +Frame-side prerequisites already on main: `Frame.mass_log`, +`Frame.metadata`, `MassChangeRecord` and `_freeze_metadata` +(`packages/microcosm-frame/src/microcosm/frame/bundle.py`). + +Interface lock on base matches the files exactly: +`decl.py ed0a859adcae12510d5ba74d51c694617201f7b448b108a3f602410f5da44876`, +`kernel.py dbf57c137330f0f12744c557ee594586a1b308b6d6adaba1938e2b6efded21ca`. + +## Actual consumer read before specifying shape + +`packages/microcosm-build/src/microcosm/build/uk_runtime/graph_population.py` +at `051fb972` (SHA-256 `fc6f5b33127020f6e0529b39304715fe2028d47a6627b152b1e52e0d69f2efdc`): + +- `context_frame` (L58-80) reads `context.frame_column_order.get(entity, ...)`, + `context.weights`, `context.strata`, `getattr(context, "frame_mass_log", ())` + and `getattr(context, "frame_metadata", {})`. +- `UKSampleNormalizationKernel` (L386-412 region) declares + `WeightUpdate("household", weight_kind, "Normalize sampled source-family mass.")` + with `mass="declared"` and places `weight_update_receipt(ids)` under + `receipt["weight_update"]`. + +## Done + +- (nothing yet; baseline recorded) + +## Next + +- Amendment 25: `WeightUpdate` + ordered-axis receipt. +- Amendment 26: `KernelContext` frame metadata / mass log / column order. + +--- + +## Done (2026-09-13) + +| Commit | What | +| --- | --- | +| `97428cd9f` | Lane baseline: absence proof + the exact #901 consumer read | +| `ef2dc69c3` | Amendment 25: `WeightUpdate` + `weight_update_receipt` | +| `f33d47cda` | Amendment 26: `KernelContext` frame metadata / mass log / column order | +| `895aabf19` | Acceptance suite B2 field set, isolated (as amendment 19's was, `a2b6dfb0b`) | + +The two amendments are separable: 25 touches `decl.py` (re-locked) and +leaves `kernel.py` byte-identical; 26 touches `kernel.py` (re-locked) and +leaves `decl.py` byte-identical. Either can be dropped without the other. + +## Contract decisions + +1. **`reason` is normative.** It enters the node key, so two updates that + state different purposes are different nodes. Follows #901's own + declaration; the consequence is stated in the amendment. +2. **`to_kind` stays a property, not a field.** That is what keeps the two + declarations' field sets disjoint (`{entity, to_kind, mass}` vs + `{entity, kind, reason, mass}`), so neither canonical bytes nor + declaration JSON can confuse them — and existing `to_kind` readers (the + design-weight cap, the calibration view) keep working unchanged. +3. **No `free` mass on an update.** #901 declares this too. An update that + neither moves kind nor bounds mass records nothing checkable. +4. **Replay is structural, not a parallel rule.** `_load_cached_result` + already reconstructs the `KernelResult` and re-applies REWEIGHT to the + current base, so a cache hit re-enters `_apply_weight_update`. No + executor change was needed for the axis check. +5. **The frozen interface does not import another shard's private name.** + #901's `kernel.py` imports `microcosm.frame.bundle._freeze_metadata`. + This lane does not: `Frame` has already deeply frozen the metadata the + executor passes, and `KernelContext` adds a read-only view over it. The + docstring says exactly that and claims no deep freeze of its own. +6. **The three fields ride between `artifacts` and `tolerances`**, not at + the end as in #901, so amendment 17's "numerics rides at the end of the + context" stays literally true. Only amendment 19's unit assertion of + *adjacency* relaxes, to the ordering it actually claimed. +7. **A column order may not name an unprojected column.** Set equality + with the projected columns, not a subset: a column *name* is itself + information about the version. + +## Remaining risks + +- **Runtime is UNTESTED here.** No pytest, import, engine or install was + run, per the lane's instructions. Everything below the source level is + unverified; see the runtime plan. +- The acceptance-suite commit `895aabf19` is the one change this lane made + to a file the charter assigns to the suite lane. It is isolated to one + file and follows the precedent the amendment-19 doc text states + explicitly ("the acceptance suite's B2 field set gains it in its own + commit"). If root's owner disagrees, dropping that commit leaves B2 red + and the rest intact. + *(2026-09-13, fix round: "isolated to one file" is true of the* + *acceptance suite only. `test_graph_kernel_contract.py` — not a suite* + *file — was also edited, and the fix round adds three properties to* + *`test_graph_population.py`. The full list is in the receipts.)* +- Amendment 25 changes no node key; amendment 26 changes none either. + Neither re-pins a spec digest. If a spec/seed digest moves in CI, that + is main drift, not this lane (see `[[spec-engine-attested-modules]]`). +- `_context_digest` (B4's mutation check) was **not** extended to the + three new fields. They are immutable views over an immutable `Frame`, so + there is nothing for a kernel to mutate; stated here so the omission is + a decision rather than an oversight. + *(2026-09-13, fix round: this reasoning was wrong and the decision is* + *reversed. A frozen dataclass still yields to `object.__setattr__`, so* + *the fields are now handed out detached and all three are digested —* + *commit `6ec46d62c`. Superseded; kept for the record.)* +- No claim is made that any UK build, native lane, calibration or release + passes. This lane read source only. + +## Next (for root, before execution) + +Finite, invented-only runtime plan — nothing below touches a country +model, engine, native source, gated microdata or the network. + +``` +uv sync --all-packages --locked +uv run pytest packages/microcosm-graph/tests/test_graph_weight_update.py \ + packages/microcosm-graph/tests/test_graph_frame_context.py +uv run pytest packages/microcosm-graph/tests/test_graph_interface_lock.py \ + packages/microcosm-graph/tests/test_acceptance_b_ownership.py \ + packages/microcosm-graph/tests/test_graph_kernel_contract.py \ + packages/microcosm-graph/tests/test_graph_serialize.py \ + packages/microcosm-graph/tests/test_graph_decl.py \ + packages/microcosm-graph/tests/test_acceptance_d_weights.py \ + packages/microcosm-graph/tests/test_graph_population.py \ + packages/microcosm-graph/tests/test_graph_executor.py \ + packages/microcosm-graph/tests/test_graph_explain.py \ + packages/microcosm-graph/tests/test_acceptance_replays.py +uv run pytest packages/microcosm-graph packages/microcosm-frame +uv run ruff check . +uv run python tools/ci_test_groups.py --verify +``` + +Expected: 19 + 19 new tests pass; the ten regression files stay green; +`ruff check` and `--verify` already pass here. The four places a source-only +lane could be wrong, in the order worth checking: + +1. `test_graph_frame_context.py::test_a_retained_mutating_observer_changes_nothing` + mutates a detached snapshot with `table.loc[:, column] = table[column].iloc[0]`. + If pandas copy-on-write makes that a no-op on the snapshot, the test + passes vacuously rather than falsely; tighten it rather than trust it. +2. The toy person column order is asserted from `fixtures/toy_country/person.csv` + (`person_id, person_household_id, person_release_id, age, income, ...`). + If that fixture changes, `test_column_order_is_the_versions_own_order_not_the_projection` + is the test that notices. +3. `_append_frame_mass_log` requires the record to bracket the real + household totals; the probe states an unchanged total from + `context.weights["household"].values.sum()`. A mismatch would surface + as `PopulationError` from `test_mass_log_is_the_incoming_log`. +4. `test_round_trip_refuses_a_mixed_weights_payload` edits canonical JSON + by string surgery and depends on lexicographic key order + (`entity, kind, mass, reason`). + +--- + +# Fix round (2026-09-13, after the independent Fable adjudication) + +## State + +The lane's four commits (`97428cd9f`..`6f4ba4ec9`) stand. An independent +read-only adjudication of exactly `6f4ba4ec9` over base `15ebde806` +returned **REQUEST_CHANGES**; its verbatim text is now filed in this +packet as `FABLE-REVIEW.md` (the reviewer ran without a write tool and +returned the review through its tool result instead). + +Root owns adjudication. Findings F1, F3, F4, F5 and F6 are accepted and +implemented in this round. **F2 is not accepted as stated** — see the +point-by-point response below and in `FIX-RESULT.md`. + +`origin/main` re-fetched before editing: still +`15ebde806cd1a262363f7217fe535c7234ff757f`, nothing new to merge, branch +is 5 commits ahead and 0 behind. + +**Runtime remains UNTESTED in this round too.** No pytest, no production +import, no engine, no install, no network. Source, stdlib `ast`, `ruff` +and the CI group inventory only. + +## Next + +- F1: project ordinary nodes from their version boundary's mass log. +- F3/F4: detach the exposed frame objects; digest all three fields. +- F2: source-backed rejection plus the anchor-invariance tests. +- F5/F6: correct the docs' motivating claim and the disclosure. + +## Fix round, resumed (2026-09-13) + +The first fix-round process died on an external API DNS error after two +commits. Nothing was reset; the checkout resumed clean at `6ec46d62c`. +`origin/main` re-fetched on resume: still `15ebde806`, branch 8 ahead / 0 +behind. Restrictions unchanged — no pytest, import, engine, native source, +install, publication or disallowed log; source, stdlib `ast`, `ruff` and +the CI group inventory only. + +### Verified on resume, not assumed from the commit messages + +- `executor.py` boundary selection: `boundary_mass_logs[node.id]` is + written at exactly one place — beside `populations[node.id] = updated` + in the structural arm of the admission step (`executor.py:2669-2674`), + which both a cold run and a restored hit reach — and read at exactly one + place, the `StructuralDelta.NONE` arm that projects a context + (`executor.py:2508`). Its key set is therefore a subset of + `populations`', so the ordinary lookup cannot miss a version whose + incumbent lookup succeeded. +- Key binding re-read at source: `keys.py:206-209` binds an ordinary + node's `population_input` to `frame_key(version_key)` only, and + `keys.py:214-225` binds a structural node's `base` *and* `members`. The + projection matches that split exactly. +- `store.py:1157-1174`/`1334-1368` round-trip `Frame.mass_log`, so a + restored boundary carries the same records a computed one does. +- Detachment: `Frame.__init__` calls `_freeze_metadata` + (`bundle.py:112`), and `_freeze_metadata_value` rebuilds every nested + mapping, tuple and frozenset (`bundle.py:1382-1405`), so the + `_observer_snapshot` metadata hand-off shares no mutable-by-`setattr` + object with its parent. `_project_context` deep-copies the metadata and + rebuilds every record through `_detached_record`. +- `_context_digest` additions cannot raise while computing the comparison + that reports a mutation: `canonical_json` raises only `TypeError` / + `ValueError` (`canonical.py:_json_value`), `_encode_frame_metadata` + only `TypeError` (`store.py:1060-1082`), and both are caught alongside + `RecursionError`. +- `_project_context`'s only other caller, + `packages/microcosm-build/tests/test_uk_uc_capital_coherence.py:223`, + passes no `mass_log` and so takes the documented empty default. +- `ruff check .` clean; `ruff format --check` clean on every touched file; + stdlib `ast` parses clean. + +### Next + +- F4b: the required-replay axis property is still the weak one the review + named. Strengthen it. +- F2: source-backed rejection plus the anchor-invariance properties. +- F5/F6: the unsupported `calibrate.adam` motivating claim, the + transition-only wording, and an accurate receipts disclosure. + +## Done (fix round) + +| Commit | What | +| --- | --- | +| `6104459f6` | File `FABLE-REVIEW.md` and open the fix round | +| `c0e275543` | F1: a node sees the mass log its own key binds | +| `6ec46d62c` | F3/F4: detach the frame view, and digest it for mutation | +| `09827aec8` | Record the resume and what it re-verified | +| `f5aa65d40` | F2/F4/F5/F6: design ancestry stated and proven; replay axis properties; the `calibrate.adam` claim withdrawn; `decl.py` re-locked | +| `c1a7920f7` | The clone fixture copies its household's members, as a copied group requires | +| `ecc98107e` | Receipts: fix-round deltas and refreshed identity | + +`decl.py` is now `11c2abd77f50389c6cd0b51e26cb3ebd5cbd0770ba96e9de1b53c71e9725eaa4` +and `kernel.py` `2df6cc5b5b396adae578f885aedbd038d2c0e51f56b4a31b64c2f6b69e1b918d`; +`docs/graph-interface.lock` matches both. `kernel.py` moved only in +amendment 26's commits, `decl.py` only in amendment 25's. + +## F2: why the re-anchoring was refused + +Read at source, not argued from the declaration text: + +1. `Population.design_weights` is set from the frame **once**, at CREATE + (`population.py:333-338` via `_create_population`, `executor.py:1461`). +2. Every later version gets its anchors from `_carry_design_weights` + (`population.py:2125-2181`), which aligns the incumbent anchors to the + new version by stable entity id. +3. `patch` passes those carried anchors to `Population.from_frame` + **explicitly** (`population.py:1159-1180`), so the `design_weights is + None` default that would re-derive them from the frame + (`population.py:333-338`) is never reached after CREATE. +4. `_apply_weight_update` (`population.py:2032-2040`) replaces the frame's + weight values. It touches `design_weights` not at all, and it runs + *before* step 2 in `patch`. +5. So a design-kind update moves no existing row's anchor. The EXPAND arm + of `_carry_design_weights` maps copied rows back to their **source + row's original anchor** (`population.py:2145-2157`, and the cached + twin at `941-981`), not to the source's current value, so a clone after + an update inherits the pre-update anchor too. Only a row with no + lineage reads `frame.weights_for(entity)` — and it has no earlier + weight to be anchored on. +6. The cap therefore stays what its own refusal has always called it: + `current > design * cap` against the **original** design weights + (`population.py:2333-2350`), with `realized_max_weight_ratio` reported + on the same denominator (`population.py:2354-2375`). + +The review's T2 case is real behaviour and is the intended outcome, now +asserted as such: calibrated weights equal to design weights an update +doubled are refused at `max_weight_ratio=1.5` and admitted exactly at +`2.0`, realized ratio `2.0`. Re-anchoring would make the same numbers a +ratio of `1.0` and would widen every cap declared upstream of an unrelated +normalization by that normalization's factor — a non-local change to an +already-declared contract, which is what the assignment forbade. + +No mixed-anchor defect beyond that definition was found, so nothing was +stopped. The one consequence worth naming, and now named in the +amendment: a row admitted *after* an update is anchored on whatever design +weight the EXPAND installs for it. If a kernel derives an entrant's design +weight from the updated incumbent values, that derived number is the +entrant's anchor — because it is the weight the row entered carrying. + +## Next (for root, before execution) — fix round + +Superseding the earlier plan's first block; the rest of that plan stands. + +``` +uv sync --all-packages --locked +uv run pytest packages/microcosm-graph/tests/test_graph_weight_update.py \ + packages/microcosm-graph/tests/test_graph_frame_context.py \ + packages/microcosm-graph/tests/test_graph_population.py +uv run pytest packages/microcosm-graph/tests/test_graph_interface_lock.py \ + packages/microcosm-graph/tests/test_acceptance_b_ownership.py \ + packages/microcosm-graph/tests/test_graph_kernel_contract.py \ + packages/microcosm-graph/tests/test_graph_executor.py \ + packages/microcosm-graph/tests/test_acceptance_replays.py \ + packages/microcosm-graph/tests/test_acceptance_d_weights.py +uv run pytest packages/microcosm-graph packages/microcosm-frame +uv run ruff check . +uv run python tools/ci_test_groups.py --verify +``` + +Where a source-only round could still be wrong, in the order worth +checking: + +1. `test_graph_population.py::test_a_design_update_moves_no_anchor_for_retained_clone_or_entrant_rows` + builds an EXPAND by hand. The clone-ordinal rule in + `_remapped_expand_memberships` already forced one correction here + (`c1a7920f7`); if another of `_patch_expand`'s guards fires, it will be + a `PopulationError` naming the guard, not a wrong anchor. +2. The three cached-axis properties in `test_graph_weight_update.py` + re-file a store record through `ContentStore.put_json(..., + verify_existing=False)`. That is the documented replace path, but it is + the one place these tests touch store internals; the third property + (re-file unchanged, still replays) exists to separate "the binding was + refused" from "the re-filed record was unreadable". +3. `_context_digest`'s metadata arm calls `store._encode_frame_metadata` + twice per node. It is guarded against a kernel-planted value the codec + cannot encode, but the guard's `except` list is reasoned from the + codec's source, not observed. +4. The four risks the first plan listed are unchanged, except that the + observer property named there now also rewrites nested metadata and + mass records, so a pandas copy-on-write no-op could no longer make the + whole property vacuous. diff --git a/changelog.d/uk-shared-graph-contracts-frame-context.added.md b/changelog.d/uk-shared-graph-contracts-frame-context.added.md new file mode 100644 index 000000000..b70127ecc --- /dev/null +++ b/changelog.d/uk-shared-graph-contracts-frame-context.added.md @@ -0,0 +1 @@ +Added `KernelContext.frame_metadata`, `frame_mass_log` and `frame_column_order`, so a kernel can reconstruct its population version's declared slices without seeing any column it did not declare. The mass log a node receives is the one its key binds: its version's structural boundary for an ordinary node, the base version's cumulative log for a structural one. All three are detached from the live population before a kernel sees them, and all three enter the executor's input-mutation check (graph amendment 26). diff --git a/changelog.d/uk-shared-graph-contracts-weight-update.added.md b/changelog.d/uk-shared-graph-contracts-weight-update.added.md new file mode 100644 index 000000000..76d148bf4 --- /dev/null +++ b/changelog.d/uk-shared-graph-contracts-weight-update.added.md @@ -0,0 +1 @@ +Added `WeightUpdate`, a declared same-kind replacement of an entity's weight values, with `weight_update_receipt` binding the ordered entity axis the replacement values are positional against, checked against the incumbent axis on cold execution and on every replay. An update replaces weight values only: design anchors, and so any `max_weight_ratio` declared against them, are unchanged (graph amendment 25). diff --git a/docs/graph-acceptance.md b/docs/graph-acceptance.md index c78073571..ef930562d 100644 --- a/docs/graph-acceptance.md +++ b/docs/graph-acceptance.md @@ -512,6 +512,148 @@ lock unchanged: cost before scaling. Extracted with the independently reviewed observer isolation repair on 2026-09-12. +25. **A weight update that keeps its kind is declarable.** + `WeightTransition` only ever moves a kind forward, so a stage that + recomputes weights it already holds — a sampling normalization is the + case this was extracted for — could not be declared at all, and the + only way to express it was to misdeclare a transition. + `WeightUpdate(entity, kind, reason, mass)` is that declaration and is + deliberately narrower than a transition: the incumbent kind, the + declared kind and the returned weights' kind must all be the same one; + `mass` is `conserve` or `declared` (`WEIGHT_UPDATE_MASS_POLICIES`), + because an update that neither changes kind nor bounds mass records + nothing a reader could check it against; and `reason` is required, + non-empty and normative. + + Positional replacement values are not self-describing: the same vector + is correct against one row order and silently wrong against another. + A kernel therefore binds its ordered entity axis with + `microcosm.graph.weight_update.weight_update_receipt` under + `receipt['weight_update']`, and the executor recomputes that binding + from the incumbent axis it is about to apply the values to. This is + checked on replay by construction rather than by a parallel rule: a + cache hit reconstructs the `KernelResult` with its restored weights + and receipt and re-applies the REWEIGHT to the current base, so it + re-enters the same function. A count mismatch, a missing binding and a + binding against a different axis are each a rejection. + + An update replaces weight *values* and does not re-anchor design + ancestry. `Population.design_weights` is captured once, at `CREATE` + (`Population.from_frame`), and afterwards only carried by stable entity + id (`_carry_design_weights`), which `patch` passes on explicitly so the + re-derive-from-the-frame default is never taken. A design-kind update + therefore leaves every existing row's anchor where it was; a later + `EXPAND`'s copied rows still inherit the anchor of the row they copy + rather than that row's current value; and `max_weight_ratio` with + `weight_anchor='design'` keeps the denominator it was written against — + the error text has always said "original design weight". A row admitted + with no ancestor is anchored on whatever design weight the `EXPAND` + installs for it, because it has no earlier weight to be anchored on; + that is the anchor definition applied to a row with no ancestry, not a + mixture. Re-anchoring on a same-kind update would instead let an + unrelated normalization silently widen every cap declared upstream of + it by that normalization's factor, which is a non-local change to an + already-declared contract. A stage that wants a cap against normalized + weights states the ratio it means. + + The shared calibration kernel does **not** consume this yet: + `calibrate.adam@1` emits no `receipt['weight_update']` + (`packages/microcosm-calibrate/src/microcosm/calibrate/kernels.py`), so + declaring a re-solve of an existing calibration as a `WeightUpdate` + would be refused by the axis check as unverifiable, and its own guard + still asks for a `WeightTransition` by name. Re-solving an existing + calibration through the shared kernel is a **future consumer + adaptation**, not a case this amendment already covers. + + `WeightUpdate.to_kind` is a property, not a field, so the two + declarations have disjoint field sets (`{entity, to_kind, mass}` and + `{entity, kind, reason, mass}`) and a `WeightUpdate` can never + canonicalize, or serialize, to the same bytes as a + `WeightTransition`. Declaration JSON discriminates on those names, and + a transition's payload is byte-for-byte what it was before this + amendment, so every declaration written earlier restores unchanged. + Existing `to_kind` readers — the design-weight cap and the calibration + view — keep working through the property; the view drops the arrow + that would claim a kind moved. `Node` gains no field, so no existing + node key moves. `decl.py` is re-locked. Raised by the source review of + the UK full-build graph (#901, head `051fb972`), whose + `uk.full.normalize` node is the first consumer; its UK graph stages + and calibration science stay in that branch. + +26. **The context carries the version's metadata, mass log and column + order.** The executor projects each entity table in *declaration* + order, so `KernelContext.tables` is not the population version's + layout, and the version's `Frame` metadata and mass log were not + reachable from a kernel at all. A kernel that has to hand a declared + projection back to a legacy function as a `Frame` therefore could not + reconstruct one without inventing the parts it could not see. + `kernel.py` gains three read-only fields: + + - `frame_metadata` — the version's own metadata. The executor passes + `Frame.metadata`, which `Frame` has already deeply frozen; + `KernelContext` adds a read-only view over it and does **not** itself + deep-freeze a mapping built some other way. (This is a deliberate + difference from the UK branch, which imports + `microcosm.frame.bundle._freeze_metadata` into the frozen interface: + a frozen contract should not depend on another shard's private name.) + - `frame_mass_log` — the `Frame` mass records the node's *key* binds, + in order. An ordinary node's key binds its version's structural + boundary (`population_input`) and the owners of the columns it + declared (`input_artifacts`); it does not bind the other ordinary + members of its version. Its log is therefore the version's + **boundary** log, captured when the structural node was admitted, and + a record another member appends afterwards is not visible to it. The + alternative — the cumulative log the version carries at the moment the + node runs — would be a kernel input no key binds: adding or + re-parameterising an unrelated sibling would change what the node + sees while its key, and so its cache entry, stayed put, and a hit + would replay output computed against a different log. A structural + node is given its base version's cumulative log instead, because its + key does bind it: `keys.py` binds the base's frame identity *and* + every ordinary member of that version through `members`, which + `compile_graph` fills with `members.get(base, ())`. Boundaries are + captured where the version is admitted, so cold execution and a + restored cache hit record the same one. A node needing a stage's + completed records therefore runs after that stage's structural + boundary or reads its predecessor's evidence; incidental node order is + not authority. The write side already existed + (`receipt['frame_mass_log_append']`); only the read side was missing. + - `frame_column_order` — entity to the version's own column order, + restricted to the columns projected into `tables`. An entry that is + not exactly an ordering of that table's columns is refused, so an + order can neither hide a column the node was given nor name one it + was not: a column *name* is itself information about the version, and + B1's "nothing else is visible" covers names as well as values. + + All three are detached before a kernel sees them and are covered by + B4's before/after mutation check. Detachment is not redundant with + `Frame`'s freezing: a frozen dataclass still yields to + `object.__setattr__`, so passing the version's own `_FrozenMapping` + leaves and `MassChangeRecord`s by reference would make every kernel — and + anything that retains a context past its own mutation check — a live + handle on the population. The executor therefore hands out a deep copy + of the metadata and rebuilt mass records (the rule `_observer_snapshot` + already followed, now shared), and `_context_digest` binds the metadata + through the frame format's own store codec, the mass records field by + field, and the projected column order. Neither is a substitute for the + other: the digest catches a kernel whose output stops being a function + of its declared inputs, while detachment is what stops a retained view + from rewriting the live version after that check has passed. + + The three ride after `artifacts` and before `tolerances`, so amendment + 17's statement that `numerics` rides at the end of the context stays + literally true and amendment 19's that `artifacts` rides before the + pair does too — the unit assertion of *adjacency* becomes the ordering + amendment 19 actually claimed. The acceptance suite's B2 field set + gains the three in its own commit, as amendment 19's did. Nothing here + is normative: `Node` is untouched, no canonical projection changes, and + no node key moves. The fields are rebuilt from a restored `Frame` on a + cache hit exactly as they are from a computed one, which is what + amendment 22's metadata-preserving Frame format makes possible. + `kernel.py` is re-locked. Raised by the same source review as amendment + 25; the UK full-build graph's `context_frame` helper (#901, head + `051fb972`) is the first consumer. + Adding a normative field with a default changes the canonical projection of every node that carries it, so node keys moved with amendments 11 and 13's sibling field `entrants`; no released artifact pins a graph key yet. diff --git a/docs/graph-interface.lock b/docs/graph-interface.lock index b9b01741c..85920c480 100644 --- a/docs/graph-interface.lock +++ b/docs/graph-interface.lock @@ -1,2 +1,2 @@ -ed0a859adcae12510d5ba74d51c694617201f7b448b108a3f602410f5da44876 decl.py -dbf57c137330f0f12744c557ee594586a1b308b6d6adaba1938e2b6efded21ca kernel.py +11c2abd77f50389c6cd0b51e26cb3ebd5cbd0770ba96e9de1b53c71e9725eaa4 decl.py +2df6cc5b5b396adae578f885aedbd038d2c0e51f56b4a31b64c2f6b69e1b918d kernel.py diff --git a/experiments/901-uk-shared-graph-contracts-receipts.md b/experiments/901-uk-shared-graph-contracts-receipts.md new file mode 100644 index 000000000..bc3c11199 --- /dev/null +++ b/experiments/901-uk-shared-graph-contracts-receipts.md @@ -0,0 +1,194 @@ +# Shared graph contracts extracted for the UK full-build graph (#901) + +> Historical source-only receipt. Later fixes and runtime verification are +> recorded in the [13 September acceptance record](uk-shared-graph-contract-acceptance-20260913.md). + +Source-only lane receipt, 2026-09-13. Runtime **UNTESTED** — see +`PROGRESS-uk-shared-graph-contracts.md` for the decisions, risks and the +finite invented-only runtime plan this lane leaves for root. + +## Heads + +| Thing | Exact identity | +| --- | --- | +| Worktree | `_worktrees/microcosm-uk-shared-graph-contracts-20260913` | +| Branch | `uk-shared-graph-contracts-20260913` | +| Base (reviewed main) | `15ebde806cd1a262363f7217fe535c7234ff757f` | +| #901 head (reviewed, re-verified) | `051fb972b19d319d58277bd63306d0d0e0947ce2` | +| #901 live state | OPEN, draft, `CONFLICTING`, updated 2026-09-10T21:03:10Z | +| Source review followed | `uk-parallel-review.md`, 2026-09-12 | + +`origin/main` was re-fetched after the work and is still `15ebde806`. + +## Absence, before anything was written + +`grep -rni weightupdate` over the worktree at `15ebde806` returns nothing. +`decl.py:320` carries only `WeightTransition`, which requires `to_kind` +strictly later in `WEIGHT_KINDS`; `population.py:1944` rejects a +non-forward move. `kernel.py:361-370` is the complete `KernelContext` +field list and has no metadata, mass-log or column-order field. The +interface lock matched both files exactly. So both extensions were +genuinely absent, and no duplicate patch was produced. + +The write half of the mass-log contract already existed on main +(`population._append_frame_mass_log`, `receipt['frame_mass_log_append']`); +only the kernel's view of the incoming log was missing. That is why this +lane adds a read side and no second ledger. + +## Consumer that fixed the shape + +`packages/microcosm-build/src/microcosm/build/uk_runtime/graph_population.py` +at `051fb972`, SHA-256 +`fc6f5b33127020f6e0529b39304715fe2028d47a6627b152b1e52e0d69f2efdc`: + +- `context_frame` (L58-80): `context.frame_column_order.get(...)`, + `getattr(context, "frame_mass_log", ())`, + `getattr(context, "frame_metadata", {})`. +- `uk.full.normalize`: `WeightUpdate("household", weight_kind, "Normalize + sampled source-family mass.")`, `mass="declared"`, and + `receipt["weight_update"] = weight_update_receipt(ids)` where `ids` is + the household-id axis of the context frame. + +Both signatures land exactly as #901 calls them, so no UK edit is needed +to consume this. Nothing was copied from #901's executor, and +`graph.attachments._PopulationRetention` is not imported anywhere. + +## Amendments + +**25 — a same-kind weight update is declarable.** `decl.py` gains +`WeightUpdate(entity, kind, reason, mass)` and +`WEIGHT_UPDATE_MASS_POLICIES`; a new non-frozen +`microcosm/graph/weight_update.py` gains `weight_update_receipt`. The +incumbent, declared and returned kinds must agree; mass is `conserve` or +`declared`; `reason` is required, non-empty and normative. The kernel +binds its ordered entity axis and the executor recomputes that binding +from the incumbent axis, on cold execution and on replay — replay for +free, because `_load_cached_result` already re-applies REWEIGHT to the +current base through the same function. `to_kind` is a property, so the +two declarations' field sets are disjoint and declaration JSON round-trips +each as itself; the transition payload is byte-for-byte unchanged. + +**26 — the context carries the version's frame view.** `kernel.py` gains +`frame_metadata`, `frame_mass_log` and `frame_column_order`, riding after +`artifacts` and before `tolerances` so amendment 17's "numerics rides at +the end" stays literally true. A column order must be exactly an ordering +of the projected columns, so it can neither hide a column the node was +given nor name one it was not. The frozen interface does **not** import +`microcosm.frame.bundle._freeze_metadata` (as #901's does); the executor +passes `Frame.metadata`, already deeply frozen by `Frame`, and the +docstring claims only the read-only view it actually adds. + +Neither amendment adds a `Node` field, changes a canonical projection, or +moves a node key. `decl.py` is re-locked by 25, `kernel.py` by 26. + +## Identity + +Files as of the fix round (`f5aa65d40`). The four earlier rows this table +carried for `decl.py`, `executor.py` and the two new test files were the +pre-fix bytes and are superseded here. + +| File | SHA-256 | +| --- | --- | +| `decl.py` | `11c2abd77f50389c6cd0b51e26cb3ebd5cbd0770ba96e9de1b53c71e9725eaa4` | +| `kernel.py` | `2df6cc5b5b396adae578f885aedbd038d2c0e51f56b4a31b64c2f6b69e1b918d` | +| `weight_update.py` | `0ccfe6fcd257ef62b1f771b12eecc8ac5d447f5aa7d0403102e0ae290de16720` | +| `population.py` | `33d1bb7bacea22870940288bf1907fb9eb24df7c245a216ff802e7fb41f5208f` | +| `serialize.py` | `e5bf83c1082154f148626b6a36676614c6ff6c3fe0721aed94a1501da3021b1f` | +| `executor.py` | `5ab918e495fe4f8dd32c16155fe8c7a911e60e171cdbc8edb790626ce2d58c19` | +| `explain.py` | `734a7b0e31692c31a99528cd83d9e74d3e508a317d913c1c169724a42d69d0de` | +| `graph/__init__.py` | `697c59a37989a36124e6d43c7b07dd3b0582d965f97303c1fb02c88b41db2d48` | +| `tests/test_graph_weight_update.py` | `6ff1296b1d769454b9664ca95bf084b91cf8bd05ada99ed58d54c8d4768751c7` | +| `tests/test_graph_frame_context.py` | `bcee6caccb7ffce47f04785557d246d2e8d5211d760ce5a344f3023cb1d3bf0c` | +| `tests/test_graph_population.py` | `b1aeabac04dbe6af8aeaf3b1691c7306a1442a142279043868500fdfe6476eec` | +| `tests/test_graph_kernel_contract.py` | `b13105cd302702eadcb30278545e794c16f3d819f571693810fb8f838473b126` | +| `tests/test_acceptance_b_ownership.py` | `a78ad49a341d08fe2a76e98e94d6dfe2fb60140ce5589101603b93f933f5bb34` | +| `docs/graph-interface.lock` | `b857403be2206156b844958cbd6abcb25ef951d05c0cc11e22554169a9343d2e` | + +### Every test file this lane touched + +An earlier draft of this receipt said the lane's test change was "isolated +to one file". That is true only of the **acceptance suite**, which the +charter assigns to the suite lane. In full: + +| File | Suite? | Change | +| --- | --- | --- | +| `test_graph_weight_update.py` | no | new (amendment 25) | +| `test_graph_frame_context.py` | no | new (amendment 26) | +| `test_graph_population.py` | no | three design-anchor properties added beside the existing ones (fix round, F2) | +| `test_graph_kernel_contract.py` | no | one assertion of *adjacency* relaxed to the ordering amendment 19 actually claims | +| `test_acceptance_b_ownership.py` | **yes** | B2's `KernelContext` field set, in its own commit (`895aabf19`), as amendment 19's was | + +The lock was re-recorded as part of each numbered amendment, never +refreshed to make a test green: amendment 25 moved only the `decl.py` +line, amendment 26 only the `kernel.py` line. + +## What stays with María + +Every UK graph stage, kernel, calibration target, geography ladder, +scorecard and release gate in #901. This lane adds no country graph, no +UK node, and no second definition of anything #901 owns. + +## Checks actually run + +`ruff check .` (clean), `ruff format --check` on every file this lane +touched (clean), `python -I -B -S` stdlib `ast` parses of all 17 changed +files, and `tools/ci_test_groups.py --verify` (`verification=ok`; both new +test files land in `fast/rest` and the engine lane beside their 27 sibling +graph tests, neither `[defaulted]`). No pytest, no import of the +production package, no engine, no install, no network beyond `gh` +metadata and public blob reads. + +--- + +## Fix round (2026-09-13), after the independent adjudication + +An independent read-only review of `6f4ba4ec9` over `15ebde806` returned +REQUEST_CHANGES; its verbatim text is `FABLE-REVIEW.md` in this packet. +Root adjudicated. What changed in the contracts above: + +**Amendment 26, mass log (F1).** The projection handed every node +`population.frame.mass_log`, which for an ordinary node is its version's +*cumulative* log. An ordinary node's key binds only its version's +structural boundary and the owners of the columns it declared, so a +sibling that ran earlier in the same version and appended a record could +change what the node was shown without moving its key. `run_graph` now +records each version's log as that version is admitted — one line, which +cold execution and a restored hit both reach — and projects ordinary nodes +from that boundary. A structural node still receives the cumulative log, +which its key binds through `base` and `members`. + +**Amendment 26, isolation (F3/F4).** `Frame` deeply freezes its metadata +and its mass records, but a frozen dataclass still yields to +`object.__setattr__`, so passing those objects by reference made every +kernel a live handle on the population. The projection now hands out a +deep copy of the metadata and rebuilt records, through the rule +`_observer_snapshot` already followed (now the shared `_detached_record`), +and `_context_digest` binds all three fields — the metadata through the +frame format's own store codec, each mass record field by field, and the +projected column order. + +**Amendment 25, design ancestry (F2 — not accepted as stated).** The +review asked a design-kind update to re-anchor `Population.design_weights`. +Root refused: an anchor is the design weight a row entered carrying, it is +captured once at CREATE and afterwards only carried by stable entity id, +and `_carry_design_weights` maps an EXPAND's copied rows back to their +source's *original* anchor rather than its current value. Re-anchoring +would silently redefine the denominator of every `max_weight_ratio` +declared upstream of an unrelated normalization. The amendment and the +`WeightUpdate` docstring now state the anchor rule instead of "ancestry is +untouched", and `test_graph_population.py` asserts it. + +**Amendment 25, motivating claim (F5).** The amendment claimed a re-solve +of an existing calibration as a covered case. `calibrate.adam@1` emits no +`receipt['weight_update']`, so that declaration would be refused by the +axis check; the claim is now marked a future consumer adaptation. + +### Checks actually run in the fix round + +`ruff check .` (clean), `ruff format --check` on every touched file +(clean), stdlib `ast` parses of every touched Python file, and +`tools/ci_test_groups.py --verify` (`verification=ok`; the three touched +graph test files appear under `[fast]` and `[engine]`, none under +`[defaulted]`). Still no pytest, no import of the production package, no +engine, no install, no native source and no publication. `origin/main` +re-fetched on resume: `15ebde806`, nothing to merge. diff --git a/experiments/uk-shared-graph-contract-acceptance-20260913.md b/experiments/uk-shared-graph-contract-acceptance-20260913.md new file mode 100644 index 000000000..32fbaf17b --- /dev/null +++ b/experiments/uk-shared-graph-contract-acceptance-20260913.md @@ -0,0 +1,73 @@ +# UK shared graph contract acceptance + +13 September 2026. The shared graph changes at +`5742c17cef27f02db40997b39d7ac7e8b4f8492e` pass 101 selected tests using the +actual executor, population, store and explanation code. These are invented +contract tests; they do not certify an FRS candidate or a UK release. + +## Behavior available to country graphs + +`WeightUpdate` lets a node replace weights while retaining their kind. It +requires an explicit reason, a mass policy and a receipt bound to the ordered +entity IDs. Original design anchors remain unchanged, including for copied +rows, so normalization cannot silently enlarge a later calibration cap. + +`KernelContext` now carries detached frame metadata, mass records and the +original order of its projected columns. Ordinary nodes see the mass log of +their structural boundary; unrelated sibling execution cannot change that +input under the same cache key. The executor checks all three fields for +mutation. Column-order hashing frames the entity and column counts explicitly. + +The explanation's weight-ratio fallback follows the original CREATE anchors +through same-kind updates and filtering. When EXPAND ancestry is unavailable +in the retained manifest, it reports that limitation; explicit receipt samples +remain available. + +These interfaces support the UK full-build graph in [PR #901](https://github.com/PolicyEngine/microcosm/pull/901). +That country graph still needs integration and its own candidate verification. +The shared `calibrate.adam` kernel has not been adapted to consume +`WeightUpdate`; its existing transition contract remains in force. + +## Verification + +| Selected test group | Passed | +| --- | ---: | +| Weight updates | 22 | +| Frame context | 29 | +| Interface lock | 2 | +| Ownership acceptance | 7 | +| Store acceptance | 5 | +| Kernel contract | 18 | +| Population transitions and original design anchors | 15 | +| Explanation and replay | 3 | +| Total | 101 | + +The final run collected exactly 101 cases, with no failures, errors or skips. +It used Python 3.14.4, took 9.96 seconds wall time and 9.63 seconds CPU, and +peaked at 325,844,992 bytes RSS. All 50 frozen source/configuration/fixture +files, seven owned runtime inputs and five toy resources remained unchanged. +It used one numerical thread, created no child process and recorded no +unexpected access refusal. No native survey inputs or country engine were +admitted. + +The source underwent independent review, including Fable review of the +original design-anchor decision and the final test corrections. The final +runtime packet received separate source and execution-result reviews. + +| Evidence | SHA-256 | +| --- | --- | +| Closed acceptance receipt | `527cc5d4846ea8bd3e49b9453f35d1566e9e9337acc14a3f01f62fba7edaa4ce` | +| JUnit result | `6e9f595630b0862927ab660d0b960c12beabf56ae07a661c98492bdf55baac86` | +| Exact runner | `da44f4366968a77386c80c87103f0a232167967c4abf5e2d4b0a96a275e573db` | + +Earlier source-only proposals and unsuccessful runner attempts remain +preserved. One attempt stopped before tests because its authorization strings +differed. Another passed all 101 tests but failed the runner's final checks +because pytest probed unrelated ancestor directories while deriving module +names. The accepted runner uses the frozen source as pytest's root and permits +normal absence for 22 exact source-local metadata probes; existing unmapped +files and unrelated directory scans remain denied. No production or test code +changed between the two 101-test runs. + +The full PR matrix, UK country-graph integration, real FRS build, calibration, +geographic support, holdouts and full/compact exports remain separate gates. diff --git a/packages/microcosm-graph/src/microcosm/graph/__init__.py b/packages/microcosm-graph/src/microcosm/graph/__init__.py index fdc4d6be5..37227aed5 100644 --- a/packages/microcosm-graph/src/microcosm/graph/__init__.py +++ b/packages/microcosm-graph/src/microcosm/graph/__init__.py @@ -16,6 +16,7 @@ PARTITION_DTYPES, ROWS_ALL, WEIGHT_KINDS, + WEIGHT_UPDATE_MASS_POLICIES, ArtifactInput, ArtifactOutput, ArtifactType, @@ -30,6 +31,7 @@ SourceRef, StructuralDelta, WeightTransition, + WeightUpdate, compile_graph, ) from .errors import ( @@ -57,6 +59,7 @@ ) from .keys import platform_fingerprint from .randomness import keyed_uniform +from .weight_update import WEIGHT_UPDATE_AXIS_SCHEMA, weight_update_receipt __all__ = [ "platform_fingerprint", @@ -117,6 +120,9 @@ "StoreUnavailable", "StructuralDelta", "WeightTransition", + "WeightUpdate", + "WEIGHT_UPDATE_AXIS_SCHEMA", + "WEIGHT_UPDATE_MASS_POLICIES", "compile_graph", "describe", "explain_html", @@ -127,6 +133,7 @@ "load_source_bytes", "run_graph", "source_hash", + "weight_update_receipt", ] _FRAME_SERIES = "0.1" diff --git a/packages/microcosm-graph/src/microcosm/graph/decl.py b/packages/microcosm-graph/src/microcosm/graph/decl.py index 16b2ba0e2..a443f7f9b 100644 --- a/packages/microcosm-graph/src/microcosm/graph/decl.py +++ b/packages/microcosm-graph/src/microcosm/graph/decl.py @@ -59,6 +59,7 @@ "PARTITION_DTYPES", "ROWS_ALL", "WEIGHT_KINDS", + "WEIGHT_UPDATE_MASS_POLICIES", "CompiledGraph", "Graph", "GraphError", @@ -70,6 +71,7 @@ "SourceRef", "StructuralDelta", "WeightTransition", + "WeightUpdate", "compile_graph", ] @@ -97,6 +99,10 @@ #: Mass policies a weight transition or structural node may declare. MASS_POLICIES = frozenset({"conserve", "free", "declared"}) +#: Mass policies a same-kind :class:`WeightUpdate` may declare. ``free`` is +#: deliberately absent (amendment 25). +WEIGHT_UPDATE_MASS_POLICIES = frozenset({"conserve", "declared"}) + #: The dtypes a mass-partition column may have. PARTITION_DTYPES = frozenset({"int32", "int64", "string"}) @@ -348,6 +354,86 @@ def __post_init__(self) -> None: ) +@dataclass(frozen=True) +class WeightUpdate: + """A declared numerical replacement of weights that keeps their kind. + + :class:`WeightTransition` only ever moves a weight *kind* forward, so a + stage that recomputes the numbers of weights it already holds — a + sampling normalization is the case this was extracted for — cannot be + declared at all. This is that declaration, and it is deliberately + narrower than a transition (amendment 25): + + - The kind does not move. The executor checks the incumbent kind, the + declared kind and the returned weights' kind are the same one. + - Mass is ``conserve`` or ``declared``; ``free`` is not offered, + because an update that may move mass arbitrarily and does not + change kind records nothing a reader could check it against. + - ``reason`` is required, non-empty, and normative: it enters the node + key, so a node that replaces weights for a different stated purpose + is a different node. + - The kernel must bind the ordered entity axis its replacement values + are positional against, through + :func:`~microcosm.graph.weight_update.weight_update_receipt`. The + executor recomputes that binding from the incumbent axis, on cold + execution and on replay. + + An update replaces weight *values*; it does not re-anchor design + ancestry, and it is not a way to redefine a calibration cap. A row's + design anchor is the design weight it entered the population carrying: + ``Population.design_weights`` is captured once, at ``CREATE`` + (``population.Population.from_frame``), and afterwards only carried by + stable entity id (``population._carry_design_weights``), which ``patch`` + passes on explicitly so the re-derive-from-the-frame default is never + taken. So a design-kind update leaves every existing row's anchor + exactly where it was, a later ``EXPAND``'s copied rows still inherit + the anchor of the row they copy rather than that row's current value, + and ``max_weight_ratio`` with ``weight_anchor='design'`` keeps the + denominator it was written against. A row admitted with no ancestor is + anchored on the design weight the ``EXPAND`` installs for it, whatever + that is — it has no earlier weight to be anchored on. Re-anchoring here + instead would let an unrelated normalization silently widen every cap + declared upstream of it by that normalization's factor; a stage that + wants a cap against normalized weights states the ratio it means. + + Attributes: + entity: The entity whose explicit weights are replaced. + kind: The unchanged weight kind; one of :data:`WEIGHT_KINDS`. + reason: Why the numbers are replaced. Normative, non-empty. + mass: ``conserve`` or ``declared``. + """ + + entity: str + kind: str + reason: str + mass: str = "declared" + + def __post_init__(self) -> None: + _name("WeightUpdate.entity", self.entity) + _nonempty("WeightUpdate.reason", self.reason) + if self.kind not in WEIGHT_KINDS: + raise GraphError( + f"WeightUpdate.kind {self.kind!r} is not one of {WEIGHT_KINDS}." + ) + if self.mass not in WEIGHT_UPDATE_MASS_POLICIES: + raise GraphError( + f"WeightUpdate.mass {self.mass!r} is not one of " + f"{sorted(WEIGHT_UPDATE_MASS_POLICIES)}; an update that keeps " + "its kind does not get unconstrained free mass." + ) + + @property + def to_kind(self) -> str: + """The unchanged kind, for shared structural-weight accounting. + + A property, not a field, so it stays out of the normative + projection and a :class:`WeightUpdate` can never canonicalize to + the same bytes as a :class:`WeightTransition`. + """ + + return self.kind + + @dataclass(frozen=True) class Node: """One unit of computation and cell ownership. @@ -376,7 +462,9 @@ class Node: base: For a structural node other than ``CREATE``: the population version it transforms. sources: Names of :class:`SourceRef` entries this node reads. - weights: A declared weight-kind transition, if any. + weights: A declared :class:`WeightTransition` (the kind moves + forward) or :class:`WeightUpdate` (the numbers are replaced + and the kind does not move), if any. mass: Mass policy for structural nodes that change rows or weights. entrants: ``EXPAND`` nodes only: the kernel may add rows that copy no base row. Such a row has null lineage, the kernel supplies @@ -396,7 +484,7 @@ class Node: structural: StructuralDelta = StructuralDelta.NONE base: str | None = None sources: tuple[str, ...] = () - weights: WeightTransition | None = None + weights: WeightTransition | WeightUpdate | None = None mass: str = "conserve" description: str = "" citation: str = "" @@ -474,6 +562,13 @@ def __post_init__(self) -> None: f"Node {self.id!r}: entrants add mass, so an entrant-admitting " "node cannot declare mass='conserve'." ) + if self.weights is not None and not isinstance( + self.weights, WeightTransition | WeightUpdate + ): + raise GraphError( + f"Node {self.id!r}: weights must be a WeightTransition or a " + "WeightUpdate, not a look-alike." + ) if self.weights is not None and self.structural is not StructuralDelta.REWEIGHT: raise GraphError( f"Node {self.id!r}: a weight transition changes the population " @@ -482,12 +577,15 @@ def __post_init__(self) -> None: if self.structural is StructuralDelta.REWEIGHT: if self.weights is None: raise GraphError( - f"Node {self.id!r}: a REWEIGHT node declares its WeightTransition." + f"Node {self.id!r}: a REWEIGHT node declares its " + "WeightTransition or WeightUpdate." ) if self.mass != self.weights.mass: + # Reached by a WeightTransition and a WeightUpdate alike, so + # the text names neither (amendment 25). raise GraphError( - f"Node {self.id!r}: mass policy {self.mass!r} disagrees with its " - f"weight transition's {self.weights.mass!r}." + f"Node {self.id!r}: mass policy {self.mass!r} disagrees with " + f"its declared weight change's {self.weights.mass!r}." ) declared_inputs = {(s.entity, c) for s in self.inputs for c in s.columns} for s in self.inputs: diff --git a/packages/microcosm-graph/src/microcosm/graph/executor.py b/packages/microcosm-graph/src/microcosm/graph/executor.py index b61a678d4..33116b689 100644 --- a/packages/microcosm-graph/src/microcosm/graph/executor.py +++ b/packages/microcosm-graph/src/microcosm/graph/executor.py @@ -17,7 +17,7 @@ import numpy as np import pandas as pd -from microcosm.frame import Frame, WeightKind, Weights +from microcosm.frame import Frame, MassChangeRecord, WeightKind, Weights from . import keys as graph_keys from .artifact_edges import scope_payload, typed_contracts, value_from_descriptor @@ -78,6 +78,7 @@ StoreCorrupt, StoreMiss, StoreUnavailable, + _encode_frame_metadata, ) __all__ = ["NodeRejected", "NodeRejectedError", "run_graph"] @@ -351,6 +352,25 @@ def _freeze_frame(table: pd.DataFrame) -> pd.DataFrame: return frozen +def _detached_record(record): + """Rebuild one frozen record with independently copied field values. + + Reconstructed through the dataclass constructor rather than by a + reflective copy of its namespace, because source identities may seal + these classes. A frozen dataclass is not immune to ``object.__setattr__``, + so a record handed out by reference is a live handle on whatever holds + it; every hand-out goes through here. + """ + + return replace( + record, + **{ + field.name: deepcopy(getattr(record, field.name)) + for field in fields(record) + }, + ) + + def _observer_snapshot(population: Population) -> Population: """Detach every observation from the executable population and its cache. @@ -369,22 +389,12 @@ def _observer_snapshot(population: Population) -> Population: tables = {name: frame.table(name) for name in frame.entities} tables.update({name: frame.link(name) for name in frame.links}) tables, strata = pickle.loads(pickle.dumps((tables, frame.strata), protocol=5)) - - def copied_record(record): - return replace( - record, - **{ - field.name: deepcopy(getattr(record, field.name)) - for field in fields(record) - }, - ) - snapshot = Frame( tables, replace( frame.schema, group_entities=deepcopy(frame.schema.group_entities), - links=tuple(copied_record(link) for link in frame.schema.links), + links=tuple(_detached_record(link) for link in frame.schema.links), ), { entity: Weights( @@ -393,7 +403,9 @@ def copied_record(record): for entity in frame.weighted_entities }, strata, - mass_log=tuple(copied_record(record) for record in frame.mass_log), + mass_log=tuple(_detached_record(record) for record in frame.mass_log), + # ``Frame.__init__`` re-freezes metadata, which rebuilds every nested + # mapping, so the snapshot shares no metadata object with its parent. metadata=frame.metadata, ) return Population( @@ -401,7 +413,9 @@ def copied_record(record): population.version, dict(population.owners), dict(population.weight_kind), - mass_ledger=tuple(copied_record(record) for record in population.mass_ledger), + mass_ledger=tuple( + _detached_record(record) for record in population.mass_ledger + ), design_weights=population.design_weights, ) @@ -485,9 +499,58 @@ def _context_digest(context: KernelContext) -> bytes: ) digest.update(len(value.payload).to_bytes(8, "little")) digest.update(value.payload) + # The amendment-26 frame fields are kernel inputs like any other, so B4's + # before/after comparison covers them too. They are handed out detached + # (`_project_context`), so a kernel that rewrites one cannot reach the + # live version -- but it can still make its own node's output a function + # of something other than what it was given, and that is what this + # catches. + digest.update(b"frame-metadata\0") + digest.update(_frame_metadata_payload(context.frame_metadata)) + digest.update(b"frame-column-order\0") + # Frame both levels: flattening entity names and columns lets a changed + # mapping reinterpret an entity name as a column without changing bytes. + digest.update(len(context.frame_column_order).to_bytes(8, "little")) + for entity, columns in context.frame_column_order.items(): + _update_scalar(digest, entity) + digest.update(len(columns).to_bytes(8, "little")) + for column in columns: + _update_scalar(digest, column) + digest.update(b"frame-mass-log\0") + for record in context.frame_mass_log: + for value in ( + getattr(record, "entity", None), + getattr(record, "old_total", None), + getattr(record, "new_total", None), + getattr(record, "declared_factor", None), + getattr(record, "reason", None), + ): + _update_scalar(digest, value) return digest.digest() +#: What a metadata tree digests to when the store codec cannot encode it. +#: Reached only after a kernel has replaced a value with something ``Frame`` +#: would never have admitted, which is itself the mutation being reported. +_UNCODABLE_FRAME_METADATA = b"" + + +def _frame_metadata_payload(metadata: Mapping[str, object]) -> bytes: + """Digest bytes for a frame-metadata view, without trusting its contents. + + ``store._encode_frame_metadata`` is the codec the frame format already + uses, so an unchanged view digests exactly as it persists. It is a codec + for *valid* metadata, though, and this runs a second time after a kernel + has held the object: it has to report a mutation rather than raise while + computing the comparison that would report it. + """ + + try: + return canonical_json(_encode_frame_metadata(metadata)) + except (TypeError, ValueError, RecursionError): + return _UNCODABLE_FRAME_METADATA + + def _structural_columns(frame: Frame, entity: str) -> list[str]: columns = [frame.schema.entity_id_column(entity)] if entity == frame.schema.person_entity: @@ -561,7 +624,17 @@ def _project_context( tolerances: Mapping[tuple[str, str], Tolerance | None], numerics: Mapping[tuple[str, str], NumericScope], artifacts: Mapping[str, ArtifactValue] | None = None, + mass_log: tuple[MassChangeRecord, ...] = (), ) -> KernelContext: + """Project one node's frozen inputs out of the population it reads. + + ``mass_log`` is the ``Frame`` mass log this node's *key* binds, which is + not in general the cumulative log carried by ``population``: see the + boundary selection in :func:`run_graph`. It defaults to the empty log, + so a caller that projects a context outside the executor states no mass + history rather than inheriting one it never bound (amendment 26). + """ + if population is None: return KernelContext( node=node, @@ -655,6 +728,16 @@ def _project_context( ) strata = frame.strata.loc[person_mask].copy() _freeze_series(strata) + # The projection above orders each table by declaration, not by the + # version's own layout, so a consumer rebuilding the version's tables + # needs that layout separately -- restricted to what it was given, so it + # never learns the name of a column it cannot read (amendment 26). + column_order: dict[str, tuple[str, ...]] = {} + for entity, table in tables.items(): + projected = set(table.columns) + column_order[entity] = tuple( + column for column in frame.table(entity).columns if column in projected + ) return KernelContext( node=node, tables=MappingProxyType(tables), @@ -664,6 +747,14 @@ def _project_context( rng=np.random.default_rng(seed(key)), sources=MappingProxyType({name: sources[name] for name in node.sources}), artifacts={} if artifacts is None else artifacts, + # Detached before the kernel sees them: a frozen dataclass still + # yields to ``object.__setattr__``, so passing the version's own + # ``_FrozenMapping`` leaves and mass records by reference would make + # every kernel -- and anything that retains a context past its own + # mutation check -- a live handle on the population (amendment 26). + frame_metadata=deepcopy(frame.metadata), + frame_mass_log=tuple(_detached_record(record) for record in mass_log), + frame_column_order=MappingProxyType(column_order), tolerances=tolerances, numerics=numerics, ) @@ -2284,6 +2375,11 @@ def run_graph( _preflight_require(compiled, store, keys, implementations, kernels) populations: dict[str, Population] = {} + # Each structural version's ``Frame`` mass log as that version was + # admitted. ``populations`` cannot answer this: an ordinary member + # replaces its version's entry there, so by the time a later member runs + # the boundary state is gone. + boundary_mass_logs: dict[str, tuple[MassChangeRecord, ...]] = {} receipts: dict[str, NodeReceipt] = {} receipt_payloads: dict[str, Mapping[str, object]] = {} for node_id in compiled.order: @@ -2404,6 +2500,25 @@ def run_graph( ) for binding in node.artifact_inputs } + if incumbent is None: + incoming_mass_log: tuple[MassChangeRecord, ...] = () + elif node.structural is StructuralDelta.NONE: + # An ordinary node's key binds its version's structural + # boundary and the owners of the columns it declared + # (``population_input`` and ``input_artifacts`` in keys.py) -- + # never a sibling that merely ran earlier in the same version + # and appended a mass record. Projecting the cumulative log + # would hand the kernel an input its own key does not bind, so + # adding or re-parameterising that sibling would change what a + # node sees while its key, and therefore its cache entry, + # stayed put. It is projected from the boundary instead. + incoming_mass_log = boundary_mass_logs[compiled.versions[node_id]] + else: + # A structural node's key binds its base *and* every ordinary + # member of that version (``members`` in keys.py, built from + # ``compiled.predecessors``), so the base version's cumulative + # log is bound by the key that will name its cache entry. + incoming_mass_log = incumbent.frame.mass_log context = _project_context( node, incumbent, @@ -2412,6 +2527,7 @@ def run_graph( tolerances=input_tolerances, numerics=input_numerics, artifacts=artifact_values, + mass_log=incoming_mass_log, ) before = _context_digest(context) try: @@ -2557,6 +2673,9 @@ def run_graph( populations[compiled.versions[node_id]] = updated else: populations[node.id] = updated + # Captured at admission, so cold execution and a restored cache + # hit record the same boundary: both reach this line. + boundary_mass_logs[node.id] = updated.frame.mass_log if _population_observer is not None: _population_observer(node_id, _observer_snapshot(updated)) diff --git a/packages/microcosm-graph/src/microcosm/graph/explain.py b/packages/microcosm-graph/src/microcosm/graph/explain.py index d17d8be94..2a3b87d76 100644 --- a/packages/microcosm-graph/src/microcosm/graph/explain.py +++ b/packages/microcosm-graph/src/microcosm/graph/explain.py @@ -13,7 +13,7 @@ from typing import TYPE_CHECKING from .availability import execution_state -from .decl import GATE_OUTCOMES, CompiledGraph, StructuralDelta +from .decl import GATE_OUTCOMES, CompiledGraph, StructuralDelta, WeightUpdate from .manifest import NodeReceipt, RunManifest from .population import mass_record_receipt from .view import describe @@ -884,9 +884,11 @@ def _frame_ratios(frame, anchor, entity: str) -> list[float]: except (KeyError, ValueError): return [] design_by_id = dict(zip(design_ids, design_weights, strict=True)) + if any(entity_id not in design_by_id for entity_id in entity_ids): + return [] ratios = [] for entity_id, current in zip(entity_ids, values, strict=True): - design = float(design_by_id.get(entity_id, 0.0)) + design = float(design_by_id[entity_id]) current_value = float(current) if design > 0: ratios.append(current_value / design) @@ -902,6 +904,14 @@ def _population_ratios( manifest: RunManifest, node: Node, ) -> tuple[list[float], list[float]]: + """Infer ratios only when the manifest retains the original anchor. + + Manifest populations are Frames, not Populations with carried design + anchors. CREATE design weights suffice across updates and filters, but + an EXPAND needs copy/entrant ancestry not carried by those Frames. In + that case leave the fallback unavailable; explicit receipt samples + remain usable. A later design-kind update never becomes a new anchor. + """ if node.weights is None: return [], [] before = None if node.base is None else manifest.populations.get(node.base) @@ -912,17 +922,24 @@ def _population_ratios( anchor_node = node.base anchor = None while anchor_node is not None: - candidate = manifest.populations.get(anchor_node) - if candidate is not None and entity in candidate.weighted_entities: - weights = candidate.weights_for(entity) - if str(_value(weights.kind)) == "design": - anchor = candidate - break declaration = compiled.graph.node(anchor_node) + if declaration.structural is StructuralDelta.EXPAND: + return [], [] + if declaration.structural is StructuralDelta.CREATE: + candidate = manifest.populations.get(anchor_node) + if candidate is not None and entity in candidate.weighted_entities: + weights = candidate.weights_for(entity) + if str(_value(weights.kind)) == "design": + anchor = candidate + break anchor_node = declaration.base if anchor is None: return [], [] - return _frame_ratios(before, anchor, entity), _frame_ratios(after, anchor, entity) + before_ratios = _frame_ratios(before, anchor, entity) + after_ratios = _frame_ratios(after, anchor, entity) + if not before_ratios or not after_ratios: + return [], [] + return before_ratios, after_ratios def _numeric_sequence(value: object) -> list[float]: @@ -965,7 +982,10 @@ def _histogram_svg(before: Sequence[float], after: Sequence[float]) -> str: finite = [*finite_before, *finite_after] nonfinite = len(before) + len(after) - len(finite) if not finite: - return '
Weight-ratio samples are not present in this manifest.
' + return ( + 'Weight-ratio samples are not recorded or original ' + "design ancestry is unavailable in this manifest.
" + ) low, high = min(finite), max(finite) bin_count = min(12, max(4, int(math.sqrt(len(finite))))) if math.isclose(low, high): @@ -1196,12 +1216,22 @@ def _render_calibration(compiled: CompiledGraph, manifest: RunManifest) -> str: mass = _mass_payload(manifest, node, receipt) transition = node.weights assert transition is not None + # A same-kind update does not move the kind, so it does not get the + # arrow that says it did (amendment 25). + kind_label = ( + f"{_escape(transition.entity)} → {_escape(transition.to_kind)}" + if not isinstance(transition, WeightUpdate) + else ( + f"{_escape(transition.entity)} · {_escape(transition.kind)} " + f"updated ({_escape(transition.reason)})" + ) + ) cards.append( '{_escape(node.kernel)} · '
- f"{_escape(transition.entity)} → {_escape(transition.to_kind)}