Skip to content

Add the transport skeleton composer, rules registry and local build driver (NZ graph G6) - #1161

Merged
MaxGhenis merged 8 commits into
mainfrom
transport/compose-cli
Oct 9, 2026
Merged

MaxGhenis merged 8 commits into
mainfrom
transport/compose-cli

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

G6 of the Microcosm NZ graph plan (Max's d962): the transport skeleton composer, its rules registry, the local build driver and the shim. Under the hub's G6/G7 split amendment, the AS chain is out of scope here. The bridges, nz.as, scenarios, variants, gaps, bands and the WFF tripwire come in G7 through the extension point this PR adds. The PR builds on #1159 (G5a, merged).

What it adds (packages/microcosm-build/src/microcosm/build/transport/)

  • registry.py (extends G5a's): build_transport_registry(...) checks the rules bindings: it hashes the actual RuleSpec module and tree bytes, and checks schema, roots, entity names (each engine reports its own entity), output variables and graph dtypes. The engine commit, wheel SHA-256 and exported-tree RuleSpec commit are caller declarations that are format-checked only; they are not verified against the installed engine. It binds every rules binding into one simulate.rules_by_ref@1 instance, and registers the codecs plus the G5a population, geography, G5b target/ordered-calibration/terminal and G5c gate kernels. Each engine's schema and nesting come from the benefit-unit rule, the same expression CREATE uses, so no country schema is named in build/transport/. Injected pure-Python adapters declare their cache_identity() and get refs that identify Python execution.
  • compose.py: compose_transport_graph(spec, config) -> Graph.
    • The country's transport_graph.json (a legacy_json spec resource) declares the skeleton.
    • A node parameter can select a resource value, its JSON, or its SHA-256, or the prepared engine refs. So each node binds only the resources it names, and no whole-spec fingerprint enters any key.
    • Rules nodes take their engine ref and period from their binding.
    • SKELETON_KERNELS admits only kernels that exist; anything else must come through TransportGraphConfig.extensions, a typed protocol. Composition refuses an extension that would change a skeleton node's compiled key. Extensions may declare additional sources (collisions refused) and ordered pre-export checkpoints; a public helper builds a rules node from a binding's prepared reference and period. Every extension factory gets its own deep copy of the spec.
    • Selected resource values stay literal data: lists become tuples without interpreting objects inside them, so selector-shaped data cannot smuggle a selection. The whole-spec fingerprint and the graph resource's own digest are refused as selections.
    • Hold-out ancestry is enforced at compose time on the declaration and the compiled skeleton plus extensions: no ancestor of nz.calibrate may declare hold-out facts, select hold-out references, carry the prepared reference map, or embed the family/AS engine reference.
    • Node ids with path separators, ., .. or NUL are refused at compose time.
    • validate_transport_activation refuses before CREATE or any engine is built. It checks every selected reference resource for non-active rows, every null scenario knob (no country knob name in the transport code) and every selected resource, and names each gap.
  • cli.py + tools/build_transport.py (6-line shim).
    • It reads CREATE's column inventory by running transport.create@1 once, so CREATE's logic isn't copied. The inventory is cached under a key of the kernel implementation, resolved parameters and source bytes, so warm reruns skip it; a cold build still runs CREATE twice (inventory, then in the graph).
    • It then runs checkpoints on one <out>/.graph-store: geography, calibration, export.prepare (then writes skeleton.h5), each terminal, then the whole graph.
    • It writes graph.json, manifest.json, the per-checkpoint manifests, the evidence index, the export descriptor, calibration-diagnostics.json, explorer.html, and a build-status.json that lists not_yet_produced outputs from the spec's pending_outputs.
    • Local-only, with no upload or staging import (an AST test checks this). Every file it writes, including .graph-store, lies under --out: before opening the store it refuses any link in an existing store tree (root, objects, tmp, shards), because the store creates those directories by following links. It does not guard against links planted while a build runs. --resume require refuses a cold store before any preparation. An --out that overlaps an input or --spec-dir is refused. Argument and build errors print a refusal, never a traceback.
    • The export is rewritten timestamp-free after G5b's export is staged and validated (object timestamps and unused search indexes off, same tables), so two fresh builds into fresh --out directories give byte-identical skeleton.h5 and identical keys for every node.
  • Layout decision. Group encoding sits on an nz.as FILTER based on nz.calibrate, not in nz.open. A REWEIGHT node's key binds every member of its base version, so encoding in nz.open would make calibration depend on family entitlement inputs and break hold-out ancestry. This follows plan §2.2/§2.3.

Invariants (property tests on an engine-free toy NZ package with three pure-Python engines in one run_graph; toy run ≈4 s)

  • Purity: composing twice, or composing a deep copy, gives equal graph_to_json.
  • Resource locality (Hypothesis, parametrized over all 15 selected resources): editing one resource re-keys exactly the nodes that select it and their descendants. This generalises A3 and C2.
  • A1/A2: two fresh driver runs into fresh --out directories give equal keys for every node and byte-identical artifacts, including skeleton.h5; a reloaded require run has zero misses.
  • A4: prose moves no key, at both declaration and execution level.
  • A3: a calibration-reference change misses exactly descendants(nz.targets.compile).
  • A5: one RuleSpec byte re-keys exactly the three rules nodes and their descendants.
  • C2: a gates.json threshold or hold-out reference edit is descendant-exact and disjoint from the calibration closure.
  • Hold-out ancestry: no ancestor of nz.calibrate declares nz_holdout_facts, selects hold-out references, carries the prepared reference map, embeds the family engine reference, or touches family-engine columns; enforced at compose time and tested for each rule. The packaged spec's reference names are also checked.
  • B1: the graph compiles. A structural column is refused at run time; a second owner of an engine or family-input column is refused at compile time.
  • E3, and extension isolation.
  • Differentials: the router matches stock simulate.rules@1 per node and column; the prepared CREATE inventory equals G5a's composition; the registry schema equals CREATE's frame schema.
  • No literals: no policy literal in build/transport/*.py, plus the existing source-scan rules in test_transport_graph.py (no country word, no float literal).
  • Mutants: binding a whole-spec digest into every node fails the locality property, and removing the extension guard fails the isolation test.

Real-spec activation (refuses, by design)

The packaged NZ spec refuses before reading any input, naming every gap:

  • the benefit-unit age limit;
  • the non-active precal, target and hold-out references (G8);
  • a structured receipt_contract.json (the closure's receipt requirements are prose);
  • the engine commit and wheel pins;
  • S4's rent_stock_factor;
  • the NZ transport_graph.json itself.

That file is left for D1/G8, because several of its inputs are still undecided.

Tests (one file at a time, by the hub)

At fe47f88 (store containment fix): test_transport_composed_graph.py 123 (115 plus 8 containment cases), test_transport_graph.py 15. With the old root-only store check restored in memory, the 4 execution and main() containment cases fail.
At e7953f3 (review fixes): test_transport_composed_graph.py 115, test_transport_registry.py 31, test_transport_graph.py 15. At 25c47a5 (after merging main with #1160 and #1162): test_transport_composed_graph.py 115, test_transport_population_kernels.py 27, test_telemetry_emitter.py 33. All passed. The fix lane's baselines show the new tests failing on 5bd688c (determinism 1 failed; refusals 20 failed; composer 29 failed; CLI 4 failed; entity echo 2 failed); evidence in the hub's job folder.

Also in this PR

Both prose property tests now map their strategies through str.strip, because a whitespace-only draw is an invalid Node.description. The G5a A4 test failed this way in 2 of 6 local runs on main. The same one-line fix is microcosm#1160; the change is identical, so whichever lands second merges cleanly.

Review

  • r1 (independent in-session Opus 5.5 review at 5bd688c): REQUEST_CHANGES, Needs Max: no. Core code sound; asked for a deterministic H5, a complete hold-out ancestry check, removal of the hidden _param indirection, narrowed claims, and driver/test fixes; also flagged main's telemetry date failure (fixed by Stamp migrated-spool fixture rows inside the retention window #1162) and a plan wording error (fixed in the hub's plan and decision d1043).
  • Fixes in e7953f3 (GPT-6.1 Sol lane), merge of main in 25c47a5.
  • r2 (GPT-6.1 Sol at 25c47a5): REQUEST_CHANGES on one blocking defect, an executed probe showing a pre-existing .graph-store/objects link sent store writes outside --out; every other r1 fix held in its probes (8/8 composition probes). The review stopped early on its disk floor, so test runs and mutation checks were not completed.
  • Fix in fe47f88 (hub, Claude Opus 5.5): store-tree link refusal, tests, and scoped docs/changelog claims.
  • r3 review pending (GPT-6.1 Sol). Do not merge before it agrees.

Part of Microcosm NZ v0 (d962), package G6 of the NZ hub's graph plan.

  • Round 1 (registry): built on a GPT-6.1 Sol lane.
  • Round 2 (drafts): a Sol lane stopped at the disk floor.
  • Round 3 (composer, CLI, acceptance): in-session Claude Opus 5.5, because lanes were held for disk.

Verified, committed and opened by the NZ hub.

🤖 Generated with Claude Code

MaxGhenis and others added 8 commits October 9, 2026 02:43
Prepare isolated source codecs and shared population, geography, target,
calibration, gate and terminal kernels. Route all pinned rules bindings through
one simulate.rules_by_ref kernel, validate adapter roots and relations, and bind
injected Python adapters to explicit cache configuration identities.

Add engine-free property and differential coverage for registry dispatch.
Full composer and CLI work remains blocked on the absent AS bridge, gap and
bands kernels; no stand-in policy calculations are introduced.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- compose.py: compose_transport_graph(spec, config) builds the skeleton a
  country's transport_graph.json spec resource declares. A node parameter
  that is an object must be an explicit resource selection (value, json or
  the resource's SHA-256) or the prepared engine-reference map, so a node
  binds only the resources it selects; no whole-spec fingerprint enters
  any node. Rules nodes name a binding and get its authenticated engine_ref
  and period, all through the one simulate.rules_by_ref@1 router. Kernels
  outside the skeleton vocabulary enter only through
  TransportGraphConfig.extensions, the typed extension point for the
  entitlement package; composition refuses an extension that changes any
  skeleton node's predecessors. validate_transport_activation refuses
  unresolved evidence with every item named (the benefit-unit age limit,
  placeholder references, the receipt contract, engine pins, the S4 rent
  factor, a missing declaration or selected resource).
- cli.py: prepare (registry; CREATE's column inventory, read by running
  transport.create@1 once on the declared params), compile, then run
  ancestor-closed checkpoints on <out>/.graph-store: geography, calibration,
  export preparation (then the skeleton H5), each terminal endpoint, the
  whole graph. Writes graph.json, manifest.json, checkpoint manifests,
  artifacts, the H5 and descriptor, calibration diagnostics, explorer.html
  and build-status.json (with the outputs still owed) under --out only.
  --resume require refuses a cold store before creating anything; an
  --out that overlaps an input is refused. No upload or staging path.
- registry.py: build the engines' schema and unit nesting from the
  benefit-unit rule (the schema CREATE builds) instead of naming a country
  schema, check injected and Axiom adapters against both, and drop
  country words from messages, so the transport source rules pass.
- tools/build_transport.py: the six-line shim.
- Tests on an engine-free toy package (synthetic donor, facts for both
  roles, toy RuleSpec tree, three pure-Python engines in one run_graph):
  purity; a Hypothesis resource-locality property (one resource edit
  re-keys exactly its selectors and their descendants); A1, A2, A4, A5;
  A3 for a calibration reference; C2 for a gates.json threshold and a
  hold-out reference; hold-out ancestry; B1; extension isolation; E3; CLI
  outputs, warm resume, H5 rewrite and corruption refusal; differentials
  (router vs stock simulate.rules@1, prepared vs independent CREATE
  inventory, registry vs CREATE schema, declared vs executed keys); the
  packaged spec's named refusals; no policy literal or remote import.

Invariants: the graph is a pure function of the spec resources and the
prepared config; editing one resource re-keys exactly the nodes that select
it and their descendants; the extension point never re-keys a skeleton
node; no calibration ancestor reads the hold-out facts or a family-engine
column; every rules node runs through the one router; every output lies
under --out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both prose properties drew from alphabets that include a space, so
Hypothesis could draw a whitespace-only string such as " ", which
graph/decl.py's _description refuses. The G5a A4 test failed this way in
2 of 6 local runs on main (the same fix is microcosm#1160); the composed
graph's purity/prose test has the same latent case. Map both strategies
through str.strip: the result is empty or contains a non-whitespace
character, which is what decl.py accepts.

tests: composed prose property 4/4 runs passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Write timestamp-free HDF5 exports without changing shared export kernels,
validate calibration ancestry and activation selections, and preserve literal
resource data. Cache CREATE inventories, refuse unsafe CLI paths and ids,
isolate extension specs, and expose binding-driven rules nodes with additional
sources and checkpoints. Narrow pin-provenance claims and expand regressions.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round-2 review of #1161 showed that a pre-existing
`<out>/.graph-store/objects` link sends ContentStore writes outside --out:
ContentStore creates `objects`, `tmp` and shard directories with
mkdir(exist_ok=True), which follows an existing link, and the driver only
checked the store root itself.

`_contained_store_root(out)` now checks the root with `_destination` and,
when a store tree already exists, walks it without following links and
refuses any entry that is a link (root, objects, tmp, shards, objects).
Both store entry points use it: main(), before the spec is loaded, and
execute_transport_graph(), before the ContentStore is opened. The agent
guide and changelog now say what is guarded (links present when a build
starts) and what is not (links planted while it runs).

Tests: a link at objects, tmp or objects/aa is refused by
_contained_store_root, by execute_transport_graph and by main(), and
nothing is written to the link target; a plain store tree is accepted.
With the old root-only check restored in memory (a pytest plugin, no
tracked file changed), the 4 execution and main() cases fail (DID NOT
RAISE; main() reached spec loading). test_transport_composed_graph.py
123 passed; test_transport_graph.py 15 passed; ruff check and format pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 523a18a into main Oct 9, 2026
10 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the NZ hub (session local_e407a527) at fe47f88 with --merge. Gates: gh pr checks exit 0; MERGEABLE; not draft; no CHANGES_REQUESTED review. Reviewed head: fe47f88. Independent review chain: r1 (in-session Opus 5.5 at 5bd688c) REQUEST_CHANGES → fixes e7953f3 (GPT-6.1 Sol lane); r2 (GPT-6.1 Sol at 25c47a5) REQUEST_CHANGES on one executed defect, a pre-existing .graph-store link sending store writes outside --out → fix fe47f88 (hub); r3 (GPT-6.1 Sol at fe47f88) APPROVE_WITH_NITS, Needs Max: no (jobs/review-microcosm-1161/REPORT-r3.md): 188 passes, all 154 new test cases fail under mutation, containment probes at every link position refused. Non-blocking nits go to a follow-up PR: the 60 s wall-clock assertion is host-sensitive (91 s once on an overloaded host; passes in CI), and the activation preflight wording is broader than the code (a selected null scalar or an empty receipt contract is refused one step later). All 10 checks green, MERGEABLE/CLEAN, no CHANGES_REQUESTED.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant