Range the replay-safety guard over the substrates the factory can build

apps/api/src/cora/operation/adapters/decider_replayability.py

@@ -7,7 +7,8 @@
 satisfy this (the grid walker, the Sobol seeder, the in-memory fake); a GP
 Bayesian-optimization brain does not, because its fit + acquisition are not
 bit-reproducible across BLAS / threading / hardware / library versions even
-with a fixed seed.
+with a fixed seed, and neither does an LLM brain, whose provider may resample
+or move the model under an unpinned snapshot.
 
 This module makes that distinction EXPLICIT and machine-checkable rather than
 leaving it as prose. `is_replay_safe(model_ref)` answers whether a run whose
@@ -33,75 +34,148 @@
 a GP run at once. This classifier remains how a consumer that RE-ASKS tells the
 two classes apart.
 
-## Keep this in lockstep with the deciders
-
-Every decider's `_MODEL_REF` constant must appear in exactly one of the two
-sets below. A fitness test asserts the union covers every shipped decider's
-`model_ref`, so adding a decider without classifying it fails CI rather than
-defaulting to a silent (and possibly wrong) replay-safety assumption.
+## What the coverage guard ranges over, and why it is the factory's Literal
+
+`SUBSTRATE_REPLAYABILITY` below is keyed by `DecideSubstrate`, the Literal that
+`decide_port_config` maintains as the set of substrates its factory can build.
+That list is the INDEPENDENT side of the check: it is edited for a different
+reason (making an adapter buildable) by whoever adds an adapter, and a
+substrate that is not in it cannot be constructed at all. The fitness test
+asserts the two sets are equal in BOTH directions, so adding a seventh
+substrate fails CI until it is classified, and deleting one fails until its
+classification goes too.
+
+The guard this replaces did not have an independent side. It asserted a
+hand-written literal set of four refs against the classifier's own hand-written
+sets, so it agreed by construction, and it never ranged over the adapters at
+all. It reported nothing while the `llm` substrate shipped a `model_ref` that
+no set could ever contain.
+
+## Substrate name is not the same thing as recorded ref
+
+Four substrates record their own name verbatim, so their recorded ref and their
+substrate key coincide. Two do not:
+
+  - `staged` records NO ref of its own. It returns the child brain's advice
+    unchanged, so what lands on the iteration is `sobol` or `botorch`. Its
+    classification is `DELEGATED`, and seeing the literal string `"staged"` on
+    an iteration means the composite grew a ref it should not have, which is
+    why that input raises rather than classifying.
+  - `llm` records `f"{provider}:{model}"`, so its recorded refs are an open set
+    (one per model ever configured) rather than a single constant. A ref
+    carrying a colon is read as this form; substrate names are bare
+    identifiers and never carry one.
 """
 
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from enum import StrEnum
+from typing import TYPE_CHECKING, Final
 
 if TYPE_CHECKING:
-    from collections.abc import Iterable
-
-_REPLAY_SAFE_MODEL_REFS = frozenset(
-    {
-        "in_memory",  # the deterministic fake: advice is seeded by iteration index
-        "grid_walk",  # pure function of space + observation count
-        "sobol",  # unscrambled Sobol: pure function of space + draw index
-    }
-)
-"""`model_ref`s whose advice is a pure function of the evidence.
-
-A run decided entirely by these brains replays faithfully by re-asking the
-brain, so it carries the conduct loop's replay-determinism property.
-"""
+    from collections.abc import Iterable, Mapping
+
+    from cora.operation.adapters.decide_port_config import DecideSubstrate
+
+
+class Replayability(StrEnum):
+    """How a decider substrate behaves when its advice is RE-ASKED."""
+
+    REPLAY_SAFE = "replay_safe"
+    """Advice is a pure function of the evidence, so a re-ask reproduces it."""
+
+    FORWARD_ONLY = "forward_only"
+    """Advice is not bit-reproducible on re-ask.
 
-_FORWARD_ONLY_MODEL_REFS = frozenset(
-    {
-        "botorch",  # GP fit + acquisition: not bit-reproducible across environments
-    }
-)
-"""`model_ref`s whose advice is NOT bit-reproducible on RE-ASK.
-
-"forward-only" here means NOT re-ask-reproducible -- re-running the brain over
-the same evidence may diverge (GP fit + acquisition are not bit-reproducible
-across BLAS / threading / hardware / version). It is DISTINCT from "not
-recorded" AND from "not resumable": TIER-1 replay records the brain's
-advised_next_point on the iteration event + surfaces it in the iteration
-projection, so a finished GP-steered run IS reconstructable by READING the
-recorded trail; and `conduct_until_advised_from` now RESUMES such a run by
-re-seeding the brain from the recorded (x, y) history (re-seed, not re-ask).
-botorch STAYS in this set regardless: the classifier judges the RE-ASK path,
-which is still non-reproducible. A run is not reclassified replay-safe because
-its decisions are recorded or because it can be resumed; re-seed-from-record
-side-steps the re-ask rather than making the re-ask reproducible. Flipping a
-ref to replay-safe would require the RE-ASK itself to become bit-reproducible,
-which a GP's fit + acquisition are not.
+    DISTINCT from "not recorded" AND from "not resumable": the iteration event
+    records the brain's advised_next_point and the projection surfaces it, so a
+    finished run IS reconstructable by READING the recorded trail, and
+    `conduct_until_advised_from` RESUMES such a run by re-seeding the brain from
+    the recorded (x, y) history. A substrate stays in this class regardless: the
+    classifier judges the RE-ASK path. A run is not reclassified replay-safe
+    because its decisions are recorded or because it can be resumed. Moving a
+    substrate out of this class would require the RE-ASK itself to become
+    bit-reproducible.
+    """
+
+    DELEGATED = "delegated"
+    """Records no ref of its own; the child brain's ref is what lands.
+
+    A composite. Its runs classify per iteration by whichever child decided
+    each one, so the composite's own name never reaches `is_replay_safe`.
+    """
+
+
+SUBSTRATE_REPLAYABILITY: Final[Mapping[DecideSubstrate, Replayability]] = {
+    "in_memory": Replayability.REPLAY_SAFE,
+    "grid_walk": Replayability.REPLAY_SAFE,
+    "sobol": Replayability.REPLAY_SAFE,
+    "botorch": Replayability.FORWARD_ONLY,
+    "staged": Replayability.DELEGATED,
+    "llm": Replayability.FORWARD_ONLY,
+}
+"""Every substrate `build_decide_port` can materialise, with its replayability.
+
+`in_memory` is seeded by iteration index, `grid_walk` is a pure function of
+space + observation count, and unscrambled `sobol` is a pure function of space
++ draw index, so all three reproduce on re-ask. `botorch` does not: GP fit +
+acquisition are not bit-reproducible across BLAS / threading / hardware /
+version. Neither does `llm`: the provider may sample, and an unpinned snapshot
+may move the model under a caller who never changed a line. `staged` records no
+ref of its own.
+
+Kept equal to `DecideSubstrate` in both directions by
+`tests/unit/operation/test_decider_replayability.py`.
 """
 
-_CLASSIFIED_MODEL_REFS = _REPLAY_SAFE_MODEL_REFS | _FORWARD_ONLY_MODEL_REFS
+_LLM_REF_SEPARATOR = ":"
 
 
-def is_replay_safe(model_ref: str) -> bool:
-    """True if a run decided by `model_ref` replays faithfully by re-asking.
+def _is_llm_ref(model_ref: str) -> bool:
+    """True if `model_ref` has the `llm` substrate's `provider:model` shape.
 
-    Raises `ValueError` for an unclassified `model_ref` so an unknown brain is
-    a loud failure, never a silent assume-safe. The staged composite never
-    appears here: it records the CHILD brain's `model_ref` on each iteration
-    (it adds no `model_ref` of its own), so a staged run's iterations classify
-    individually (its Sobol passes are safe, its BoTorch passes are not).
+    Partitions on the FIRST separator only: a provider's model identifier may
+    itself carry one (`ollama:llama3:8b`), and everything after the provider is
+    the model. A half-empty ref (`anthropic:` or `:claude-sonnet-4-6`) is NOT
+    this shape, so it falls through to the unclassified branch and raises
+    rather than being read as a nameless model.
     """
-    if model_ref not in _CLASSIFIED_MODEL_REFS:
+    provider, _, model = model_ref.partition(_LLM_REF_SEPARATOR)
+    return bool(provider) and bool(model)
+
+
+def replayability_of(model_ref: str) -> Replayability:
+    """The `Replayability` of the brain that recorded `model_ref`.
+
+    Raises `ValueError` for a ref this module cannot place, so an unknown brain
+    is a loud failure and never a silent assume-safe. `DELEGATED` substrates
+    raise too: a composite records its child's ref, so its own name appearing on
+    an iteration is a defect in the composite rather than a classifiable run.
+    """
+    if _is_llm_ref(model_ref):
+        return SUBSTRATE_REPLAYABILITY["llm"]
+
+    replayability = SUBSTRATE_REPLAYABILITY.get(model_ref)  # pyright: ignore[reportArgumentType]
+    if replayability is None:
+        raise ValueError(
+            f"unclassified decider model_ref {model_ref!r}; add its substrate to "
+            "SUBSTRATE_REPLAYABILITY"
+        )
+    if replayability is Replayability.DELEGATED:
         raise ValueError(
-            f"unclassified decider model_ref {model_ref!r}; add it to "
-            "_REPLAY_SAFE_MODEL_REFS or _FORWARD_ONLY_MODEL_REFS"
+            f"decider model_ref {model_ref!r} names a composite substrate, which "
+            "records its child brain's ref rather than its own; an iteration "
+            "carrying it means the composite grew a ref it should not have"
         )
-    return model_ref in _REPLAY_SAFE_MODEL_REFS
+    return replayability
+
+
+def is_replay_safe(model_ref: str) -> bool:
+    """True if a run decided by `model_ref` replays faithfully by re-asking.
+
+    Raises `ValueError` for a ref `replayability_of` cannot place.
+    """
+    return replayability_of(model_ref) is Replayability.REPLAY_SAFE
 
 
 def run_is_replay_safe(model_refs: Iterable[str | None]) -> bool:
@@ -118,6 +192,9 @@ def run_is_replay_safe(model_refs: Iterable[str | None]) -> bool:
 
 
 __all__ = [
+    "SUBSTRATE_REPLAYABILITY",
+    "Replayability",
     "is_replay_safe",
+    "replayability_of",
     "run_is_replay_safe",
 ]

apps/api/tests/unit/operation/test_decider_replayability.py

@@ -1,21 +1,37 @@
 """Replayability classification of decider brains.
 
-Pins the explicit forward-only-vs-replay-safe boundary for GP-steered runs
-(the P1-a "explicit, not silent" requirement): BoTorch is forward-only, the
-pure-function brains are replay-safe, an unknown ref is a loud failure, and a
-run classifies by the worst of its per-iteration model_refs. A lockstep
-fitness test asserts every shipped decider's _MODEL_REF is classified.
+Pins the explicit forward-only-vs-replay-safe boundary for GP-steered and
+LLM-steered runs: the pure-function brains are replay-safe, BoTorch and the LLM
+brain are forward-only, the staged composite classifies through its children
+rather than under its own name, an unknown ref is a loud failure, and a run
+classifies by the worst of its per-iteration model_refs.
+
+The coverage guard ranges over `DecideSubstrate`, the factory's own Literal,
+which is the independent side: it is maintained for a different reason (which
+adapters `build_decide_port` can construct) than the classification is, and a
+substrate absent from it cannot be built at all. The guard this replaces
+asserted a hand-written set of four refs against the classifier's own
+hand-written sets, so it agreed by construction and never saw the `llm`
+substrate at all.
 """
 
 from __future__ import annotations
 
+from typing import get_args
+
 import pytest
 
+from cora.operation.adapters.decide_port_config import DecideSubstrate
 from cora.operation.adapters.decider_replayability import (
+    SUBSTRATE_REPLAYABILITY,
+    Replayability,
     is_replay_safe,
+    replayability_of,
     run_is_replay_safe,
 )
 
+_LLM_REF = "anthropic:claude-sonnet-4-6"
+
 
 @pytest.mark.parametrize("model_ref", ["in_memory", "grid_walk", "sobol"])
 def test_pure_function_brains_are_replay_safe(model_ref: str) -> None:
@@ -26,11 +42,42 @@ def test_botorch_is_forward_only() -> None:
     assert is_replay_safe("botorch") is False
 
 
+def test_an_llm_ref_is_forward_only() -> None:
+    assert is_replay_safe(_LLM_REF) is False
+
+
+def test_an_unrecognised_provider_still_routes_through_the_llm_branch() -> None:
+    """There is no provider registry: the SHAPE of the ref is what routes it.
+
+    Asserting against `SUBSTRATE_REPLAYABILITY["llm"]` instead would prove
+    nothing, because `botorch` carries the same replayability and the
+    comparison would hold whichever of the two it routed to.
+    """
+    assert replayability_of("nosuchprovider:nosuchmodel") is Replayability.FORWARD_ONLY
+
+
+def test_an_llm_ref_whose_model_carries_a_separator_still_classifies() -> None:
+    """`ollama:llama3:8b`: only the first separator splits provider from model."""
+    assert is_replay_safe("ollama:llama3:8b") is False
+
+
+@pytest.mark.parametrize("model_ref", [":claude-sonnet-4-6", "anthropic:"])
+def test_a_half_empty_llm_ref_is_not_read_as_an_llm_ref(model_ref: str) -> None:
+    with pytest.raises(ValueError, match="unclassified decider model_ref"):
+        is_replay_safe(model_ref)
+
+
 def test_unclassified_model_ref_raises() -> None:
     with pytest.raises(ValueError, match="unclassified decider model_ref"):
         is_replay_safe("mystery_brain")
 
 
+def test_the_staged_composite_name_raises_rather_than_classifying() -> None:
+    """It records its child's ref, so its own name on an iteration is a defect."""
+    with pytest.raises(ValueError, match="names a composite substrate"):
+        is_replay_safe("staged")
+
+
 def test_run_with_all_pure_iterations_is_replay_safe() -> None:
     assert run_is_replay_safe(["sobol", "sobol", "grid_walk", None]) is True
 
@@ -40,19 +87,44 @@ def test_run_not_replay_safe_when_any_iteration_forward_only() -> None:
     assert run_is_replay_safe(["sobol", "sobol", "botorch"]) is False
 
 
+def test_run_not_replay_safe_when_any_iteration_was_llm_decided() -> None:
+    assert run_is_replay_safe(["sobol", _LLM_REF]) is False
+
+
 def test_run_replay_safe_ignores_none_entries() -> None:
     assert run_is_replay_safe([None, None]) is True
 
 
-def test_every_shipped_decider_model_ref_is_classified() -> None:
-    """Lockstep guard: every shipped decider's model_ref must be classified.
+def test_every_buildable_substrate_is_classified() -> None:
+    """Lockstep guard: `SUBSTRATE_REPLAYABILITY` covers every buildable substrate.
+
+    Adding an arm to `DecideSubstrate` without classifying it should fail here
+    rather than default to a silent (and possibly wrong) replay-safety
+    assumption.
+    """
+    assert set(SUBSTRATE_REPLAYABILITY) == set(get_args(DecideSubstrate))
+
+
+def test_no_classification_names_a_substrate_the_factory_cannot_build() -> None:
+    """The other direction: a deleted substrate must lose its classification.
+
+    Without this arm the guard would pass while `SUBSTRATE_REPLAYABILITY` kept a
+    stale entry, and `is_replay_safe` would keep answering for a brain that no
+    longer ships.
+    """
+    stale = set(SUBSTRATE_REPLAYABILITY) - set(get_args(DecideSubstrate))
+    assert stale == set(), f"classified substrates the factory cannot build: {sorted(stale)}"
+
+
+def test_exactly_one_substrate_delegates() -> None:
+    """`replayability_of` refuses DELEGATED names, so the set must stay known.
 
-    Adding a decider without classifying its model_ref should fail here, not
-    default to a silent (and possibly wrong) replay-safety assumption. The
-    expected set is pinned literally; a new decider whose model_ref is added
-    to this set but not to the classifier sets makes is_replay_safe raise.
+    A second composite would need its own refusal reasoning rather than
+    inheriting staged's, so this pins the count until that is thought through.
     """
-    shipped_refs = {"in_memory", "grid_walk", "sobol", "botorch"}
-    for ref in shipped_refs:
-        # Must not raise (every ref is in one of the two classified sets).
-        is_replay_safe(ref)
+    delegated = {
+        substrate
+        for substrate, replayability in SUBSTRATE_REPLAYABILITY.items()
+        if replayability is Replayability.DELEGATED
+    }
+    assert delegated == {"staged"}