feat(novelty): --repeats for routing noise floor; keep declined replies

dokimi/changelog.d/20261021_230000_napetrov_routing-repeats.md

@@ -0,0 +1,3 @@
+### Added
+
+- **`novelty routing --repeats N` and the raw reply of a declined pick** — `--repeats N` (with `--model`) asks the model N times per query, with the same card order each time, so the spread between repeats is the model's own sampling noise. The `model` block gains `noise`: each query's agreement across its repeats (`per_query`), the selection rate of each repeat, and `half_range`, half the gap between the highest and lowest of those rates, a crude noise floor.  The rates cover only the queries with no failed call in any repeat (`n_queries_compared`). Each repeat is a fresh sample, so with `DOKIMI_RESPONSE_CACHE` set a repeat is no longer a replay of the first reply. Every abstention or unparseable pick now keeps the model's raw reply under `per_query[].raw_reply` (first 2000 characters). `LLMClient.text` takes a `draw` index for the same reason.

dokimi/docs/skill-novelty.md

@@ -588,7 +588,7 @@ the absence of detectable corruption — not proof that the wordings are faithfu
 ```bash
 python cli.py novelty routing --artifact skill:path/to/skill \
     --competitors skill:sibling-a skill:sibling-b ... \
-    --queries queries.json [--model M]
+    --queries queries.json [--model M [--repeats N]]
 ```
 
 `--catalog <dir>` takes the competitors from a skill directory instead: every `SKILL.md`
@@ -608,6 +608,18 @@ Two numbers, reported side by side and never merged:
   `selection_rate`** — not picking anything is the routing failure — with
   `selection_rate_among_answered` beside it; a failed call is excluded from both and
   counted, and one failed query does not lose the rest.
+  Each abstention or unparseable pick keeps the model's raw reply under
+  `per_query[].raw_reply` (the first 2000 characters), so a decline can be read.
+  `--repeats N` asks each query N times with the same card order, so the spread between
+  repeats is the model's own sampling noise. The result's `noise` block gives each query's
+  agreement across its repeats (`per_query`), the selection rate of each repeat, and
+  `half_range`, half the gap between the highest and lowest of those rates: a crude noise
+  floor. The rates cover only the queries with no failed call in any repeat
+  (`n_queries_compared`), so a transient error is not read as noise. Each repeat is a
+  fresh sample, so under `DOKIMI_RESPONSE_CACHE` a repeat is not a
+  replay. Repeats need `--model`, and cost N times the calls. With repeats, `n_calls` is
+  queries x repeats, `n_abstained` and `n_failed` count calls, and each `per_query` row
+  carries its `repeat` index.
 
 `recall_at_3` comes back `null` with three or fewer candidates, where every rank is `<= 3`
 and the figure would be a structural 1.0. The lexical arm also reports `n_queries_given`

dokimi/dokimi/commands/novelty.py

@@ -264,6 +264,11 @@ def cmd_novelty_items(args: argparse.Namespace) -> None:
 
 def cmd_novelty_routing(args: argparse.Namespace) -> None:
     """Can this artifact be selected against its competitors? Lexical floor, model on top."""
+    if args.repeats > 1 and not args.model:
+        raise SystemExit(
+            "--repeats needs --model: the lexical floor is deterministic, so repeating it "
+            "measures nothing"
+        )
     from dokimi.novelty.routing import (
         Candidate,
         RoutingInputError,
@@ -338,7 +343,8 @@ def card(ref: str) -> tuple[Candidate, str]:
     if args.model:
         selector = _client(args.model, args.provider, "selector", args)
         payload["model"] = dict(
-            model_routing(candidates, queries, client=selector, gold=0, seed=args.seed),
+            model_routing(candidates, queries, client=selector, gold=0, seed=args.seed,
+                          repeats=args.repeats),
             queries_self_derived=self_derived,
         )
 
@@ -369,6 +375,17 @@ def _print_routing_summary(payload: dict, n_competitors: int, self_derived: bool
               f"({row['selection_rate_among_answered']} among answered; "
               f"{row['n_abstained']} abstained, {row['n_failed']} failed)  "
               f"[{row['model']}]  n_queries {row['n_queries']} given{tag}")
+        if row["n_abstained"]:
+            of_calls = f" of {row['n_calls']} calls" if row["repeats"] > 1 else ""
+            print(f"{row['n_abstained']}{of_calls} abstained or unparseable: the raw reply "
+                  "is under per_query[].raw_reply in the output")
+        if row["noise"]:
+            noise = row["noise"]
+            print(f"noise    {row['repeats']} repeats per query: mean agreement "
+                  f"{noise['mean_agreement']}, {noise['unstable_queries']} queries picked "
+                  f"differently across repeats; selection by repeat "
+                  f"{noise['selection_rate_by_repeat']}, half range "
+                  f"{noise['half_range']} (a crude sampling floor)")
         print("The model figure is an upper bound on harness behaviour: a harness that "
               "truncates descriptions in its own listing selects on less text.")
     if self_derived:
@@ -640,6 +657,10 @@ def register(sub, positive_int) -> None:
     routing_p.add_argument("--seed", type=int, default=0,
                            help="Seeds the per-query card permutation, so a position-biased "
                                 "model scores ~1/N and the run is reproducible")
+    routing_p.add_argument("--repeats", type=positive_int, default=1,
+                           help="Ask the model this many times per query, with the same card "
+                                "order each time, and report per-query agreement and the "
+                                "run-to-run noise floor. Needs --model; costs N times the calls")
     routing_p.add_argument("--out", default=None)
     _add_spend_cap(routing_p)
     routing_p.set_defaults(func=_stops_at_cap(cmd_novelty_routing))

dokimi/dokimi/novelty/client.py

@@ -106,7 +106,8 @@ class LLMClient:
     cap: SpendCap | None = None
 
     def text(
-        self, prompt: str, *, system: str | None = None, accept: Accept | None = None
+        self, prompt: str, *, system: str | None = None, accept: Accept | None = None,
+        draw: int = 0,
     ) -> str:
         """Return the model's reply as plain text.
 
@@ -115,13 +116,18 @@ def text(
         ``DOKIMI_RESPONSE_CACHE`` set, a cut-off or unparseable reply is stored and every
         later run replays the failure without calling the model. An injected transport
         is the caller's own and is not given it.
+
+        ``draw`` is the sample index of this prompt. With the response cache on, a repeat
+        of a byte-identical prompt replays draw 0 unless it has its own index, so a
+        repeated ask is only a new sample when it passes a new one. Draw 0 is the default
+        and keeps every existing cache entry. An injected transport is not given it.
         """
         if self.cap is not None:
             self.cap.before_call(self.model)
         if self.transport is not None:
             reply, usage = self.transport(prompt, system)
         else:
-            reply, usage = self._default_transport(prompt, system, accept=accept)
+            reply, usage = self._default_transport(prompt, system, accept=accept, draw=draw)
         self.calls += 1
         if usage is not None:
             # UsageRecord is frozen; re-stamp the role so cost rolls up per role.
@@ -180,7 +186,8 @@ def cost_summary(self) -> dict[str, Any]:
         }
 
     def _default_transport(
-        self, prompt: str, system: str | None, *, accept: Accept | None = None
+        self, prompt: str, system: str | None, *, accept: Accept | None = None,
+        draw: int = 0,
     ) -> tuple[str, UsageRecord | None]:
         from dokimi.llm import llm_call_with_usage
 
@@ -191,6 +198,7 @@ def _default_transport(
             system=system,
             max_tokens=self.max_tokens,
             cache_if=accept,
+            draw=draw,
         )
 
 

dokimi/dokimi/novelty/routing.py

@@ -229,11 +229,15 @@ def _as_pick_position(value: object) -> int:
     return 0
 
 
+# A reply that declines to route is kept verbatim, up to this many characters.
+_RAW_REPLY_CHARS = 2000
+
+
 def model_routing(
     candidates: list[Candidate], queries: list[str], *, client: LLMClient, gold: int = 0,
-    seed: int = 0,
+    seed: int = 0, repeats: int = 1,
 ) -> dict:
-    """Ask a model to pick from the same cards, one call per query.
+    """Ask a model to pick from the same cards, ``repeats`` times per query.
 
     The candidate order is a seeded random permutation per query, so a model that always
     picks the first (or last) option scores about ``1/N`` rather than a figure set by how
@@ -242,14 +246,24 @@ def model_routing(
     second-to-last -- exactly the positions where primacy and recency bias are strongest.
     The permutation is recorded per query, so a result can be re-read or replayed.
 
+    Each query keeps one card order across its repeats, so the spread between repeats is
+    the model's own sampling noise rather than a change of layout. Each repeat is its own
+    sample (``draw``), so with the response cache on a repeat is not a replay.
+
     Returns:
-        ``selection_rate``, ``n_queries``, ``n_abstained``, ``n_failed``, the per-query pick
-        with the position the gold card was shown at, and the caveat that this bounds
-        harness behaviour from above.
+        ``selection_rate``, ``n_queries``, ``n_calls`` (queries x repeats; ``n_abstained`` and
+        ``n_failed`` count calls), ``n_abstained``, ``n_failed``, the per-query pick
+        with the position the gold card was shown at, the raw reply of every abstention or
+        unparseable pick, and the caveat that this bounds harness behaviour from above.
+        With ``repeats > 1``, ``noise`` holds each query's agreement across its repeats and
+        the run-to-run spread of the selection rate; it is ``None`` for one repeat.
     """
     if not 0 <= gold < len(candidates):
         raise ValueError(f"gold index {gold} is outside 0..{len(candidates) - 1}")
+    if repeats < 1:
+        raise ValueError(f"repeats must be at least 1, not {repeats}")
     picks: list[dict] = []
+    groups: list[list[dict]] = []
     for index_of_query, query in enumerate(queries):
         shuffler = random.Random(f"{seed}:{index_of_query}:{len(candidates)}")
         order = list(range(len(candidates)))
@@ -258,28 +272,36 @@ def model_routing(
             f"{position + 1}. {candidates[index].card}"
             for position, index in enumerate(order)
         )
-        try:
-            reply = client.text(
-                _SELECT_PROMPT.format(query=query, cards=cards),
-                accept=partial(_is_pick_reply, n_cards=len(order)),
-            )
-        except Exception as exc:  # noqa: BLE001 - one failed query must not lose the rest
-            picks.append({"query": query, "picked": None, "error": str(exc)[:200]})
-            continue
-        # A reply that is not a usable pick is the model declining to route, not a
-        # transport failure: it stays in the denominator like an explicit abstention.
-        try:
-            raw = _as_pick_position(parse_json_object(reply).get("pick", 0))
-        except ValueError:
-            raw = 0
-        if raw < 1 or raw > len(order):
-            picks.append({"query": query, "picked": None, "abstained": True})
-            continue
-        chosen = order[raw - 1]
-        picks.append({"query": query, "picked": candidates[chosen].name,
-                      "correct": chosen == gold, "picked_at": raw,
-                      "gold_shown_at": order.index(gold) + 1,
-                      "order": [candidates[i].name for i in order]})
+        prompt = _SELECT_PROMPT.format(query=query, cards=cards)
+        group: list[dict] = []
+        for repeat in range(repeats):
+            base = {"query": query, "repeat": repeat}
+            try:
+                reply = client.text(
+                    prompt, accept=partial(_is_pick_reply, n_cards=len(order)), draw=repeat,
+                )
+            except Exception as exc:  # noqa: BLE001 - one failed query must not lose the rest
+                row = {**base, "picked": None, "error": str(exc)[:200]}
+            else:
+                # A reply that is not a usable pick is the model declining to route, not a
+                # transport failure: it stays in the denominator like an explicit abstention,
+                # and its raw text is kept so the decline can be read rather than inferred.
+                try:
+                    raw = _as_pick_position(parse_json_object(reply).get("pick", 0))
+                except ValueError:
+                    raw = 0
+                if raw < 1 or raw > len(order):
+                    row = {**base, "picked": None, "abstained": True,
+                           "raw_reply": reply[:_RAW_REPLY_CHARS]}
+                else:
+                    chosen = order[raw - 1]
+                    row = {**base, "picked": candidates[chosen].name,
+                           "correct": chosen == gold, "picked_at": raw,
+                           "gold_shown_at": order.index(gold) + 1,
+                           "order": [candidates[i].name for i in order]}
+            picks.append(row)
+            group.append(row)
+        groups.append(group)
 
     answered = [p for p in picks if "correct" in p]
     hits = sum(1 for p in answered if p["correct"])
@@ -289,6 +311,8 @@ def model_routing(
         "model": client.model,
         "n_candidates": len(candidates),
         "n_queries": len(queries),
+        "repeats": repeats,
+        "n_calls": len(picks),
         "n_abstained": sum(1 for p in picks if p.get("abstained")),
         "n_failed": sum(1 for p in picks if p.get("error")),
         "seed": seed,
@@ -300,6 +324,7 @@ def model_routing(
         "selection_rate_among_answered": (
             round(hits / len(answered), 4) if answered else None
         ),
+        "noise": _noise(groups) if repeats > 1 else None,
         "per_query": picks,
         "note": (
             "An upper bound on harness behaviour, not a harness measurement: a harness "
@@ -309,6 +334,58 @@ def model_routing(
     }
 
 
+def _noise(groups: list[list[dict]]) -> dict:
+    """Agreement across the repeats of each query, and the run-to-run spread of the rate.
+
+    A query's agreement is the share of its answered repeats that gave its most common
+    pick; an abstention counts as a pick. ``selection_rate_by_repeat`` reads each repeat as
+    its own run over the same queries, and ``half_range`` is half the gap between the
+    highest and the lowest of those rates. It is a crude sampling noise floor: a few repeats
+    estimate it coarsely, and it says nothing about a change of cards or queries. The
+    per-repeat rates cover only the queries with no failed call in any repeat
+    (``n_queries_compared``), so a transient error does not read as noise.
+
+    Args:
+        groups: One list per query, holding that query's rows in repeat order.
+    """
+    per_query: list[dict] = []
+    for group in groups:
+        labels = [
+            "(abstained)" if row.get("abstained") else row["picked"]
+            for row in group if not row.get("error")
+        ]
+        modal, agreement = None, None
+        if labels:
+            modal, count = Counter(labels).most_common(1)[0]
+            agreement = round(count / len(labels), 4)
+        per_query.append({
+            "query": group[0]["query"],
+            "modal_pick": modal,
+            "agreement": agreement,
+            "picks": labels,
+            "n_correct": sum(1 for row in group if row.get("correct")),
+        })
+    repeats = len(groups[0]) if groups else 0
+    rates: list[float | None] = []
+    # Every repeat's rate is taken over the same queries: those with no failed call in any
+    # repeat. Dividing each repeat by its own non-errored rows would compare different
+    # denominators and report a transient provider error as sampling noise.
+    complete = [group for group in groups if not any(row.get("error") for row in group)]
+    for repeat in range(repeats):
+        rates.append(round(sum(1 for g in complete if g[repeat].get("correct")) / len(complete), 4)
+                     if complete else None)
+    known = [rate for rate in rates if rate is not None]
+    agreements = [row["agreement"] for row in per_query if row["agreement"] is not None]
+    return {
+        "mean_agreement": round(sum(agreements) / len(agreements), 4) if agreements else None,
+        "unstable_queries": sum(1 for a in agreements if a < 1),
+        "selection_rate_by_repeat": rates,
+        "n_queries_compared": len(complete),
+        "half_range": round((max(known) - min(known)) / 2, 4) if len(known) > 1 else None,
+        "per_query": per_query,
+    }
+
+
 # Only spaces indent a fence (a tab is four columns, so it makes indented code), and a
 # backtick opener may not carry a backtick in its info string; both are CommonMark rules.
 _FENCE = re.compile(r" {0,3}(`{3,}(?!.*`)|~{3,})")

dokimi/scripts/growth_baseline.json

@@ -1,6 +1,6 @@
 {
   "files": {
-    "dokimi/commands/novelty.py": {"lines": 864, "rationale": "W3 adds routing no_queries handling, W10 a --catalog help line; file split tracked separately to avoid conflicts across the W-series"},
+    "dokimi/commands/novelty.py": {"lines": 885, "rationale": "W3 adds routing no_queries handling, W8 a routing flag, W10 a --catalog help line; file split tracked separately to avoid conflicts across the W-series"},
     "dokimi/harnesses/benchflow.py": {"lines": 869},
     "dokimi/harnesses/docker_solver.py": {"lines": 927, "rationale": "pre-existing when the gate was introduced"},
     "dokimi/llm.py": {"lines": 1118, "rationale": "grew on main in #180 (replay a stored retry) after #93 seeded this baseline; split it in its own PR"},
@@ -24,7 +24,7 @@
     "dokimi/commands/catalog.py::cmd_catalog_trigger_set": {"complexity": 18},
     "dokimi/commands/evaluate.py::cmd_eval_judge_validity": {"complexity": 29, "rationale": "grew on main in #73 (judge-validity --context stored) from 25 to 29 before the ratchet landed; split it in its own PR"},
     "dokimi/commands/judge_reliability.py::cmd_eval_cross_judge": {"complexity": 32, "rationale": "pre-existing when the gate was introduced"},
-    "dokimi/commands/novelty.py::cmd_novelty_routing": {"complexity": 12},
+    "dokimi/commands/novelty.py::cmd_novelty_routing": {"complexity": 13, "rationale": "W8 adds the --repeats guard after main's no_queries handling raised the file; the command is split with the rest of novelty.py in a separate PR"},
     "dokimi/commands/repo.py::cmd_repo_scan": {"complexity": 13},
     "dokimi/commands/repo.py::cmd_repo_validate": {"complexity": 11, "rationale": "grew on main in #150 (repo validate --fail-on-warning) from 10 to 11 before the ratchet landed; split it in its own PR"},
     "dokimi/commands/repo_run.py::_context": {"complexity": 11, "rationale": "went over the cap on main in #120 (built-in review graders) after #93 seeded this baseline; split it in its own PR"},