feat(context, repo): consistency of repeated runs; pass^k and path stability in repo compare

dokimi/benchmarks/spec.v1.yaml

@@ -76,11 +76,11 @@ future:
 # on the include/exclude globs, not a target: min_docs catches a glob that
 # matched almost nothing, max_docs catches one that escaped into the tree (8
 # markdown files live outside docs/). Measured selection at the time of
-# writing: 31 files. Adding a doc past the ceiling is meant to be a deliberate
+# writing: 32 files. Adding a doc past the ceiling is meant to be a deliberate
 # edit here, so raise max_docs in the same commit that adds the doc.
 golden_manifest:
   min_docs: 10
-  max_docs: 31
+  max_docs: 32
   include:
     - docs/**/*.md
   exclude:

dokimi/changelog.d/20261022_120000_napetrov_consistency-review-fixes.md

@@ -0,0 +1,6 @@
+### Fixed
+
+- `context consistency`: shell control syntax (`for`/`do`/`done`, `if`, subshells, comments) is no longer a step; a shell edit matches an `edit` action, and a command whose output is saved to a file (`pytest > test.log`) does not; an empty arm is "no runs"; malformed `--outcomes`/`--contract` files end in a clean error; an outcome key shared by two traces is refused; a trace given twice to one arm (a repeated `--arm`, overlapping globs) counts once.
+- `repo compare --path-stability`: an empty trace is unmeasured, not a steady path, and a deeply nested line no longer aborts the comparison.
+- `context consistency`: a Bash line is read with quotes, backslashes, `$(...)`, backticks, `&` and `case` kept right, and `timeout`, `nohup`, `nice`, `sudo`, `env` and `VAR=1` are not the program, so `find -exec ... \;` and `timeout 300 pytest` are one step each; a step's action and `command` text lose the `for`/`do`/`if` words; a shell edit behind a prefix (`sudo sed -i`) is an edit; `pytest; echo done` and `pytest || true` hide the check's status like a pipe; a run with tool calls and no `result` event counts as failed; `tool_calls_*` count calls, not steps; a scalar or nested rule argument or `tool` is a clean contract error.
+- `repo compare --pass-k` / `--path-stability`: tasks the measure does not apply to (`pass^K` of a score task) are not listed as unpaired, and tasks with too few cells are one line with a count.

dokimi/changelog.d/20261022_120000_napetrov_run-consistency.md

@@ -0,0 +1,4 @@
+### Added
+
+- **Run consistency** — `dokimi context consistency` reads repeated Claude Code `stream-json` traces of one task and reports three things apart: result stability (`success_rate`, `pass@k`, `pass^k`), path stability (`mix_jsd`, `order_distance`, `action_distance`, spread of calls, turns and tokens, grouped into strategies), and conformance to an optional contract of named actions and rules (`requires`, `forbids`, `before`, `after_last`, `last_ok`). A `Bash` call is one step per command it chains, a shell write carries its file, and a blocked Stop hook is a failed step. A failed call fails every command it chains, and a check piped without `pipefail` does not pass `last_ok`. It calls no model. See `docs/run-consistency.md`.
+- **pass^k and path stability in `repo compare`** — `--pass-k K` adds `pass^K` per task (the chance K of its cells all pass the gate), and `--path-stability` adds `path_distance` (how much the tool paths of a task's cells differ, read from `events.jsonl`). Both are descriptive and outside Holm's correction.

dokimi/docs/README.md

@@ -258,6 +258,9 @@ code.
 - [context-window.md](context-window.md) — read a run's context window from
   its Claude Code trace with `cli.py context trace`: start-up load, growth,
   compactions, tool-result weight. No LLM call.
+- [run-consistency.md](run-consistency.md) — repeated runs of one task with
+  `cli.py context consistency`: result stability (pass^k), path stability
+  (tool mix, order, actions, strategies), process-contract conformance.
 
 ## Content novelty
 

dokimi/docs/architecture.md

@@ -315,6 +315,11 @@ tokens, one call per message id), the compaction events, tool-result sizes and
 failures per tool, and repeated tool calls. `python cli.py context trace`
 prints one or more traces as a comparison table; it calls no model. Guide:
 [context-window.md](context-window.md).
+`dokimi/context/consistency.py` reads repeated runs of one task into three
+separate profiles: result (`pass@k`, `pass^k`), path (tool-mix JSD, edit
+distance of tool and action sequences, resource spread, strategy groups) and
+conformance to a YAML process contract; `python cli.py context consistency`.
+Guide: [run-consistency.md](run-consistency.md).
 
 ## Shared building blocks
 

dokimi/docs/repo-evaluation-contract.md

@@ -1032,6 +1032,15 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo
   unpaired with the reason: no cells in one arm, a task that is gated in one
   arm and score-only in the other, or a measure (such as a metric) one arm
   lacks. A metric's `n_baseline`/`n_treatment` count the cells that report it.
+  With repeated cells per task (from `repo run --reps N`), two descriptive measures read the repetitions themselves:
+  `--pass-k K` adds `pass^K`, the chance K of a task's cells all pass its gate
+  (`C(c,K)/C(n,K)`; a task with fewer than K cells has none), so an arm that
+  moves a task from 2/3 to 3/3 is told apart from one that moves the rate
+  without making any task reliable; `--path-stability` adds `path_distance`,
+  the mean order distance of the cells' tool paths from `events.jsonl` (0 same
+  steps in the same order, 1 nothing in common; see
+  [run-consistency.md](run-consistency.md)). A steadier path is not a better
+  one: read it beside the gate.
   The output names the batches behind each selector and each batch's pin
   (`tree`, `eval_dir`, `image`, `graders` from `_eval/batch.json`) with its cells'
   harness versions, and warns when a bare `ARM` pools several batches, when a

dokimi/docs/run-consistency.md

@@ -0,0 +1,142 @@
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Run consistency: result, path, contract
+
+An agent that solved a task once may not solve it again, and an agent that
+solves it every time may get there by a different route each time. Those are
+two questions, and a third sits beside them: did the route keep the rules the
+task sets. `context consistency` reads repeated runs of one task from their
+Claude Code `stream-json` traces and answers each one apart. It calls no model.
+
+```bash
+python cli.py context consistency \
+  --arm no-skill='runs/base/*.jsonl' --arm skill='runs/skill/*.jsonl' \
+  --outcomes verdicts.json --contract contract.yaml
+```
+
+| Axis | Question | Numbers |
+|---|---|---|
+| Result | Does it solve the task every time? | `success_rate`, `pass@k` (any of k), `pass^k` (all of k, τ-bench) |
+| Path | Do the runs take the same steps? | `mix_jsd` (same tools in same amounts), `order_distance` (same order), `action_distance` (same files and commands), CV of tool calls, turns, output tokens |
+| Contract | Did each run keep the required process? | `conformance_rate`, violations per rule, `passed_but_nonconforming` |
+
+The path is the main thread's steps. By default (`--shell split`) one `Bash`
+call is one step per command it chains (`&&`, `||`, `;`, newline outside
+quotes; heredoc bodies and comments dropped), each named by program
+(`Bash:pytest`, `Bash:sed`), and a command that writes a file (`sed -i`, `>`,
+`>>`, `tee`) carries it as `file_path`. Without this a model that does
+`sed -i … pricing.py; sed -i … CHANGELOG.md; sh finish.sh` in one call shows as
+one step, its path looks perfectly stable and a contract sees no edit and no
+check. `--shell call` keeps one step per tool call. Subagent calls are left
+out; a `Task` call is one step.
+
+A failed shell call marks every command it chains as failed: the trace does not
+say which one failed, and a failed `make check && git diff` must not read as a
+passing `git diff`. A command piped into another (`pytest | tail`) without
+`set -o pipefail` has its status hidden: the call's status is the last stage's.
+A command followed by another after `;`, a newline, `||` or `&` is hidden the
+same way (`pytest; echo done` reports `echo`'s status), unless `set -e` makes `;`
+and a newline stop the chain; `&&` does not hide it. `last_ok` does not accept such
+a step as a passing check.
+
+The line is read for what it runs, not parsed as a shell would: quotes,
+backslashes, `$(...)` and backticks are kept whole; `for`/`if`/`while`/`case`
+syntax, `( )` and case labels are peeled off; `sudo`, `env`, `time`, `nohup`,
+`nice -n N`, `timeout N`, `VAR=1`, `uv run` and `python -m` are not the program.
+`tool_calls_*` and `failed_calls_mean` count tool calls, so one chained command is
+one call however many steps it has; the JSON records the `shell` grain.
+
+A Stop hook that blocked the stop is a failed `Hook:Stop` step. A hook that
+passed leaves nothing in the trace, so a check the harness enforces is not
+visible as a step; count it from the harness configuration.
+
+## Where the outcome comes from
+
+Pass `--outcomes`: a JSON object of trace path (or file name, or stem) to
+`true`/`false`, written by a verifier that checked the workspace. Without it a
+run passes when the harness ended cleanly (`result` event, `success`), which
+says the agent stopped, not that the task is done. The report says which source
+it used. A run that made tool calls but wrote no `result` event (it crashed,
+timed out or was cut off) counts as failed without `--outcomes`, so it cannot drop
+out of the pass rate; the table shows `runs` and `judged`. A file name or stem that two traces share (`base/1.jsonl`, `skill/1.jsonl`)
+is refused: key those by full path.
+
+## Strategies
+
+A task with two valid routes would read as unstable if every pair of runs were
+compared. Runs are grouped by the order distance of their tool paths, repeats
+collapsed (`--threshold`, default 0.34), and each group gets a verdict:
+
+| Verdict | Reading |
+|---|---|
+| one strategy, stable success | reproducible process |
+| several strategies, all stable success | valid diversity |
+| one strategy, unstable | the route is right, execution breaks |
+| one strategy, stable failure | a reproducible mistake |
+
+A group is a shape, not a proof the shape is right; the result and the
+contract judge that.
+
+## Contract
+
+Named actions match tool calls by tool (or kind, like `Bash:pytest`) and regexes
+over the call's input. Rules constrain only what matters; any order they do not
+constrain conforms.
+
+```yaml
+actions:
+  guide: {tool: Read, file_path: 'AGENTS\.md$'}
+  edit:  {tool: [Edit, Write, MultiEdit]}
+  test:  {tool: Bash, command: '\b(pytest|make test)\b'}
+  push:  {tool: Bash, command: 'git push'}
+rules:
+  - before: [guide, edit]      # the guide is read before the first edit
+  - after_last: [edit, test]   # a test runs after the last edit
+  - last_ok: test              # the last test did not fail
+  - forbids: push
+  - requires: test
+```
+
+An `edit` action with a write tool (`Edit`, `Write`, `MultiEdit`) also matches a shell
+step that writes a file (`sed -i`, `echo`/`printf`/`cat` into `>`, `tee`), so an edit
+through the shell counts. Another program's output saved to a file (`pytest > test.log`,
+`ls > /tmp/files.txt`) is not an edit.
+
+Name every equivalent way to do a step: `make check`, `sh tools/check.sh` and
+the skill's `finish.sh` are one check. A contract that knows only `make check`
+reports agents that ran the script behind it as never checking.
+
+`passed_but_nonconforming` counts runs the verifier passed that broke a rule:
+what a result-only score hides, such as a change that happened to be right but
+was never tested.
+
+## Designing the experiment
+
+Keep the sources of variation apart; one number over all of them explains
+nothing.
+
+| Series | What changes | What it measures |
+|---|---|---|
+| Repeat | nothing; the workspace reset each run | the agent's own variance |
+| Paraphrase | the wording, not the task | robustness to the prompt |
+| Fault | injected tool errors, timeouts | recovery |
+
+A fault measures recovery only in runs that reach the faulted step. Put it on a
+step the procedure requires and report how many runs reached it; an arm whose
+agents never run the check is not robust to a check failure, it is untested.
+
+Compare arms (no skill, prose skill, skill with scripts, skill with executable
+checks) within one series. The useful finding is which uncertainty a skill
+removed: the tool choice (`mix_jsd`), the order (`order_distance`), the targets
+(`action_distance`), the failure handling (`last_ok`, failed calls) or the
+final check (`after_last`).
+
+Background: *Towards a Science of AI Agent Reliability*
+(arXiv:2602.16666) for the outcome / trajectory-distribution /
+trajectory-sequence / resource split; *Consistency as a Testable Property*
+(arXiv:2605.10516) for strategy grouping; Yao et al., τ-bench
+(arXiv:2406.12045) for `pass^k`; process-mining conformance checking for the
+contract.

dokimi/dokimi/commands/context.py

@@ -7,6 +7,7 @@
 
 import argparse
 import json
+import os
 import sys
 from pathlib import Path
 
@@ -74,6 +75,123 @@ def _table(rows: list[dict], window: int | None) -> str:
     return "\n".join(lines)
 
 
+def _trace_arms(args: argparse.Namespace) -> dict[str, list[str]]:
+    """``label -> trace paths`` from ``--arm LABEL=GLOB`` and the positional traces."""
+    import glob
+
+    arms: dict[str, list[str]] = {}
+    for spec in args.arm or []:
+        label, sep, pattern = spec.partition("=")
+        if not sep or not label:
+            sys.exit(f"--arm wants LABEL=GLOB, got {spec!r}")
+        arms.setdefault(label, []).extend(sorted(glob.glob(pattern, recursive=True)) or [pattern])
+    if args.traces:
+        arms.setdefault("runs", []).extend(args.traces)
+    if not arms:
+        sys.exit("give traces, or --arm LABEL=GLOB")
+    return arms
+
+
+def _read_arms(arms: dict[str, list[str]], shell: str) -> tuple[dict[str, list], bool]:
+    """The runs of each arm, and whether some trace could not be read (reported on stderr)."""
+    from dokimi.context import TraceError
+    from dokimi.context.consistency import read_run
+
+    loaded: dict[str, list] = {}
+    failed = False
+    for label, names in arms.items():
+        loaded[label] = []
+        for name in _unique_traces(label, names):
+            try:
+                loaded[label].append(read_run(Path(name), shell))
+            except TraceError as exc:
+                print(f"✗ {exc}", file=sys.stderr)
+                failed = True
+    return loaded, failed
+
+
+def cmd_context_consistency(args: argparse.Namespace) -> None:
+    """Result stability, path stability and contract conformance of repeated runs of one task."""
+    from dokimi.context.consistency import apply_outcomes, load_contract, load_outcomes, profile
+
+    arms = _trace_arms(args)
+    try:
+        contract = load_contract(Path(args.contract)) if args.contract else None
+        outcomes = load_outcomes(Path(args.outcomes)) if args.outcomes else None
+    except (OSError, ValueError) as exc:
+        sys.exit(f"✗ {exc}")
+
+    loaded, failed = _read_arms(arms, args.shell)
+    missing: set[str] = set()
+    if outcomes is not None:  # over every arm at once: a key that two arms share is caught
+        try:
+            missing = set(apply_outcomes([r for runs in loaded.values() for r in runs], outcomes))
+        except ValueError as exc:
+            sys.exit(f"✗ {exc}")
+
+    report = {}
+    for label, runs in loaded.items():
+        for run in runs:
+            if run.name in missing:
+                print(f"warning: {label}: no outcome for {run.name}; left unjudged", file=sys.stderr)
+        if len(runs) < 2:
+            print(f"warning: {label}: {len(runs)} run(s); path stability needs at least 2", file=sys.stderr)
+        report[label] = profile(runs, contract, args.threshold)
+        report[label]["outcome_source"] = "outcomes file" if outcomes is not None else "harness result event"
+        report[label]["shell"] = args.shell  # the grain the step counts were taken at
+
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        print(_consistency_text(report))
+    if failed:
+        sys.exit(1)
+
+
+def _unique_traces(label: str, names: list[str]) -> list[str]:
+    """Each trace once: a repeated ``--arm`` or overlapping globs must not count one run twice."""
+    seen: dict[str, str] = {}
+    for name in names:
+        if os.path.abspath(name) in seen:
+            print(f"warning: {label}: {name} given twice; counted once", file=sys.stderr)
+        seen.setdefault(os.path.abspath(name), name)
+    return list(seen.values())
+
+
+def _fmt(v, pct=False) -> str:
+    if v is None:
+        return "–"
+    return f"{v:.0%}" if pct else f"{v:.2f}"
+
+
+def _consistency_text(report: dict) -> str:
+    cols = ["runs", "judged", "success", "pass^k", "mix JSD", "order dist", "action dist", "calls CV",
+            "strategies", "conform"]
+    lines = ["| arm | " + " | ".join(cols) + " |", "|" + "---|" * (len(cols) + 1)]
+    for label, p in report.items():
+        r, path = p["result"], p["path"]
+        hats = [k for k in r if k.startswith("pass^")]
+        top = hats[-1] if hats else None
+        conform = p.get("contract", {}).get("conformance_rate")
+        lines.append("| " + " | ".join([
+            label, str(r["runs"]), str(r["judged"]), _fmt(r["success_rate"], True),
+            f"{_fmt(r[top], True)} ({top})" if top else "–",
+            _fmt(path["mix_jsd"]), _fmt(path["order_distance"]), _fmt(path["action_distance"]),
+            _fmt(path["tool_calls_cv"]), str(len(p["strategies"])), _fmt(conform, True)]) + " |")
+    for label, p in report.items():
+        lines.append(f"\n{label}: {p['verdict']}  (outcome from {p['outcome_source']})")
+        for i, g in enumerate(p["strategies"], 1):
+            shown = " → ".join(g["path"][:12]) + (" …" if len(g["path"]) > 12 else "")
+            lines.append(f"  S{i} ×{len(g['runs'])} {g['verdict']} ({g['passed']}/{g['judged']}): {shown}")
+        c = p.get("contract")
+        if c:
+            for rule, count in c["violations"].items():
+                lines.append(f"  ✗ {rule}: {count}/{p['result']['runs']} runs")
+            if c["passed_but_nonconforming"]:
+                lines.append(f"  ! {c['passed_but_nonconforming']} passing run(s) broke the contract")
+    return "\n".join(lines)
+
+
 def register(sub, positive_int) -> None:
     """Add the `context` subcommand group."""
     context_p = sub.add_parser(
@@ -96,3 +214,26 @@ def register(sub, positive_int) -> None:
                          help="Also give every call's token counts and every compaction event")
     trace_p.add_argument("--json", action="store_true", help="Print the summaries as JSON")
     trace_p.set_defaults(func=cmd_context_trace)
+
+    cons_p = context_sub.add_parser(
+        "consistency",
+        help="Repeated runs of one task, read from their stream-json traces: result stability "
+        "(success, pass@k, pass^k), path stability (tool mix JSD, order and action edit distance, "
+        "resource spread), strategy groups, and conformance to an optional process contract.",
+    )
+    cons_p.add_argument("traces", nargs="*", help="Traces of one arm (or use --arm)")
+    cons_p.add_argument("--arm", action="append", metavar="LABEL=GLOB",
+                        help="An arm to compare, e.g. no-skill='runs/base/*.jsonl'; repeatable")
+    cons_p.add_argument("--outcomes", metavar="JSON",
+                        help="Verifier verdicts, trace -> true/false; without it a run passes "
+                        "when the harness ended cleanly, which is not task success")
+    cons_p.add_argument("--contract", metavar="YAML",
+                        help="Process contract: named actions and rules (requires, forbids, "
+                        "before, after_last, last_ok)")
+    cons_p.add_argument("--threshold", type=float, default=0.34,
+                        help="Max order distance of collapsed paths within one strategy (default 0.34)")
+    cons_p.add_argument("--shell", choices=("split", "call"), default="split",
+                        help="split (default): one step per chained shell command, shell writes "
+                        "count as edits; call: one step per tool call")
+    cons_p.add_argument("--json", action="store_true", help="Print the profiles as JSON")
+    cons_p.set_defaults(func=cmd_context_consistency)