diff --git a/dokimi/benchmarks/spec.v1.yaml b/dokimi/benchmarks/spec.v1.yaml index 6b0158ee..41c6e1fe 100644 --- a/dokimi/benchmarks/spec.v1.yaml +++ b/dokimi/benchmarks/spec.v1.yaml @@ -76,11 +76,11 @@ future: # on the include/exclude globs, not a target: min_docs catches a glob that # matched almost nothing, max_docs catches one that escaped into the tree (8 # markdown files live outside docs/). Measured selection at the time of -# writing: 31 files. Adding a doc past the ceiling is meant to be a deliberate +# writing: 32 files. Adding a doc past the ceiling is meant to be a deliberate # edit here, so raise max_docs in the same commit that adds the doc. golden_manifest: min_docs: 10 - max_docs: 31 + max_docs: 32 include: - docs/**/*.md exclude: diff --git a/dokimi/changelog.d/20261022_120000_napetrov_consistency-review-fixes.md b/dokimi/changelog.d/20261022_120000_napetrov_consistency-review-fixes.md new file mode 100644 index 00000000..09c14867 --- /dev/null +++ b/dokimi/changelog.d/20261022_120000_napetrov_consistency-review-fixes.md @@ -0,0 +1,6 @@ +### Fixed + +- `context consistency`: shell control syntax (`for`/`do`/`done`, `if`, subshells, comments) is no longer a step; a shell edit matches an `edit` action, and a command whose output is saved to a file (`pytest > test.log`) does not; an empty arm is "no runs"; malformed `--outcomes`/`--contract` files end in a clean error; an outcome key shared by two traces is refused; a trace given twice to one arm (a repeated `--arm`, overlapping globs) counts once. +- `repo compare --path-stability`: an empty trace is unmeasured, not a steady path, and a deeply nested line no longer aborts the comparison. +- `context consistency`: a Bash line is read with quotes, backslashes, `$(...)`, backticks, `&` and `case` kept right, and `timeout`, `nohup`, `nice`, `sudo`, `env` and `VAR=1` are not the program, so `find -exec ... \;` and `timeout 300 pytest` are one step each; a step's action and `command` text lose the `for`/`do`/`if` words; a shell edit behind a prefix (`sudo sed -i`) is an edit; `pytest; echo done` and `pytest || true` hide the check's status like a pipe; a run with tool calls and no `result` event counts as failed; `tool_calls_*` count calls, not steps; a scalar or nested rule argument or `tool` is a clean contract error. +- `repo compare --pass-k` / `--path-stability`: tasks the measure does not apply to (`pass^K` of a score task) are not listed as unpaired, and tasks with too few cells are one line with a count. diff --git a/dokimi/changelog.d/20261022_120000_napetrov_run-consistency.md b/dokimi/changelog.d/20261022_120000_napetrov_run-consistency.md new file mode 100644 index 00000000..06f5c7b9 --- /dev/null +++ b/dokimi/changelog.d/20261022_120000_napetrov_run-consistency.md @@ -0,0 +1,4 @@ +### Added + +- **Run consistency** — `dokimi context consistency` reads repeated Claude Code `stream-json` traces of one task and reports three things apart: result stability (`success_rate`, `pass@k`, `pass^k`), path stability (`mix_jsd`, `order_distance`, `action_distance`, spread of calls, turns and tokens, grouped into strategies), and conformance to an optional contract of named actions and rules (`requires`, `forbids`, `before`, `after_last`, `last_ok`). A `Bash` call is one step per command it chains, a shell write carries its file, and a blocked Stop hook is a failed step. A failed call fails every command it chains, and a check piped without `pipefail` does not pass `last_ok`. It calls no model. See `docs/run-consistency.md`. +- **pass^k and path stability in `repo compare`** — `--pass-k K` adds `pass^K` per task (the chance K of its cells all pass the gate), and `--path-stability` adds `path_distance` (how much the tool paths of a task's cells differ, read from `events.jsonl`). Both are descriptive and outside Holm's correction. diff --git a/dokimi/docs/README.md b/dokimi/docs/README.md index 32cfc1c0..84c1867d 100644 --- a/dokimi/docs/README.md +++ b/dokimi/docs/README.md @@ -258,6 +258,9 @@ code. - [context-window.md](context-window.md) — read a run's context window from its Claude Code trace with `cli.py context trace`: start-up load, growth, compactions, tool-result weight. No LLM call. +- [run-consistency.md](run-consistency.md) — repeated runs of one task with + `cli.py context consistency`: result stability (pass^k), path stability + (tool mix, order, actions, strategies), process-contract conformance. ## Content novelty diff --git a/dokimi/docs/architecture.md b/dokimi/docs/architecture.md index 76de3508..589dd62f 100644 --- a/dokimi/docs/architecture.md +++ b/dokimi/docs/architecture.md @@ -315,6 +315,11 @@ tokens, one call per message id), the compaction events, tool-result sizes and failures per tool, and repeated tool calls. `python cli.py context trace` prints one or more traces as a comparison table; it calls no model. Guide: [context-window.md](context-window.md). +`dokimi/context/consistency.py` reads repeated runs of one task into three +separate profiles: result (`pass@k`, `pass^k`), path (tool-mix JSD, edit +distance of tool and action sequences, resource spread, strategy groups) and +conformance to a YAML process contract; `python cli.py context consistency`. +Guide: [run-consistency.md](run-consistency.md). ## Shared building blocks diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 00aec556..3680f6f6 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1032,6 +1032,15 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo unpaired with the reason: no cells in one arm, a task that is gated in one arm and score-only in the other, or a measure (such as a metric) one arm lacks. A metric's `n_baseline`/`n_treatment` count the cells that report it. + With repeated cells per task (from `repo run --reps N`), two descriptive measures read the repetitions themselves: + `--pass-k K` adds `pass^K`, the chance K of a task's cells all pass its gate + (`C(c,K)/C(n,K)`; a task with fewer than K cells has none), so an arm that + moves a task from 2/3 to 3/3 is told apart from one that moves the rate + without making any task reliable; `--path-stability` adds `path_distance`, + the mean order distance of the cells' tool paths from `events.jsonl` (0 same + steps in the same order, 1 nothing in common; see + [run-consistency.md](run-consistency.md)). A steadier path is not a better + one: read it beside the gate. The output names the batches behind each selector and each batch's pin (`tree`, `eval_dir`, `image`, `graders` from `_eval/batch.json`) with its cells' harness versions, and warns when a bare `ARM` pools several batches, when a diff --git a/dokimi/docs/run-consistency.md b/dokimi/docs/run-consistency.md new file mode 100644 index 00000000..85d3f2b5 --- /dev/null +++ b/dokimi/docs/run-consistency.md @@ -0,0 +1,142 @@ + + +# Run consistency: result, path, contract + +An agent that solved a task once may not solve it again, and an agent that +solves it every time may get there by a different route each time. Those are +two questions, and a third sits beside them: did the route keep the rules the +task sets. `context consistency` reads repeated runs of one task from their +Claude Code `stream-json` traces and answers each one apart. It calls no model. + +```bash +python cli.py context consistency \ + --arm no-skill='runs/base/*.jsonl' --arm skill='runs/skill/*.jsonl' \ + --outcomes verdicts.json --contract contract.yaml +``` + +| Axis | Question | Numbers | +|---|---|---| +| Result | Does it solve the task every time? | `success_rate`, `pass@k` (any of k), `pass^k` (all of k, τ-bench) | +| Path | Do the runs take the same steps? | `mix_jsd` (same tools in same amounts), `order_distance` (same order), `action_distance` (same files and commands), CV of tool calls, turns, output tokens | +| Contract | Did each run keep the required process? | `conformance_rate`, violations per rule, `passed_but_nonconforming` | + +The path is the main thread's steps. By default (`--shell split`) one `Bash` +call is one step per command it chains (`&&`, `||`, `;`, newline outside +quotes; heredoc bodies and comments dropped), each named by program +(`Bash:pytest`, `Bash:sed`), and a command that writes a file (`sed -i`, `>`, +`>>`, `tee`) carries it as `file_path`. Without this a model that does +`sed -i … pricing.py; sed -i … CHANGELOG.md; sh finish.sh` in one call shows as +one step, its path looks perfectly stable and a contract sees no edit and no +check. `--shell call` keeps one step per tool call. Subagent calls are left +out; a `Task` call is one step. + +A failed shell call marks every command it chains as failed: the trace does not +say which one failed, and a failed `make check && git diff` must not read as a +passing `git diff`. A command piped into another (`pytest | tail`) without +`set -o pipefail` has its status hidden: the call's status is the last stage's. +A command followed by another after `;`, a newline, `||` or `&` is hidden the +same way (`pytest; echo done` reports `echo`'s status), unless `set -e` makes `;` +and a newline stop the chain; `&&` does not hide it. `last_ok` does not accept such +a step as a passing check. + +The line is read for what it runs, not parsed as a shell would: quotes, +backslashes, `$(...)` and backticks are kept whole; `for`/`if`/`while`/`case` +syntax, `( )` and case labels are peeled off; `sudo`, `env`, `time`, `nohup`, +`nice -n N`, `timeout N`, `VAR=1`, `uv run` and `python -m` are not the program. +`tool_calls_*` and `failed_calls_mean` count tool calls, so one chained command is +one call however many steps it has; the JSON records the `shell` grain. + +A Stop hook that blocked the stop is a failed `Hook:Stop` step. A hook that +passed leaves nothing in the trace, so a check the harness enforces is not +visible as a step; count it from the harness configuration. + +## Where the outcome comes from + +Pass `--outcomes`: a JSON object of trace path (or file name, or stem) to +`true`/`false`, written by a verifier that checked the workspace. Without it a +run passes when the harness ended cleanly (`result` event, `success`), which +says the agent stopped, not that the task is done. The report says which source +it used. A run that made tool calls but wrote no `result` event (it crashed, +timed out or was cut off) counts as failed without `--outcomes`, so it cannot drop +out of the pass rate; the table shows `runs` and `judged`. A file name or stem that two traces share (`base/1.jsonl`, `skill/1.jsonl`) +is refused: key those by full path. + +## Strategies + +A task with two valid routes would read as unstable if every pair of runs were +compared. Runs are grouped by the order distance of their tool paths, repeats +collapsed (`--threshold`, default 0.34), and each group gets a verdict: + +| Verdict | Reading | +|---|---| +| one strategy, stable success | reproducible process | +| several strategies, all stable success | valid diversity | +| one strategy, unstable | the route is right, execution breaks | +| one strategy, stable failure | a reproducible mistake | + +A group is a shape, not a proof the shape is right; the result and the +contract judge that. + +## Contract + +Named actions match tool calls by tool (or kind, like `Bash:pytest`) and regexes +over the call's input. Rules constrain only what matters; any order they do not +constrain conforms. + +```yaml +actions: + guide: {tool: Read, file_path: 'AGENTS\.md$'} + edit: {tool: [Edit, Write, MultiEdit]} + test: {tool: Bash, command: '\b(pytest|make test)\b'} + push: {tool: Bash, command: 'git push'} +rules: + - before: [guide, edit] # the guide is read before the first edit + - after_last: [edit, test] # a test runs after the last edit + - last_ok: test # the last test did not fail + - forbids: push + - requires: test +``` + +An `edit` action with a write tool (`Edit`, `Write`, `MultiEdit`) also matches a shell +step that writes a file (`sed -i`, `echo`/`printf`/`cat` into `>`, `tee`), so an edit +through the shell counts. Another program's output saved to a file (`pytest > test.log`, +`ls > /tmp/files.txt`) is not an edit. + +Name every equivalent way to do a step: `make check`, `sh tools/check.sh` and +the skill's `finish.sh` are one check. A contract that knows only `make check` +reports agents that ran the script behind it as never checking. + +`passed_but_nonconforming` counts runs the verifier passed that broke a rule: +what a result-only score hides, such as a change that happened to be right but +was never tested. + +## Designing the experiment + +Keep the sources of variation apart; one number over all of them explains +nothing. + +| Series | What changes | What it measures | +|---|---|---| +| Repeat | nothing; the workspace reset each run | the agent's own variance | +| Paraphrase | the wording, not the task | robustness to the prompt | +| Fault | injected tool errors, timeouts | recovery | + +A fault measures recovery only in runs that reach the faulted step. Put it on a +step the procedure requires and report how many runs reached it; an arm whose +agents never run the check is not robust to a check failure, it is untested. + +Compare arms (no skill, prose skill, skill with scripts, skill with executable +checks) within one series. The useful finding is which uncertainty a skill +removed: the tool choice (`mix_jsd`), the order (`order_distance`), the targets +(`action_distance`), the failure handling (`last_ok`, failed calls) or the +final check (`after_last`). + +Background: *Towards a Science of AI Agent Reliability* +(arXiv:2602.16666) for the outcome / trajectory-distribution / +trajectory-sequence / resource split; *Consistency as a Testable Property* +(arXiv:2605.10516) for strategy grouping; Yao et al., τ-bench +(arXiv:2406.12045) for `pass^k`; process-mining conformance checking for the +contract. diff --git a/dokimi/dokimi/commands/context.py b/dokimi/dokimi/commands/context.py index 073f5527..fba02c21 100644 --- a/dokimi/dokimi/commands/context.py +++ b/dokimi/dokimi/commands/context.py @@ -7,6 +7,7 @@ import argparse import json +import os import sys from pathlib import Path @@ -74,6 +75,123 @@ def _table(rows: list[dict], window: int | None) -> str: return "\n".join(lines) +def _trace_arms(args: argparse.Namespace) -> dict[str, list[str]]: + """``label -> trace paths`` from ``--arm LABEL=GLOB`` and the positional traces.""" + import glob + + arms: dict[str, list[str]] = {} + for spec in args.arm or []: + label, sep, pattern = spec.partition("=") + if not sep or not label: + sys.exit(f"--arm wants LABEL=GLOB, got {spec!r}") + arms.setdefault(label, []).extend(sorted(glob.glob(pattern, recursive=True)) or [pattern]) + if args.traces: + arms.setdefault("runs", []).extend(args.traces) + if not arms: + sys.exit("give traces, or --arm LABEL=GLOB") + return arms + + +def _read_arms(arms: dict[str, list[str]], shell: str) -> tuple[dict[str, list], bool]: + """The runs of each arm, and whether some trace could not be read (reported on stderr).""" + from dokimi.context import TraceError + from dokimi.context.consistency import read_run + + loaded: dict[str, list] = {} + failed = False + for label, names in arms.items(): + loaded[label] = [] + for name in _unique_traces(label, names): + try: + loaded[label].append(read_run(Path(name), shell)) + except TraceError as exc: + print(f"✗ {exc}", file=sys.stderr) + failed = True + return loaded, failed + + +def cmd_context_consistency(args: argparse.Namespace) -> None: + """Result stability, path stability and contract conformance of repeated runs of one task.""" + from dokimi.context.consistency import apply_outcomes, load_contract, load_outcomes, profile + + arms = _trace_arms(args) + try: + contract = load_contract(Path(args.contract)) if args.contract else None + outcomes = load_outcomes(Path(args.outcomes)) if args.outcomes else None + except (OSError, ValueError) as exc: + sys.exit(f"✗ {exc}") + + loaded, failed = _read_arms(arms, args.shell) + missing: set[str] = set() + if outcomes is not None: # over every arm at once: a key that two arms share is caught + try: + missing = set(apply_outcomes([r for runs in loaded.values() for r in runs], outcomes)) + except ValueError as exc: + sys.exit(f"✗ {exc}") + + report = {} + for label, runs in loaded.items(): + for run in runs: + if run.name in missing: + print(f"warning: {label}: no outcome for {run.name}; left unjudged", file=sys.stderr) + if len(runs) < 2: + print(f"warning: {label}: {len(runs)} run(s); path stability needs at least 2", file=sys.stderr) + report[label] = profile(runs, contract, args.threshold) + report[label]["outcome_source"] = "outcomes file" if outcomes is not None else "harness result event" + report[label]["shell"] = args.shell # the grain the step counts were taken at + + if args.json: + print(json.dumps(report, indent=2)) + else: + print(_consistency_text(report)) + if failed: + sys.exit(1) + + +def _unique_traces(label: str, names: list[str]) -> list[str]: + """Each trace once: a repeated ``--arm`` or overlapping globs must not count one run twice.""" + seen: dict[str, str] = {} + for name in names: + if os.path.abspath(name) in seen: + print(f"warning: {label}: {name} given twice; counted once", file=sys.stderr) + seen.setdefault(os.path.abspath(name), name) + return list(seen.values()) + + +def _fmt(v, pct=False) -> str: + if v is None: + return "–" + return f"{v:.0%}" if pct else f"{v:.2f}" + + +def _consistency_text(report: dict) -> str: + cols = ["runs", "judged", "success", "pass^k", "mix JSD", "order dist", "action dist", "calls CV", + "strategies", "conform"] + lines = ["| arm | " + " | ".join(cols) + " |", "|" + "---|" * (len(cols) + 1)] + for label, p in report.items(): + r, path = p["result"], p["path"] + hats = [k for k in r if k.startswith("pass^")] + top = hats[-1] if hats else None + conform = p.get("contract", {}).get("conformance_rate") + lines.append("| " + " | ".join([ + label, str(r["runs"]), str(r["judged"]), _fmt(r["success_rate"], True), + f"{_fmt(r[top], True)} ({top})" if top else "–", + _fmt(path["mix_jsd"]), _fmt(path["order_distance"]), _fmt(path["action_distance"]), + _fmt(path["tool_calls_cv"]), str(len(p["strategies"])), _fmt(conform, True)]) + " |") + for label, p in report.items(): + lines.append(f"\n{label}: {p['verdict']} (outcome from {p['outcome_source']})") + for i, g in enumerate(p["strategies"], 1): + shown = " → ".join(g["path"][:12]) + (" …" if len(g["path"]) > 12 else "") + lines.append(f" S{i} ×{len(g['runs'])} {g['verdict']} ({g['passed']}/{g['judged']}): {shown}") + c = p.get("contract") + if c: + for rule, count in c["violations"].items(): + lines.append(f" ✗ {rule}: {count}/{p['result']['runs']} runs") + if c["passed_but_nonconforming"]: + lines.append(f" ! {c['passed_but_nonconforming']} passing run(s) broke the contract") + return "\n".join(lines) + + def register(sub, positive_int) -> None: """Add the `context` subcommand group.""" context_p = sub.add_parser( @@ -96,3 +214,26 @@ def register(sub, positive_int) -> None: help="Also give every call's token counts and every compaction event") trace_p.add_argument("--json", action="store_true", help="Print the summaries as JSON") trace_p.set_defaults(func=cmd_context_trace) + + cons_p = context_sub.add_parser( + "consistency", + help="Repeated runs of one task, read from their stream-json traces: result stability " + "(success, pass@k, pass^k), path stability (tool mix JSD, order and action edit distance, " + "resource spread), strategy groups, and conformance to an optional process contract.", + ) + cons_p.add_argument("traces", nargs="*", help="Traces of one arm (or use --arm)") + cons_p.add_argument("--arm", action="append", metavar="LABEL=GLOB", + help="An arm to compare, e.g. no-skill='runs/base/*.jsonl'; repeatable") + cons_p.add_argument("--outcomes", metavar="JSON", + help="Verifier verdicts, trace -> true/false; without it a run passes " + "when the harness ended cleanly, which is not task success") + cons_p.add_argument("--contract", metavar="YAML", + help="Process contract: named actions and rules (requires, forbids, " + "before, after_last, last_ok)") + cons_p.add_argument("--threshold", type=float, default=0.34, + help="Max order distance of collapsed paths within one strategy (default 0.34)") + cons_p.add_argument("--shell", choices=("split", "call"), default="split", + help="split (default): one step per chained shell command, shell writes " + "count as edits; call: one step per tool call") + cons_p.add_argument("--json", action="store_true", help="Print the profiles as JSON") + cons_p.set_defaults(func=cmd_context_consistency) diff --git a/dokimi/dokimi/commands/repo_run.py b/dokimi/dokimi/commands/repo_run.py index ef165f37..4b8136f4 100644 --- a/dokimi/dokimi/commands/repo_run.py +++ b/dokimi/dokimi/commands/repo_run.py @@ -297,7 +297,8 @@ def cmd_repo_compare(args: argparse.Namespace) -> None: sys.exit(1) arms = [a for spec in args.arm for a in spec.split(",") if a] if args.arm else None try: - summary = compare(dirs, args.baseline, arms, args.model, tuple(args.metric or ())) + summary = compare(dirs, args.baseline, arms, args.model, tuple(args.metric or ()), + pass_k=args.pass_k, path_stability=args.path_stability) except CompareError as exc: print(f"✗ {exc}", file=sys.stderr) sys.exit(1) @@ -408,5 +409,9 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None: compare_p.add_argument("--model", action="append", default=None, help="Only this model; repeatable") compare_p.add_argument("--metric", action="append", default=None, help="Also compare this numeric or boolean grade.json metric; repeatable") + compare_p.add_argument("--pass-k", type=positive_int, default=None, dest="pass_k", metavar="K", + help="Also compare pass^K: the chance K cells of a task all pass its gate") + compare_p.add_argument("--path-stability", action="store_true", dest="path_stability", + help="Also compare path_distance: how much the tool paths of a task's cells differ") compare_p.add_argument("--json", action="store_true", help="Print the comparison as JSON") compare_p.set_defaults(func=cmd_repo_compare) diff --git a/dokimi/dokimi/context/consistency.py b/dokimi/dokimi/context/consistency.py new file mode 100644 index 00000000..f8ea91a4 --- /dev/null +++ b/dokimi/dokimi/context/consistency.py @@ -0,0 +1,558 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Repeated runs of one task -> how stable the result was, how stable the path was, and whether +the path kept the rules the task sets. Three questions, three answers; never one score. + +**Result.** A run passes when its outcome says so: an ``--outcomes`` file (trace -> bool, from a +verifier that checked the workspace) or, without one, the harness's own ``result`` event +(``subtype == "success"`` and not ``is_error``), which only says the agent stopped cleanly, not that +the task is done. ``pass@k`` is the chance at least one of ``k`` runs drawn from the ``n`` passes, +``pass^k`` the chance all ``k`` do (τ-bench); both are the unbiased estimates ``1 - C(n-c,k)/C(n,k)`` +and ``C(c,k)/C(n,k)`` over ``c`` passes. + +**Path.** Each run becomes a sequence of main-thread tool calls at two grains: ``kind`` (the tool's +name; ``Bash`` gets its program, so ``Bash:pytest`` and ``Bash:git`` differ) and ``action`` (kind plus +the call's target: a file's name, a search pattern). Over every pair of runs: + +* ``mix_jsd`` – Jensen-Shannon divergence (base 2, 0..1) of the kind counts: the same tools in + different amounts or not; +* ``order_distance`` – Levenshtein distance of the kind sequences over the longer length: the same + steps in the same order or not; +* ``action_distance`` – the same over ``action`` sequences: the same files and commands or not; +* resource spread – coefficient of variation of tool calls, turns and output tokens. + +**Strategies.** A task with several valid ways would read as unstable if every pair were compared. +Runs are grouped, single linkage, by the order distance of their kind sequences with consecutive +repeats collapsed (three ``Read`` in a row are one step), at ``threshold``. Each strategy reports its +runs, passes, inner distance and a representative path, so "two ways, both pass" is told apart from +"one way that breaks" and "one way that always fails". A cluster is a shape, not a proof that the +shape is right; the result and the contract judge that. + +**Contract.** Optional rules over named actions, checked per run (process conformance): an action +must or must not happen, must come first, must follow the last of another, or must not fail at its +last call. The contract allows any order it does not constrain, so different correct paths pass. + +Nothing here calls a model. +""" + +from __future__ import annotations + +import json +import math +import re +from dataclasses import dataclass, field +from itertools import combinations +from pathlib import Path +from typing import Any + +from dokimi.context import shell +from dokimi.context.trace import TraceError + +_WRITE_TOOLS = {"Edit", "Write", "MultiEdit", "NotebookEdit"} +RULES = ("requires", "forbids", "before", "after_last", "last_ok") +_TARGET_KEYS = ("file_path", "notebook_path", "path", "pattern", "url", "query", "subagent_type") + + +@dataclass +class Event: + tool: str + kind: str + action: str + input: dict[str, Any] + failed: bool = False + masked: bool = False # the call's status is another command's: a pipe without pipefail, or a later `;` / `||` + call: int = -1 # the tool call this step came from; a chained command is one call and several steps + + +@dataclass +class Run: + name: str + events: list[Event] = field(default_factory=list) + passed: bool | None = None + num_turns: int | None = None + output_tokens: int = 0 + calls_made: int = 0 # the next call's number, while a trace is read + + def calls(self) -> int: + """Tool calls made: a chained command (``a && b``) is several steps but one call.""" + return len({e.call if e.call >= 0 else id(e) for e in self.events}) + + def failed_calls(self) -> int: + return len({e.call if e.call >= 0 else id(e) for e in self.events if e.failed}) + + def kinds(self) -> list[str]: + return [e.kind for e in self.events] + + def actions(self) -> list[str]: + return [e.action for e in self.events] + + def shape(self) -> list[str]: + """The kind sequence with consecutive repeats collapsed: the strategy, not its length.""" + out: list[str] = [] + for k in self.kinds(): + if not out or out[-1] != k: + out.append(k) + return out + + +# --- reading ----------------------------------------------------------------------------------- + +def read_run(path: Path, grain: str = "split") -> Run: + try: + raw = path.read_text(encoding="utf-8-sig", errors="replace") + except OSError as exc: + raise TraceError(f"cannot read {path}: {exc}") from exc + run = parse_run(raw, str(path), grain) + if not run.events and run.num_turns is None: + raise TraceError(f"{path}: no tool call and no result event; is it a " + "`claude -p --output-format stream-json --verbose` trace?") + return run + + +def parse_run(raw: str, name: str = "", grain: str = "split") -> Run: + """The main thread's steps in order, their failures, and the result event. + + ``grain="split"`` turns one ``Bash`` call into one step per command it chains (``a && b; c``), + and a command that writes a file (``sed -i``, ``>``, ``tee``) carries that file as + ``file_path``, so an edit made through the shell is an edit. ``grain="call"`` keeps one step + per tool call. A Stop hook that blocked the stop is a failed ``Hook:Stop`` step; a hook that + passed leaves nothing in the trace. + + A run that made tool calls but wrote no ``result`` event (it crashed, timed out, was cut off) + did not finish: it counts as failed, so it cannot drop out of the pass rate. + """ + run = Run(name) + by_id: dict[str, list[Event]] = {} + seen_messages: dict[str, int] = {} + for line in raw.split(chr(10)): + try: + rec = json.loads(line) if line.strip() else None + except (ValueError, RecursionError): + continue + if not isinstance(rec, dict): + continue + raw_message = rec.get("message") + message: dict[str, Any] = {} + if isinstance(raw_message, dict): + message = raw_message + raw_content = message.get("content") + content: list[Any] = raw_content if isinstance(raw_content, list) else [] + main = rec.get("parent_tool_use_id") is None + kind = rec.get("type") + if kind == "assistant": + _assistant(run, rec, message, content, main, by_id, seen_messages, grain) + elif kind == "user": + _user(run, content, main, by_id) + elif kind == "result": + run.num_turns = rec.get("num_turns") if isinstance(rec.get("num_turns"), int) else None + run.passed = rec.get("subtype") == "success" and not rec.get("is_error") + run.output_tokens = sum(seen_messages.values()) + if run.passed is None and run.events: + run.passed = False + return run + + +def _assistant(run: Run, rec: dict[str, Any], message: dict[str, Any], content: list[Any], main: bool, + by_id: dict[str, list[Event]], seen_messages: dict[str, int], grain: str) -> None: + raw_usage = message.get("usage") + usage: dict[str, Any] = raw_usage if isinstance(raw_usage, dict) else {} + mid = str(message.get("id") or id(rec)) + seen_messages[mid] = max(seen_messages.get(mid, 0), int(usage.get("output_tokens") or 0)) # one call, many blocks + if not main: + return + for block in content: + if isinstance(block, dict) and block.get("type") == "tool_use": + evs = _events(str(block.get("name") or ""), block.get("input"), grain) + for e in evs: + e.call = run.calls_made + run.calls_made += 1 + run.events.extend(evs) + by_id[str(block.get("id") or "")] = evs + + +def _user(run: Run, content: list[Any], main: bool, by_id: dict[str, list[Event]]) -> None: + if main and any(_text(b).startswith("Stop hook feedback") for b in content): + run.events.append(Event("Hook", "Hook:Stop", "Hook:Stop", {}, failed=True, call=run.calls_made)) + run.calls_made += 1 + for block in content: + if isinstance(block, dict) and block.get("type") == "tool_result" and block.get("is_error"): + # The trace does not say which chained command failed, so all of them may have: + # a failed `make check && git diff` must not read as a passing `git diff`. + for hit in by_id.get(str(block.get("tool_use_id") or ""), []): + hit.failed = True + + +def run_from_events(events: list[dict[str, Any]], name: str = "") -> Run: + """A run from the normalized ``events.jsonl`` of ``repo run`` (any harness), main thread only. + + A command is split as a ``Bash`` call is; a read is ``Read``, an edit ``Edit``, another tool its + name. A command with a non-zero ``rc`` fails every command it chains, as a failed call does. + """ + run = Run(name) + for ev in events: + if not isinstance(ev, dict) or ev.get("agent"): + continue + kind = ev.get("kind") + if kind == "command": + evs = _events("Bash", {"command": str(ev.get("text") or "")}) + for e in evs: + e.failed = isinstance(ev.get("rc"), int) and ev["rc"] != 0 + e.call = len(run.events) + run.events.extend(evs) + elif kind in ("read", "edit"): + tool = "Read" if kind == "read" else "Edit" + path = str(ev.get("path") or "") + run.events.append(Event(tool, tool, f"{tool}({Path(path).name})" if path else tool, + {"file_path": path} if path else {})) + elif kind == "tool" and ev.get("name"): + tool = str(ev["name"]) + run.events.append(Event(tool, tool, tool, {}, failed=bool(ev.get("denied")))) + return run + + +def order_distance(runs: list[Run]) -> float | None: + """Mean order distance of the kind sequences over every pair of runs; ``None`` under two runs.""" + return _mean([seq_distance(a.kinds(), b.kinds()) for a, b in combinations(runs, 2)]) + + +def _text(block: Any) -> str: + return str(block.get("text") or "") if isinstance(block, dict) and block.get("type") == "text" else "" + + +def _events(tool: str, args: Any, grain: str = "split") -> list[Event]: + args = args if isinstance(args, dict) else {} + if tool == "Bash": + return _bash_events(tool, args, grain) + kind, target = tool, "" + for key in _TARGET_KEYS: + if args.get(key): + value = str(args[key]) + target = Path(value).name if key.endswith("path") else value + break + return [Event(tool, kind, f"{kind}({target})" if target else kind, args)] + + +def _hides_status(sep: str, errexit: bool) -> bool: + """Whether the call's status is a later command's, not that of the command before ``sep``; + `set -e` stops a chain at its first failure, so after `;` or a newline it is still that command's.""" + return sep in shell.HIDING and not (errexit and sep in (";", "\n")) + + +def _bash_events(tool: str, args: dict[str, Any], grain: str) -> list[Event]: + command = str(args.get("command") or "") + pipefail = bool(re.search(r"\bpipefail\b", command)) + errexit = bool(re.search(r"\bset\s+(-[a-z]*e|-o\s+errexit)", command)) + parts = shell.split_commands(command) + if grain == "split": + out = [] + for i, (seg, sep) in enumerate(parts): + prog = shell.program(seg) + if not prog: + continue + sep = sep if any(shell.clean(later) for later, _ in parts[i + 1:]) else "" # `done`, `fi` close it + written = shell.written(seg) + text = shell.step_text(seg) + ev_input = {**args, "command": text, **({"file_path": written} if written else {})} + target = Path(written).name if written else text + # `set -e` stops a chain at its first failure, so after `;` or a newline the status is still theirs + hidden = _hides_status(sep, errexit) + out.append(Event(tool, f"Bash:{prog}", f"Bash:{prog}({target})", ev_input, + masked=not pipefail and "|" in shell.tokens(seg) or hidden)) + if out: + return out + prog = shell.program(command) + kind = f"Bash:{prog}" if prog else "Bash" + masked = any((not pipefail and "|" in shell.tokens(seg)) + or (any(shell.clean(later) for later, _ in parts[i + 1:]) and _hides_status(sep, errexit)) + for i, (seg, sep) in enumerate(parts) if shell.program(seg)) + return [Event(tool, kind, f"{kind}({command.strip()})" if command.strip() else kind, args, masked=masked)] + + +def load_outcomes(path: Path) -> dict[str, bool]: + """``{"trace path or file name": true|false}``, or a list of ``{"trace": .., "passed": ..}``.""" + data = json.loads(path.read_text(encoding="utf-8")) + if isinstance(data, list): + for r in data: + if not isinstance(r, dict) or "trace" not in r or "passed" not in r: + raise ValueError(f"{path}: each entry wants `trace` and `passed`, got {r!r}") + data = {str(r["trace"]): r["passed"] for r in data} + if not isinstance(data, dict) or not all(isinstance(v, bool) for v in data.values()): + raise ValueError(f"{path}: want an object of trace -> true/false") + return data + + +def apply_outcomes(runs: list[Run], outcomes: dict[str, bool]) -> list[str]: + """Set each run's result from the verifier; returns the runs it had no verdict for. + + A key is a run's full path, or its file name or stem. A name or stem several runs share + (``base/1.jsonl``, ``skill/1.jsonl``) is ambiguous and refused: give the full path. + """ + names = [Path(r.name).name for r in runs] + stems = [Path(r.name).stem for r in runs] + missing = [] + for run, name, stem in zip(runs, names, stems, strict=True): + if run.name in outcomes: + run.passed = outcomes[run.name] + continue + for key, same in ((name, names), (stem, stems)): + if key in outcomes: + if same.count(key) > 1: + raise ValueError(f"outcome key {key!r} matches {same.count(key)} runs; " + "use the full path of the trace") + run.passed = outcomes[key] + break + else: + run.passed = None + missing.append(run.name) + return missing + + +# --- contract ---------------------------------------------------------------------------------- + +@dataclass +class Matcher: + tools: tuple[str, ...] = () + patterns: dict[str, re.Pattern[str]] = field(default_factory=dict) # input key -> regex + + def __call__(self, ev: Event) -> bool: + if self.tools and ev.tool not in self.tools and ev.kind not in self.tools: + # a shell write (`sed -i`, `echo > f`, `| tee`) is an edit for a matcher of edit tools; a + # program whose output is saved (`pytest > log`, `ls > f`) did not edit anything + if not (ev.tool == "Bash" and ev.input.get("file_path") and set(self.tools) & _WRITE_TOOLS + and shell.writes_file(ev.kind.removeprefix("Bash:"))): + return False + return all(p.search(str(ev.input.get(k, ""))) for k, p in self.patterns.items()) + + +def load_contract(path: Path) -> dict[str, Any]: + import yaml + + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except yaml.YAMLError as exc: + raise ValueError(f"{path}: not valid YAML ({exc})") from exc + return parse_contract(data, str(path)) + + +def _names(value: Any, where: str) -> tuple[str, ...]: + """A string or a list of strings; anything else (a number, a nested list, a mapping) is refused.""" + if value is None: + return () + if isinstance(value, str): + return (value,) + if isinstance(value, list) and all(isinstance(v, str) for v in value): + return tuple(value) + raise ValueError(f"{where} wants a name or a list of names, got {value!r}") + + +def parse_contract(data: Any, where: str = "contract") -> dict[str, Any]: + if not isinstance(data, dict) or not isinstance(data.get("actions"), dict): + raise ValueError(f"{where}: want `actions:` (name -> matcher) and `rules:`") + actions: dict[str, Matcher] = {} + for name, spec in data["actions"].items(): + if not isinstance(spec, dict): + raise ValueError(f"{where}: action {name!r} is not a mapping") + tools = _names(spec.get("tool"), f"{where}: action {name!r}: `tool`") + try: + patterns = {str(k): re.compile(str(v)) for k, v in spec.items() if k != "tool"} + except re.error as exc: + raise ValueError(f"{where}: action {name!r}: bad regex ({exc})") from exc + actions[str(name)] = Matcher(tools, patterns) + rules = [] + raw_rules = data.get("rules") or [] + if not isinstance(raw_rules, list): + raise ValueError(f"{where}: `rules:` wants a list") + for i, rule in enumerate(raw_rules): + if not isinstance(rule, dict) or len(rule) != 1 or next(iter(rule)) not in RULES: + raise ValueError(f"{where}: rule {i} must be one of {', '.join(RULES)}") + (kind, arg), = rule.items() + names = list(_names(arg, f"{where}: rule {i} `{kind}`")) + want = 2 if kind in ("before", "after_last") else 1 + if len(names) != want or any(n not in actions for n in names): + raise ValueError(f"{where}: rule {i} `{kind}` wants {want} defined action name(s), got {arg!r}") + rules.append((kind, names)) + return {"actions": actions, "rules": rules} + + +def check_contract(run: Run, contract: dict[str, Any]) -> list[str]: + """The rules this run broke, each as ``kind: args``.""" + hits = {name: [i for i, ev in enumerate(run.events) if m(ev)] for name, m in contract["actions"].items()} + broken = [] + for kind, names in contract["rules"]: + a = hits[names[0]] + b = hits[names[1]] if len(names) > 1 else [] + last = None + if kind == "requires": + ok = bool(a) + elif kind == "forbids": + ok = not a + elif kind == "before": # every b has an a earlier; vacuous with no b + ok = not b or (bool(a) and a[0] < b[0]) + elif kind == "after_last": # some b after the last a; vacuous with no a + ok = not a or any(j > a[-1] for j in b) + else: # last_ok: the last a did not fail, and its status was not hidden by a pipe + last = run.events[a[-1]] if a else None + ok = last is None or not (last.failed or last.masked) + if not ok: + masked = kind == "last_ok" and last is not None and not last.failed + broken.append(f"{kind}: {' -> '.join(names)}" + (" (status hidden by a pipe)" if masked else "")) + return broken + + +# --- metrics ----------------------------------------------------------------------------------- + +def levenshtein(a: list[str], b: list[str]) -> int: + prev = list(range(len(b) + 1)) + for i, x in enumerate(a, 1): + cur = [i] + for j, y in enumerate(b, 1): + cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (x != y))) + prev = cur + return prev[-1] + + +def seq_distance(a: list[str], b: list[str]) -> float: + longest = max(len(a), len(b)) + return levenshtein(a, b) / longest if longest else 0.0 + + +def jsd(a: list[str], b: list[str]) -> float: + """Jensen-Shannon divergence, base 2, of two bags of labels; 0 same mix, 1 disjoint.""" + if not a and not b: + return 0.0 + if not a or not b: + return 1.0 + keys = set(a) | set(b) + p = {k: a.count(k) / len(a) for k in keys} + q = {k: b.count(k) / len(b) for k in keys} + + def kl(x: dict[str, float], m: dict[str, float]) -> float: + return sum(x[k] * math.log2(x[k] / m[k]) for k in keys if x[k]) + + m = {k: (p[k] + q[k]) / 2 for k in keys} + return max(0.0, min(1.0, (kl(p, m) + kl(q, m)) / 2)) + + +def pass_at(n: int, c: int, k: int) -> float: + return 1.0 - math.comb(n - c, k) / math.comb(n, k) + + +def pass_hat(n: int, c: int, k: int) -> float: + return math.comb(c, k) / math.comb(n, k) + + +def _mean(xs: list[float]) -> float | None: + return sum(xs) / len(xs) if xs else None + + +def _cv(xs: list[float]) -> float | None: + if len(xs) < 2: + return None + mu = sum(xs) / len(xs) + if not mu: + return 0.0 + sd = math.sqrt(sum((x - mu) ** 2 for x in xs) / (len(xs) - 1)) + return sd / mu + + +def strategies(runs: list[Run], threshold: float) -> list[list[int]]: + """Single-linkage groups of runs whose collapsed kind paths are within ``threshold``.""" + parent = list(range(len(runs))) + + def root(i: int) -> int: + while parent[i] != i: + parent[i] = parent[parent[i]] + i = parent[i] + return i + + shapes = [r.shape() for r in runs] + for i, j in combinations(range(len(runs)), 2): + if seq_distance(shapes[i], shapes[j]) <= threshold: + parent[root(i)] = root(j) + groups: dict[int, list[int]] = {} + for i in range(len(runs)): + groups.setdefault(root(i), []).append(i) + return sorted(groups.values(), key=lambda g: (-len(g), g[0])) + + +def _verdict(passes: int, judged: int) -> str: + if not judged: + return "unjudged" + if judged == 1: # one run says nothing about stability + return "single run, passed" if passes else "single run, failed" + if passes == judged: + return "stable success" + if passes == 0: + return "stable failure" + return "unstable" + + +def profile(runs: list[Run], contract: dict[str, Any] | None = None, threshold: float = 0.34, + ks: tuple[int, ...] = (1, 2, 4, 8)) -> dict[str, Any]: + """The three-axis profile of repeated runs of one task.""" + n = len(runs) + judged = [r for r in runs if r.passed is not None] + c = sum(1 for r in judged if r.passed) + result: dict[str, Any] = {"runs": n, "judged": len(judged), "passed": c, + "success_rate": c / len(judged) if judged else None} + for k in ks: + if judged and k <= len(judged): + result[f"pass@{k}"] = pass_at(len(judged), c, k) + result[f"pass^{k}"] = pass_hat(len(judged), c, k) + + pairs = list(combinations(runs, 2)) + path = { + "mix_jsd": _mean([jsd(a.kinds(), b.kinds()) for a, b in pairs]), + "order_distance": _mean([seq_distance(a.kinds(), b.kinds()) for a, b in pairs]), + "action_distance": _mean([seq_distance(a.actions(), b.actions()) for a, b in pairs]), + "tool_calls_mean": _mean([r.calls() for r in runs]), + "tool_calls_cv": _cv([r.calls() for r in runs]), + "turns_cv": _cv([r.num_turns for r in runs if r.num_turns is not None]), + "output_tokens_cv": _cv([r.output_tokens for r in runs]), + "failed_calls_mean": _mean([r.failed_calls() for r in runs]), + } + + groups = [] + for members in strategies(runs, threshold): + rs = [runs[i] for i in members] + js = [r for r in rs if r.passed is not None] + ps = sum(1 for r in js if r.passed) + medoid = min(rs, key=lambda r: sum(seq_distance(r.shape(), o.shape()) for o in rs)) + groups.append({ + "runs": [r.name for r in rs], "passed": ps, "judged": len(js), + "verdict": _verdict(ps, len(js)), + "inner_distance": _mean([seq_distance(a.shape(), b.shape()) for a, b in combinations(rs, 2)]) or 0.0, + "path": medoid.shape(), + }) + + out: dict[str, Any] = {"result": result, "path": path, "strategies": groups, + "verdict": _summary_verdict(groups)} + if contract is not None: + per_run = {r.name: check_contract(r, contract) for r in runs} + tally: dict[str, int] = {} + for broken in per_run.values(): + for rule in broken: + tally[rule] = tally.get(rule, 0) + 1 + conforming = sum(1 for b in per_run.values() if not b) + out["contract"] = { + "rules": len(contract["rules"]), "conforming_runs": conforming, + "conformance_rate": conforming / n if n else None, + "violations": dict(sorted(tally.items(), key=lambda kv: -kv[1])), + "per_run": per_run, + # a pass that broke a rule: the result alone would have hidden it + "passed_but_nonconforming": sum(1 for r in runs if r.passed and per_run[r.name]), + } + return out + + +def _summary_verdict(groups: list[dict[str, Any]]) -> str: + if not groups: + return "no runs" + verdicts = {g["verdict"] for g in groups} + if len(groups) == 1: + return f"one strategy, {groups[0]['verdict']}" + if verdicts <= {"stable success", "single run, passed"}: + return f"{len(groups)} strategies, all stable success (valid diversity)" + return f"{len(groups)} strategies: " + ", ".join( + f"{g['verdict']} ×{len(g['runs'])}" for g in groups) diff --git a/dokimi/dokimi/context/shell.py b/dokimi/dokimi/context/shell.py new file mode 100644 index 00000000..39b22dc5 --- /dev/null +++ b/dokimi/dokimi/context/shell.py @@ -0,0 +1,270 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Reading one ``Bash`` call as the commands it runs. + +Not a shell parser: a walk over the line that knows quotes, backslashes, ``$(...)`` / backticks, +``&&``, ``||``, ``;``, ``&`` and newlines, and peels the compound-command syntax (``for``, ``if``, +``case``, ``( ... )``) and the wrappers (``sudo``, ``timeout 300``, ``nice -n 10``, ``VAR=1``, +``uv run``) off a command to find the program and the file it writes. Good for the lines agents +write; a line it cannot read gives a step of its own, never an exception. +""" + +from __future__ import annotations + +import re +import shlex +from pathlib import Path + +#: Separators between commands, and what each says about the status a call reports: ``a && b`` fails +#: as a whole when ``a`` does; after ``;``, a newline, ``||`` or ``&`` the status is a later command's. +HIDING = {";", "\n", "||", "&"} + +_OPERATORS = {"|", "||", "&&", ";", "&", ">", ">>", ">|", ">&", "<", "<<", "&>", "&>>", "(", ")"} +_SHELL_WRITERS = {"sed", "perl", "echo", "printf", "cat", "tee"} # their output written to a file is an edit +_KEYWORDS = r"(?:if|then|else|elif|do|while|until|!|\{)" +_NO_COMMAND = re.compile(r"^(?:for|select|fi|done|esac|\}|in|function)\b") +# `-i`, `-i.bak`, `-ni`, `--in-place`: the in-place flag by its tool's own option letters, so that +# `perl -Mstrict` is not an edit. +_IN_PLACE = {"sed": re.compile(r"^(-[nEsrzu]*i[^-\s]*|--in-place(=.*)?)$"), + "perl": re.compile(r"^-[pnalw0-9]*i[^-\s]*$")} + + +class _Scan: + """The state of the walk over a shell line: the quote open, the ``$(...)`` depth, a backtick.""" + + def __init__(self, command: str): + self.text, self.i = command, 0 + self.cur: list[str] = [] + self.quote, self.depth, self.tick = "", 0, False + + def eat(self, n: int = 1, keep: bool = True) -> None: + if keep: + self.cur.append(self.text[self.i:self.i + n]) + self.i += n + + def step(self) -> str: + """Consume the next piece; return the separator it was, if it was one.""" + ch, nxt = self.text[self.i], self.text[self.i + 1:self.i + 2] + if self.quote: + if ch == "\\" and self.quote == '"' and nxt: + self.eat(2) + else: + self.quote = "" if ch == self.quote else self.quote + self.eat() + elif ch == "\\": # a continued line is dropped; an escaped character is a word character + self.eat(2, keep=nxt != "\n") + elif ch in "'\"`" or (ch == "$" and nxt == "(") or (self.depth and ch in "()"): + self.open_close(ch, nxt) + elif ch == "#" and not (self.depth or self.tick) and (not self.cur or self.cur[-1].isspace()): + while self.i < len(self.text) and self.text[self.i] != "\n": # a comment runs to the end of the line + self.i += 1 + else: + sep = "" if self.depth or self.tick else _separator(self.text, self.i) + if sep: + self.i += len(sep) + return sep + self.eat() + return "" + + def open_close(self, ch: str, nxt: str) -> None: + """A quote, a backtick, ``$(`` or a parenthesis inside ``$(...)``: track it, keep it.""" + if ch in "'\"": + self.quote = ch + elif ch == "`": + self.tick = not self.tick + elif ch == "$": + self.depth += 1 + self.eat(2) + return + else: + self.depth += 1 if ch == "(" else -1 + self.eat() + + +def split_commands(command: str) -> list[tuple[str, str]]: + """``(command, separator after it)`` for each command of a shell line. + + Heredoc bodies and comments are dropped (their lines are data); a separator inside quotes, + after a backslash, or inside ``$(...)`` / backticks does not split.""" + command = re.sub(r"<<-?\s*['\"]?(\w+)['\"]?([^\n]*)\n.*?\n\s*\1\s*(?=\n|$)", r"<<\1\2", command, flags=re.S) + out: list[tuple[str, str]] = [] + scan = _Scan(command) + while scan.i < len(command): + sep = scan.step() + if sep: + out.append(("".join(scan.cur), sep)) + scan.cur = [] + out.append(("".join(scan.cur), "")) + return [(seg.strip(), sep) for seg, sep in out if seg.strip() and not seg.strip().startswith("#")] + + +def _separator(command: str, i: int) -> str: + ch, nxt, prev = command[i], command[i + 1:i + 2], command[i - 1:i] if i else "" + if ch in ";\n": + return ch + if command[i:i + 2] in ("&&", "||"): + return command[i:i + 2] + if ch == "&" and nxt != ">" and prev not in (">", "<", "&"): # not `&>`, `2>&1`, `<&3` + return "&" + return "" + + +def segments(command: str) -> list[str]: + return [seg for seg, _ in split_commands(command)] + + +def tokens(segment: str) -> list[str]: + """Shell words with unquoted operators (``|``, ``>``, ``>>``, ``2>&1``) as their own tokens, so a + ``>`` or ``|`` inside quotes stays part of a word.""" + lex = shlex.shlex(_mask(segment), posix=True, punctuation_chars=True) + lex.whitespace_split = True + try: + return list(lex) + except ValueError: + return segment.split() + + +def _mask(text: str) -> str: + """``text`` with each ``$(...)`` and backtick substitution replaced by one word, so its inner + spaces, ``;`` and ``&&`` are not read as this command's.""" + out, i, depth, quote = [], 0, 0, "" + while i < len(text): + ch = text[i] + if quote: + quote = "" if ch == quote else quote + elif ch in "'\"" and not depth: + quote = ch + if not quote and ch == "$" and text[i + 1:i + 2] == "(": + j, d = i + 2, 1 + while j < len(text) and d: + d += (text[j] == "(") - (text[j] == ")") + j += 1 + out.append("SUBST") + i = j + continue + if not quote and ch == "`": + j = text.find("`", i + 1) + j = len(text) - 1 if j < 0 else j + out.append("SUBST") + i = j + 1 + continue + out.append(ch) + i += 1 + return "".join(out) + + +def clean(segment: str) -> str: + """The command of a segment without the compound-command syntax around it: + ``do black $f`` -> ``black $f``, ``(cd sub`` -> ``cd sub``, ``pytest)`` -> ``pytest``, a case label + and its ``case ... in`` header dropped; a ``for`` header, ``fi``, ``done`` leave nothing.""" + text = segment.strip() + while True: + before = text + text = re.sub(rf"^(?:\(+|{_KEYWORDS}(?=\s|$))\s*", "", text) + text = re.sub(r"^case\b.*?\bin\b\s*", "", text) if text.startswith("case ") else text + text = re.sub(r"^\(?[^\s()]+\)(?=\s+\S)\s*", "", text) # a `case` label followed by its command + if text == before: + break + if _NO_COMMAND.match(text) or text.startswith("case "): + return "" + while text.endswith(")") and text.count(")") > text.count("("): + text = text[:-1].rstrip() + return text + + +def _words(segment: str) -> list[str]: + try: + return shlex.split(_mask(clean(segment))) + except ValueError: + return clean(segment).split() + + +_DURATION = re.compile(r"^\d+(\.\d+)?[smhd]?$") + + +def _wrapper_length(words: list[str]) -> int: + """How many leading words only wrap the program (0: ``words[0]`` is the program).""" + head = Path(words[0]).name + if re.match(r"^\w+=", words[0]) or head in ("exec", "time", "nohup", "command", "builtin"): + return 1 + if head in ("sudo", "env", "nice"): + n = 1 + while n < len(words) and (words[n].startswith("-") or (head == "env" and re.match(r"^\w+=", words[n]))): + n += 2 if head == "nice" and words[n] == "-n" else 1 + return n + if head == "timeout": + n = 1 + while n < len(words) and words[n].startswith("-"): + n += 2 if words[n] in ("-k", "-s") else 1 + return n + (n < len(words) and bool(_DURATION.match(words[n]))) + if words[:2] == ["uv", "run"] or (len(words) >= 3 and re.match(r"^python[\d.]*$", head) and words[1] == "-m"): + return 2 + return 0 + + +def strip_wrappers(words: list[str]) -> list[str]: + """``words`` past what only wraps the program: ``VAR=1``, ``sudo``, ``env``, ``time``, ``nohup``, + ``nice -n 10``, ``timeout 300``, ``uv run``, ``python -m``.""" + words = list(words) + while words and (n := _wrapper_length(words)): + del words[:n] + return words + + +def program(command: str) -> str: + """The program a shell line runs: past ``cd x &&``, wrappers and compound-command syntax.""" + for seg in segments(command): + words = strip_wrappers(_words(seg)) + if not words or words[0] in ("cd", "pushd", "source", ".", "export", "set", "[", "[["): + continue + return Path(words[0]).name + return "" + + +def step_text(segment: str) -> str: + """The command of a segment as the agent wrote it, without the compound-command syntax.""" + return clean(segment) or segment.strip() + + +def written(segment: str) -> str: + """The file a shell command writes, if it is one of the plain ways to write one.""" + words = tokens(clean(segment)) + stages: list[list[str]] = [[]] + for w in words: + stages.append([]) if w == "|" else stages[-1].append(w) + for stage in stages: + if target := _stage_writes(strip_wrappers(stage)): + return target + for i, w in enumerate(words[:-1]): + if w in (">", ">>", ">|") and not (i and _fd_redirect(segment, words[i - 1])): + target = words[i + 1] + if target not in _OPERATORS and target != "/dev/null": + return target + return "" + + +def _stage_writes(stage: list[str]) -> str: + """The file an in-place edit (``sed -i f``) or ``tee f`` writes, in one stage of a pipeline.""" + if not stage: + return "" + in_place = _IN_PLACE.get(Path(stage[0]).name) + if in_place and any(in_place.match(w) for w in stage[1:]): + files = [w for w in stage[1:] if not w.startswith("-") and w not in _OPERATORS] + if len(files) >= 2: + return files[-1] + if stage[0] == "tee": + rest = [x for x in stage[1:] if not x.startswith("-")] + if rest and rest[0] not in _OPERATORS: + return rest[0] + return "" + + +def _fd_redirect(segment: str, prev: str) -> bool: + """``2>`` / ``2>>`` (a digit glued to the operator) redirects another stream; ``echo 3 > f`` writes.""" + return prev.isdigit() and prev != "1" and re.search(rf"(?", segment) is not None + + +def writes_file(program_name: str) -> bool: + """Whether a program's saved output counts as an edit when it names a file (``cat > f``, not ``pytest > log``).""" + return program_name in _SHELL_WRITERS diff --git a/dokimi/dokimi/repos/compare.py b/dokimi/dokimi/repos/compare.py index aacc38c7..254cd97d 100644 --- a/dokimi/dokimi/repos/compare.py +++ b/dokimi/dokimi/repos/compare.py @@ -13,7 +13,16 @@ failure counts as 0); - **cost_usd**, **turns**: the per-cell means the harness reported; - **metric:**: the mean of ``grade.json`` ``metrics.`` over the - cells that report it (``true`` is 1), for ``--metric``. + cells that report it (``true`` is 1), for ``--metric``; +- **pass^k**: for ``--pass-k K``, the chance that K cells drawn from the + task's n all pass its gate, ``C(c,K)/C(n,K)`` over c passes (τ-bench). The + gate is the per-cell rate; pass^k says whether the task passes every time. + A task with fewer than K cells has none; +- **path_distance**: for ``--path-stability``, the mean order distance of the + tool paths (``events.jsonl``, main thread) over every pair of the task's + cells, 0 the same steps in the same order, 1 nothing in common + (:mod:`dokimi.context.consistency`). A task with under two traced cells has + none. A negative delta is a steadier path, not a better one. A task enters a comparison only when both arms have a value for it; the others are listed as unpaired, with the reason (no cells in one arm, a task kind that @@ -68,6 +77,7 @@ from pathlib import Path from typing import Any +from dokimi.context.consistency import Run, order_distance, pass_hat, run_from_events from dokimi.eval.stats import adjust_pvalues, bootstrap_ci, mean_ci from .report import (load_cells, mismatch_warnings, model_matches, model_mismatch, resolved_models, unmeasured_note, @@ -103,8 +113,10 @@ def selector_matches(selector: tuple[str | None, str], cell: dict[str, Any]) -> return cell["arm"] == arm and (batch is None or cell["batch"] == batch) -def load_batches(batch_dirs: list[Path], metrics: tuple[str, ...] = ()) -> tuple[list[dict[str, Any]], list[str]]: - """Finished cells of every batch, each with its ``batch`` and the requested ``grade.json`` metrics.""" +def load_batches(batch_dirs: list[Path], metrics: tuple[str, ...] = (), + paths: bool = False) -> tuple[list[dict[str, Any]], list[str]]: + """Finished cells of every batch, each with its ``batch``, the requested ``grade.json`` metrics and, + for ``paths``, its tool path (``path``, ``None`` without ``events.jsonl``).""" names = [d.resolve().name for d in batch_dirs] # resolved: Path('.').name is empty if len(set(names)) != len(names): raise CompareError(f"batch directories must have distinct names: {', '.join(names)}") @@ -115,6 +127,8 @@ def load_batches(batch_dirs: list[Path], metrics: tuple[str, ...] = ()) -> tuple for rec in found: rec["batch"] = name rec["metrics"] = _grade_metrics(batch_dir / rec["cell"] / "grade.json", metrics) if metrics else {} + if paths: + rec["path"] = _cell_path(batch_dir / rec["cell"] / "events.jsonl") cells.append(rec) return cells, missing @@ -129,20 +143,25 @@ def batch_stamp(batch_dir: Path) -> dict[str, str]: def compare(batch_dirs: list[Path], baseline: str, treatments: list[str] | None = None, - models: list[str] | None = None, metrics: tuple[str, ...] = (), seed: int = 0) -> dict[str, Any]: + models: list[str] | None = None, metrics: tuple[str, ...] = (), seed: int = 0, + pass_k: int | None = None, path_stability: bool = False) -> dict[str, Any]: metrics = tuple(dict.fromkeys(metrics)) - cells, missing = load_batches(batch_dirs, metrics) + if pass_k is not None and pass_k < 1: + raise CompareError(f"--pass-k must be at least 1, got {pass_k}") + cells, missing = load_batches(batch_dirs, metrics, path_stability) _check_metrics(metrics, cells) base_sel = parse_selector(baseline) if not any(selector_matches(base_sel, c) for c in cells): raise CompareError(f"no finished cell matches the baseline {baseline!r}") treatments = _treatments(cells, baseline, base_sel, treatments) - measures = ["gate", "score", "cost_usd", "turns", *(f"metric:{m}" for m in metrics)] - base_vals = _task_values([c for c in cells if selector_matches(base_sel, c)], metrics) + measures = ["gate", "score", *([f"pass^{pass_k}"] if pass_k else []), + *(["path_distance"] if path_stability else []), "cost_usd", "turns", *(f"metric:{m}" for m in metrics)] + base_vals = _task_values([c for c in cells if selector_matches(base_sel, c)], metrics, pass_k, path_stability) rows: list[dict[str, Any]] = [] unpaired: list[dict[str, Any]] = [] for text in treatments: - treat_vals = _task_values([c for c in cells if selector_matches(parse_selector(text), c)], metrics) + treat_vals = _task_values([c for c in cells if selector_matches(parse_selector(text), c)], metrics, + pass_k, path_stability) unpaired += _one_arm_models(text, base_vals, treat_vals, models) shared_models = sorted({m for m, _ in base_vals} & {m for m, _ in treat_vals}) for model in [m for m in shared_models if not models or m in models]: @@ -289,19 +308,7 @@ def _compare_model(treatment: str, model: str, base_vals: dict[tuple[str, str], if b != t: # batches of different eval trees: the primary result of this task has no pair unpaired.append(_unpaired(treatment, model, task, None, f"{b} task in baseline, {t} in treatment")) for measure in measures: - pairs = [] - for task in shared: - base, treat = base_vals[(model, task)], treat_vals[(model, task)] - if measure in PRIMARY and not base["kind"] == treat["kind"] == measure: - continue # a task of the other kind, or one listed above - pair = _pair(task, base, treat, measure) - if pair is None: - lacking = [arm for arm, v in (("baseline", base), ("treatment", treat)) - if v["values"].get(measure) is None] - unpaired.append(_unpaired(treatment, model, task, measure, - f"no {measure} in {' or '.join(lacking)}")) - else: - pairs.append(pair) + pairs = _measure_pairs(treatment, model, shared, base_vals, treat_vals, measure, unpaired) if pairs: flag = None if measure in PRIMARY: @@ -313,6 +320,35 @@ def _compare_model(treatment: str, model: str, base_vals: dict[tuple[str, str], **paired_summary(pairs, measure, seed=seed), "headroom": flag}) +def _measure_pairs(treatment: str, model: str, shared: list[str], base_vals: dict[tuple[str, str], dict[str, Any]], + treat_vals: dict[tuple[str, str], dict[str, Any]], measure: str, + unpaired: list[dict[str, Any]]) -> list[dict[str, Any]]: + """The paired tasks of one measure; the tasks lacking it go to ``unpaired``. + + ``pass^K`` and ``path_distance`` are optional: a score task has no ``pass^K``, and a task with too + few cells for either is one line with a count, not one entry per task.""" + pairs, short = [], 0 + for task in shared: + base, treat = base_vals[(model, task)], treat_vals[(model, task)] + if measure in PRIMARY and not base["kind"] == treat["kind"] == measure: + continue # a task of the other kind, or one listed above + if measure.startswith("pass^") and "score" in (base["kind"], treat["kind"]): + continue # pass^K is a gate measure: a score task has none to lack + pair = _pair(task, base, treat, measure) + if pair is not None: + pairs.append(pair) + elif measure.startswith("pass^") or measure == "path_distance": + short += 1 + else: + lacking = [arm for arm, v in (("baseline", base), ("treatment", treat)) + if v["values"].get(measure) is None] + unpaired.append(_unpaired(treatment, model, task, measure, f"no {measure} in {' or '.join(lacking)}")) + if short: + unpaired.append(_unpaired(treatment, model, f"{short} task(s)", measure, + f"too few cells in baseline or treatment for {measure}")) + return pairs + + def headroom(pairs: list[dict[str, Any]], tops: list[tuple[float | None, float | None]], failed: list[bool] | None = None) -> str | None: """``ceiling``, ``floor``, ``near_ceiling`` or ``None`` for one gate or score comparison (module docstring). @@ -493,7 +529,8 @@ def _format_header(summary: dict[str, Any]) -> list[str]: return lines -def _task_values(cells: list[dict[str, Any]], metrics: tuple[str, ...]) -> dict[tuple[str, str], dict[str, Any]]: +def _task_values(cells: list[dict[str, Any]], metrics: tuple[str, ...], pass_k: int | None = None, + path_stability: bool = False) -> dict[tuple[str, str], dict[str, Any]]: """One value per measure for each (model, task) of one arm.""" groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list) for rec in cells: @@ -517,10 +554,18 @@ def _task_values(cells: list[dict[str, Any]], metrics: tuple[str, ...]) -> dict[ top = (maxes.pop() if len(maxes) == 1 and not has_unknown_max else None) if score_task else 1.0 # unmeasured: a grader failure, or an agent outage (see _agent_outage); either way its 0 is not a measurement grader_failed = any(r["grade"]["status"] == "grader_failure" or _agent_outage(r) for r in recs) - out[key] = {"kind": "score" if score_task else "gate", "grader_failed": grader_failed, - "top": top, - "n": {measure: len(v) for measure, v in nums.items()}, - "values": {measure: (sum(v) / len(v) if v else None) for measure, v in nums.items()}} + n = {measure: len(v) for measure, v in nums.items()} + values = {measure: (sum(v) / len(v) if v else None) for measure, v in nums.items()} + if pass_k and not score_task: + gates = nums["gate"] + values[f"pass^{pass_k}"] = pass_hat(len(gates), int(sum(gates)), pass_k) if len(gates) >= pass_k else None + n[f"pass^{pass_k}"] = len(gates) + if path_stability: + runs: list[Run] = [r["path"] for r in recs if r.get("path") is not None] + values["path_distance"] = order_distance(runs) + n["path_distance"] = len(runs) + out[key] = {"kind": "score" if score_task else "gate", "grader_failed": grader_failed, "top": top, + "n": n, "values": values} return out @@ -537,6 +582,22 @@ def _agent_outage(rec: dict[str, Any]) -> bool: return crashed and not turns and not agent.get("timed_out") +def _cell_path(path: Path) -> Run | None: + """A cell's main-thread tool path from its ``events.jsonl``; ``None`` when it has none.""" + try: + lines = path.read_text(encoding="utf-8", errors="replace").splitlines() + except OSError: + return None + events = [] + for line in lines: + try: + events.append(json.loads(line)) + except (ValueError, RecursionError): # a deeply nested line is as unreadable as a broken one + continue + run = run_from_events(events, str(path)) + return run if run.events else None # a crashed cell leaves no path; it is not a steady one + + def _grade_metrics(path: Path, names: tuple[str, ...]) -> dict[str, float]: """The requested numeric or boolean ``metrics`` of a cell's ``grade.json``; absent ones are left out.""" try: diff --git a/dokimi/mkdocs.yml b/dokimi/mkdocs.yml index 8c8775be..dd048e95 100644 --- a/dokimi/mkdocs.yml +++ b/dokimi/mkdocs.yml @@ -113,6 +113,7 @@ nav: - adding-doc-source.md - evaluating-treatments.md - tool-preference.md + - run-consistency.md - coding-harnesses.md - context-window.md - Content novelty: diff --git a/dokimi/tests/test_context_consistency.py b/dokimi/tests/test_context_consistency.py new file mode 100644 index 00000000..687835ad --- /dev/null +++ b/dokimi/tests/test_context_consistency.py @@ -0,0 +1,452 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""`context consistency`: result stability, path stability and contract conformance of repeated runs.""" + +from __future__ import annotations + +import json + +import pytest + +from dokimi.cli import build_parser +from dokimi.context.consistency import (apply_outcomes, check_contract, jsd, + load_contract, load_outcomes, parse_contract, parse_run, + pass_at, pass_hat, profile, run_from_events, seq_distance) +from dokimi.context.shell import program as _program +from dokimi.context.shell import written as _written + + +def _trace(*steps, ok=True, turns=None): + """steps: (tool, input) or (tool, input, failed).""" + lines = [{"type": "system", "subtype": "init", "model": "m"}] + for i, step in enumerate(steps): + tool, args, failed = (*step, False)[:3] + lines.append({"type": "assistant", "parent_tool_use_id": None, + "message": {"id": f"m{i}", "usage": {"output_tokens": 10}, + "content": [{"type": "tool_use", "id": f"t{i}", "name": tool, "input": args}]}}) + lines.append({"type": "user", "parent_tool_use_id": None, + "message": {"content": [{"type": "tool_result", "tool_use_id": f"t{i}", + "content": "x", "is_error": failed}]}}) + lines.append({"type": "result", "subtype": "success" if ok else "error_max_turns", + "is_error": not ok, "num_turns": turns or len(steps)}) + return "\n".join(json.dumps(x) for x in lines) + + +READ = ("Read", {"file_path": "/w/AGENTS.md"}) +EDIT = ("Edit", {"file_path": "/w/src/a.py"}) +TEST = ("Bash", {"command": "cd /w && uv run pytest -q"}) +FAILED_TEST = ("Bash", {"command": "pytest -q"}, True) +GREP = ("Grep", {"pattern": "foo"}) + + +def _run(name, *steps, ok=True): + return parse_run(_trace(*steps, ok=ok), name) + + +def test_pass_estimates(): + assert pass_hat(4, 4, 4) == 1.0 + assert pass_hat(4, 2, 2) == pytest.approx(1 / 6) + assert pass_at(4, 2, 2) == pytest.approx(5 / 6) + assert pass_at(4, 0, 4) == 0.0 + + +def test_distances(): + assert seq_distance(["a", "b"], ["a", "b"]) == 0 + assert seq_distance(["a", "b", "c"], ["a", "c", "b"]) == pytest.approx(2 / 3) + assert jsd(["a", "b"], ["b", "a"]) == 0 # same mix, any order + assert jsd(["a"], ["b"]) == pytest.approx(1.0) + + +@pytest.mark.parametrize("cmd, want", [ + ("cd /w && uv run pytest -q", "pytest"), ("FOO=1 sudo make test", "make"), + ("python3 -m pytest", "pytest"), ("git status; ls", "git"), ("export X=1", ""), +]) +def test_program(cmd, want): + assert _program(cmd) == want + + +def test_parse_run_reads_kinds_actions_and_failures(): + run = _run("r", READ, EDIT, FAILED_TEST) + assert run.kinds() == ["Read", "Edit", "Bash:pytest"] + assert run.actions()[:2] == ["Read(AGENTS.md)", "Edit(a.py)"] + assert [e.failed for e in run.events] == [False, False, True] + assert run.passed is True and run.output_tokens == 30 + + +def test_subagent_calls_are_not_the_path(): + raw = _trace(READ) + sub = {"type": "assistant", "parent_tool_use_id": "t0", + "message": {"id": "s", "content": [{"type": "tool_use", "id": "s1", "name": "Grep", "input": {}}]}} + run = parse_run(raw + "\n" + json.dumps(sub)) + assert run.kinds() == ["Read"] + + +def test_same_steps_different_order_is_order_not_mix(): + a, b = _run("a", READ, EDIT, TEST), _run("b", READ, TEST, EDIT) + p = profile([a, b])["path"] + assert p["mix_jsd"] == 0 and p["order_distance"] == pytest.approx(2 / 3) + + +def test_two_valid_strategies_are_diversity_not_instability(): + runs = [_run(f"g{i}", GREP, GREP, EDIT, TEST) for i in range(3)] + \ + [_run(f"r{i}", READ, EDIT, TEST, READ, EDIT, TEST) for i in range(3)] + p = profile(runs) + assert len(p["strategies"]) == 2 + assert {g["verdict"] for g in p["strategies"]} == {"stable success"} + assert "valid diversity" in p["verdict"] + assert p["result"]["pass^4"] == 1.0 + + +def test_one_strategy_that_breaks_is_unstable(): + runs = [_run("a", READ, EDIT, TEST), _run("b", READ, EDIT, TEST, ok=False)] + p = profile(runs) + assert p["verdict"] == "one strategy, unstable" and p["result"]["pass^2"] == 0 + + +CONTRACT = { + "actions": {"guide": {"tool": "Read", "file_path": r"AGENTS\.md$"}, + "edit": {"tool": ["Edit", "Write"]}, + "test": {"tool": "Bash", "command": r"pytest"}, + "push": {"tool": "Bash", "command": r"git push"}}, + "rules": [{"before": ["guide", "edit"]}, {"after_last": ["edit", "test"]}, + {"last_ok": "test"}, {"forbids": "push"}], +} + + +def test_contract_catches_a_pass_that_skipped_the_final_check(): + c = parse_contract(CONTRACT) + assert check_contract(_run("ok", READ, EDIT, TEST), c) == [] + assert check_contract(_run("late", EDIT, READ, TEST), c) == ["before: guide -> edit"] + assert check_contract(_run("untested", READ, TEST, EDIT), c) == ["after_last: edit -> test"] + assert check_contract(_run("red", READ, EDIT, FAILED_TEST), c) == ["last_ok: test"] + p = profile([_run("ok", READ, EDIT, TEST), _run("untested", READ, TEST, EDIT)], c) + assert p["contract"]["conformance_rate"] == 0.5 + assert p["contract"]["passed_but_nonconforming"] == 1 + + +def test_contract_any_unconstrained_order_conforms(): + c = parse_contract(CONTRACT) + assert check_contract(_run("x", GREP, READ, GREP, EDIT, EDIT, TEST), c) == [] + + +@pytest.mark.parametrize("bad", [ + {"rules": []}, {"actions": {"a": {}}, "rules": [{"nope": "a"}]}, + {"actions": {"a": {}}, "rules": [{"before": "a"}]}, + {"actions": {"a": {"x": "("}}, "rules": []}, +]) +def test_contract_refuses_malformed(bad): + with pytest.raises(ValueError): + parse_contract(bad) + + +def test_cli_arms_outcomes_and_json(tmp_path, capsys): + for arm, steps in (("base", [(READ, TEST, EDIT), (EDIT, TEST)]), ("skill", [(READ, EDIT, TEST)] * 2)): + for i, s in enumerate(steps): + (tmp_path / arm).mkdir(exist_ok=True) + (tmp_path / arm / f"{i}.jsonl").write_text(_trace(*s)) + outcomes = tmp_path / "out.json" + outcomes.write_text(json.dumps({str(tmp_path / "base/0.jsonl"): False, str(tmp_path / "base/1.jsonl"): True, + str(tmp_path / "skill/0.jsonl"): True, str(tmp_path / "skill/1.jsonl"): True})) + contract = tmp_path / "c.yaml" + import yaml + contract.write_text(yaml.safe_dump(CONTRACT)) + args = build_parser().parse_args([ + "context", "consistency", "--arm", f"base={tmp_path}/base/*.jsonl", + "--arm", f"skill={tmp_path}/skill/*.jsonl", "--outcomes", str(outcomes), + "--contract", str(contract), "--json"]) + args.func(args) + rep = json.loads(capsys.readouterr().out) + assert rep["base"]["result"]["success_rate"] == 0.5 + assert rep["skill"]["result"]["pass^2"] == 1.0 + assert rep["skill"]["path"]["order_distance"] == 0 + assert rep["skill"]["contract"]["conformance_rate"] == 1.0 + assert rep["base"]["contract"]["conformance_rate"] == 0.0 + + args = build_parser().parse_args(["context", "consistency", "--arm", f"skill={tmp_path}/skill/*.jsonl"]) + args.func(args) + text = capsys.readouterr().out + assert "| skill | 2 | 2 | 100%" in text and "harness result event" in text + + +def test_one_shell_call_is_the_steps_it_chains(): + cmd = ("cd /w; sed -i 's/a/b/' shop/pricing.py\nsed -i 's/x/y/' CHANGELOG.md\n" + "grep -n d shop/pricing.py; sh .claude/skills/s/finish.sh") + run = _run("r", ("Bash", {"command": cmd}, True)) + assert run.kinds() == ["Bash:sed", "Bash:sed", "Bash:grep", "Bash:sh"] + assert [e.input.get("file_path") for e in run.events] == ["shop/pricing.py", "CHANGELOG.md", None, None] + assert all(e.failed for e in run.events) # the trace does not say which chained command failed + assert parse_run(_trace(("Bash", {"command": cmd})), grain="call").kinds() == ["Bash:sed"] + + +def test_shell_edit_satisfies_the_contract(): + c = parse_contract({"actions": {"log": {"tool": ["Edit", "Bash"], "file_path": r"CHANGELOG\.md$"}}, + "rules": [{"requires": "log"}]}) + assert check_contract(_run("r", ("Bash", {"command": "echo '- fix' >> CHANGELOG.md"})), c) == [] + assert check_contract(_run("r", ("Bash", {"command": "cat CHANGELOG.md"})), c) == ["requires: log"] + + +@pytest.mark.parametrize("seg, want", [ + ("sed -i 's/a/b/' f.py", "f.py"), ("sed -n 1,5p f.py", ""), ("echo x > out.txt", "out.txt"), + ("make 2>&1", ""), ("ls >/dev/null", ""), ("echo x | tee -a log.md", "log.md"), + ("echo 'a > b' | grep a", ""), ('python3 -c "print(1 > 0)"', ""), ("echo x>out.txt", "out.txt"), + ("make 2> err.log", ""), +]) +def test_written(seg, want): + assert _written(seg) == want + + +def test_heredoc_body_is_not_commands(): + cmd = "python3 - <<'EOF'\nimport os\nrm_everything()\nEOF\npytest -q" + assert _run("r", ("Bash", {"command": cmd})).kinds() == ["Bash:python3", "Bash:pytest"] + + +def test_blocked_stop_hook_is_a_failed_step(): + raw = _trace(EDIT) + "\n" + json.dumps({"type": "user", "parent_tool_use_id": None, "message": { + "content": [{"type": "text", "text": "Stop hook feedback:\n[sh hook]: Not done"}]}}) + run = parse_run(raw) + assert run.kinds() == ["Edit", "Hook:Stop"] and run.events[-1].failed + + +def test_quoted_newlines_stay_one_command(): + cmd = 'cd /w && python3 -c "\nfrom a import b\n# note\nprint(b(1); 2)\n" && make check \\\n -j2\n# done' + assert _run("r", ("Bash", {"command": cmd})).kinds() == ["Bash:python3", "Bash:make"] + + +TEST_CONTRACT = {"actions": {"test": {"tool": "Bash", "command": r"\b(pytest|make check)\b"}}, + "rules": [{"last_ok": "test"}]} + + +def test_failed_chain_fails_the_check_not_only_the_last_command(): + run = _run("r", ("Bash", {"command": "make check && git diff"}, True)) + assert check_contract(run, parse_contract(TEST_CONTRACT)) == ["last_ok: test"] + + +@pytest.mark.parametrize("cmd, broken", [ + ("pytest -q | tail -5", ["last_ok: test (status hidden by a pipe)"]), + ("set -o pipefail; pytest -q | tail -5", []), + ("pytest -q 'a|b'", []), +]) +def test_pipe_hides_the_check_status(cmd, broken): + run = _run("r", ("Bash", {"command": cmd})) + assert check_contract(run, parse_contract(TEST_CONTRACT)) == broken + assert check_contract(parse_run(_trace(("Bash", {"command": cmd})), grain="call"), + parse_contract(TEST_CONTRACT)) == broken + + +# --- review of #155: shell syntax, edits through the shell, bad input, ambiguous outcomes --------- + +def _kinds(cmd, grain="split"): + return parse_run(_trace(("Bash", {"command": cmd})), grain=grain).kinds() + + +@pytest.mark.parametrize("cmd, want", [ + ("for f in a b; do black $f; done", ["Bash:black"]), + ("if [ -f x ]; then make; fi", ["Bash:make"]), + ("(cd sub && pytest)", ["Bash:pytest"]), + ("# note && x\npytest", ["Bash:pytest"]), + ("while true; do git pull; done", ["Bash:true", "Bash:git"]), + ("if make check; then echo ok; else git diff; fi", ["Bash:make", "Bash:echo", "Bash:git"]), +]) +def test_control_syntax_is_not_a_step(cmd, want): + assert _kinds(cmd) == want + assert _kinds(cmd, "call")[0] == want[0] + + +@pytest.mark.parametrize("seg, want", [ + ("echo 3 > count.txt", "count.txt"), ("seq 1 10 > out.txt", "out.txt"), + ("make 2> err.log", ""), ("make 2>> err.log", ""), ("make &> all.log", ""), + ("perl -Mstrict -e 'print 1' foo.pl", ""), ("perl -pi -e 's/a/b/' foo.pl", "foo.pl"), + ("sed --in-place 's/a/b/' f.py", "f.py"), ("sed -i.bak 's/a/b/' f.py", "f.py"), +]) +def test_written_review_cases(seg, want): + assert _written(seg) == want + + +def test_a_shell_edit_matches_an_edit_action(): + c = parse_contract({"actions": {"guide": {"tool": "Read", "file_path": r"AGENTS\.md$"}, + "edit": {"tool": ["Edit", "Write", "MultiEdit"]}, + "test": {"tool": "Bash", "command": "pytest"}}, + "rules": [{"requires": "edit"}, {"before": ["guide", "edit"]}, + {"after_last": ["edit", "test"]}]}) + shell = ("Bash", {"command": "sed -i s/a/b/ f.py && pytest -q"}) + assert check_contract(_run("r", READ, shell), c) == [] + bare = ("Bash", {"command": "sed -i s/a/b/ f.py"}) + assert check_contract(_run("r", bare, READ), c) == ["before: guide -> edit", "after_last: edit -> test"] + # a shell step that writes nothing is not an edit + assert check_contract(_run("r", READ, ("Bash", {"command": "pytest -q"})), c) == ["requires: edit"] + + +@pytest.mark.parametrize("cmd", ["pytest -q > test.log", "pytest -q > /tmp/test.log 2>&1", + "set -o pipefail; pytest -q 2>&1 | tee test.log"]) +def test_saving_a_test_log_is_not_an_edit(cmd): + c = parse_contract({**CONTRACT, "rules": [*CONTRACT["rules"], {"requires": "edit"}]}) + assert check_contract(_run("r", READ, EDIT, ("Bash", {"command": cmd})), c) == [] + scratch = ("Bash", {"command": "ls > /tmp/files.txt"}) + assert check_contract(_run("r", scratch, READ, TEST), c) == ["requires: edit"] + for write in ("echo x > src/a.py", "cat > src/a.py <<'EOF'\nx\nEOF", "printf x | tee src/a.py"): + assert check_contract(_run("r", READ, ("Bash", {"command": write}), TEST), c) == [] + + +def test_no_runs_is_not_a_stable_success(): + p = profile([]) + assert p["verdict"] == "no runs" and p["strategies"] == [] + + +def test_malformed_outcomes_and_contract_are_value_errors(tmp_path): + o = tmp_path / "o.json" + o.write_text('[{"trace": "t1.jsonl"}]') + with pytest.raises(ValueError, match="passed"): + load_outcomes(o) + c = tmp_path / "c.yaml" + c.write_text("actions: [unclosed") + with pytest.raises(ValueError, match="c.yaml"): + load_contract(c) + c.write_text("actions: {a: {tool: Edit}}\nrules: 5") + with pytest.raises(ValueError): + load_contract(c) + + +@pytest.mark.parametrize("flag, content", [("--outcomes", '[{"trace": "t"}]'), ("--contract", "a: [")]) +def test_cli_refuses_malformed_files_cleanly(tmp_path, flag, content): + (tmp_path / "t.jsonl").write_text(_trace(READ)) + f = tmp_path / "bad" + f.write_text(content) + args = build_parser().parse_args(["context", "consistency", str(tmp_path / "t.jsonl"), flag, str(f)]) + with pytest.raises(SystemExit) as exc: + args.func(args) + assert str(exc.value).startswith("✗") + + +def test_an_outcome_key_shared_by_two_arms_must_be_the_full_path(tmp_path): + a, b = _run(str(tmp_path / "base/1.jsonl"), READ), _run(str(tmp_path / "skill/1.jsonl"), READ) + with pytest.raises(ValueError, match="1.jsonl"): + apply_outcomes([a, b], {"1.jsonl": False}) + assert apply_outcomes([a, b], {a.name: False, b.name: True}) == [] + assert (a.passed, b.passed) == (False, True) + + +def test_a_directory_name_is_not_an_outcome_key(tmp_path): + r = _run(str(tmp_path / "base/1.jsonl"), READ) + assert apply_outcomes([r], {"base": False}) == [r.name] and r.passed is None + + +def test_cli_catches_an_outcome_key_shared_across_arms(tmp_path): + for arm in ("base", "skill"): + (tmp_path / arm).mkdir() + (tmp_path / arm / "1.jsonl").write_text(_trace(READ)) + o = tmp_path / "o.json" + o.write_text('{"1.jsonl": false}') + args = build_parser().parse_args(["context", "consistency", "--arm", f"base={tmp_path}/base/*.jsonl", + "--arm", f"skill={tmp_path}/skill/*.jsonl", "--outcomes", str(o)]) + with pytest.raises(SystemExit) as exc: + args.func(args) + assert "1.jsonl" in str(exc.value) + + +def test_a_trace_given_twice_counts_once(tmp_path, capsys): + (tmp_path / "a").mkdir() + (tmp_path / "a/r1.jsonl").write_text(_trace(READ, TEST)) + (tmp_path / "a/r2.jsonl").write_text(_trace(READ, ("Bash", {"command": "git push"}))) + import yaml + contract = tmp_path / "c.yaml" + contract.write_text(yaml.safe_dump({"actions": CONTRACT["actions"], "rules": [{"forbids": "push"}]})) + args = build_parser().parse_args([ + "context", "consistency", "--arm", f"A={tmp_path}/a/*.jsonl", "--arm", f"A={tmp_path}/a/r2*.jsonl", + f"{tmp_path}/a/./r1.jsonl", f"{tmp_path}/a/r1.jsonl", "--contract", str(contract), "--json"]) + args.func(args) + out = capsys.readouterr() + rep = json.loads(out.out) + assert rep["A"]["result"]["runs"] == 2 and rep["A"]["contract"]["conformance_rate"] == 0.5 + assert rep["runs"]["result"]["runs"] == 1 + assert out.err.count("given twice") == 2 + + +# --- review of #155, test gaps: each rule and denominator fails when the code stops enforcing it --- + +def test_forbids_fires_on_a_run_that_pushes(): + c = parse_contract(CONTRACT) + push = ("Bash", {"command": "git push origin main"}) + assert "forbids: push" in check_contract(_run("p", READ, EDIT, TEST, push), c) + assert "forbids: push" not in check_contract(_run("ok", READ, EDIT, TEST), c) + + +def test_before_needs_a_strictly_earlier_action(): + both = {"actions": {"any": {"tool": "Read"}, "guide": {"tool": "Read", "file_path": r"AGENTS\.md$"}}, + "rules": [{"before": ["any", "guide"]}]} + # the one Read is both actions: it is not earlier than itself + assert check_contract(_run("r", READ), parse_contract(both)) == ["before: any -> guide"] + assert check_contract(_run("r", GREP, ("Read", {"file_path": "/w/x.py"}), READ), parse_contract(both)) == [] + + +def test_after_last_looks_past_the_last_action_not_the_first(): + c = parse_contract({"actions": {"edit": {"tool": "Edit"}, "test": {"tool": "Bash", "command": "pytest"}}, + "rules": [{"after_last": ["edit", "test"]}]}) + assert check_contract(_run("r", EDIT, TEST, EDIT), c) == ["after_last: edit -> test"] + assert check_contract(_run("r", EDIT, TEST, EDIT, TEST), c) == [] + assert check_contract(_run("r", TEST), c) == [] # vacuous with no edit + + +def test_passed_but_nonconforming_counts_passes_only(): + c = parse_contract(CONTRACT) + runs = [_run("pass-bad", READ, TEST, EDIT), _run("fail-bad", READ, TEST, EDIT, ok=False), + _run("pass-ok", READ, EDIT, TEST)] + p = profile(runs, c)["contract"] + assert p["passed_but_nonconforming"] == 1 and p["conforming_runs"] == 1 + + +def test_result_rates_count_judged_runs_only(): + runs = [_run("a", READ), _run("b", READ), _run("c", READ, ok=False), _run("d", READ)] + runs[3].passed = None # no verdict for this one + r = profile(runs)["result"] + assert (r["runs"], r["judged"], r["passed"]) == (4, 3, 2) + assert r["success_rate"] == pytest.approx(2 / 3) + assert r["pass^2"] == pytest.approx(1 / 3) and r["pass@2"] == pytest.approx(1.0) + assert "pass^4" not in r # k above the judged runs has no estimate + + +def test_run_from_events_marks_failed_commands_and_denied_tools(): + run = run_from_events([ + {"kind": "command", "text": "make check && git diff", "rc": 2}, + {"kind": "command", "text": "pytest", "rc": 0}, + {"kind": "command", "text": "ls", "rc": None}, + {"kind": "tool", "name": "WebFetch", "denied": True}, + {"kind": "tool", "name": "Task", "denied": False}, + {"kind": "read", "path": "/w/AGENTS.md"}, + {"kind": "command", "text": "sub", "rc": 1, "agent": "x"}, + ]) + assert run.kinds() == ["Bash:make", "Bash:git", "Bash:pytest", "Bash:ls", "WebFetch", "Task", "Read"] + assert [e.failed for e in run.events] == [True, True, False, False, True, False, False] + + +# --- documented paths and 911a507 guards no test reached ------------------------------------------ + +def test_a_file_name_or_stem_key_sets_the_outcome(tmp_path): + a, b = _run(str(tmp_path / "x/a.jsonl"), READ), _run(str(tmp_path / "x/b.jsonl"), READ) + assert apply_outcomes([a, b], {"a.jsonl": False, "b": False}) == [] + assert (a.passed, b.passed) == (False, False) + o = tmp_path / "o.json" + o.write_text('[{"trace": "t1.jsonl", "passed": false}]') + assert load_outcomes(o) == {"t1.jsonl": False} + + +def test_all_failed_is_a_stable_failure_and_no_verdict_is_unjudged(): + runs = [_run(str(i), READ, ok=False) for i in range(3)] + assert profile(runs)["strategies"][0]["verdict"] == "stable failure" + for r in runs: + r.passed = None + assert profile(runs)["strategies"][0]["verdict"] == "unjudged" + + +@pytest.mark.parametrize("cmd, want", [ + ("( make test )", ["Bash:make"]), ("git log --grep=#12 && make test", ["Bash:git", "Bash:make"]), +]) +def test_a_bare_paren_and_a_hash_inside_a_word(cmd, want): + assert _kinds(cmd) == want + + +def test_a_shell_write_is_not_a_read_and_fd_one_is_stdout(): + assert _written("echo x 1> out.txt") == "out.txt" + c = parse_contract({"actions": {"guide": {"tool": "Read", "file_path": r"AGENTS\.md$"}}, + "rules": [{"requires": "guide"}]}) + assert check_contract(_run("r", ("Bash", {"command": "echo x > AGENTS.md"})), c) == ["requires: guide"] diff --git a/dokimi/tests/test_context_shell.py b/dokimi/tests/test_context_shell.py new file mode 100644 index 00000000..0e275218 --- /dev/null +++ b/dokimi/tests/test_context_shell.py @@ -0,0 +1,167 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""`context consistency` reads one Bash call as the commands it runs (dokimi.context.shell).""" + +from __future__ import annotations + +import json + +import pytest + +from dokimi.context.consistency import (Matcher, apply_outcomes, check_contract, load_outcomes, parse_contract, + parse_run, profile, run_from_events, _events) +from dokimi.context.shell import program, split_commands, written + + +def _steps(command: str) -> list[tuple[str, str]]: + return [(e.kind, e.action) for e in _events("Bash", {"command": command})] + + +@pytest.mark.parametrize("command, steps", [ + ("for f in a b; do black $f; done", [("Bash:black", "Bash:black(black $f)")]), + ("for f in a; do black x; done", [("Bash:black", "Bash:black(black x)")]), + ("if [ -f x ]; then make; fi", [("Bash:make", "Bash:make(make)")]), + ("if pytest -q; then make; fi", [("Bash:pytest", "Bash:pytest(pytest -q)"), ("Bash:make", "Bash:make(make)")]), + ("(cd sub && pytest -q)", [("Bash:pytest", "Bash:pytest(pytest -q)")]), + ("# note && x\npytest", [("Bash:pytest", "Bash:pytest(pytest)")]), + (r"find . -name '*.pyc' -exec rm {} \; -print", + [("Bash:find", r"Bash:find(find . -name '*.pyc' -exec rm {} \; -print)")]), + (r"echo a\;b; ls", [("Bash:echo", r"Bash:echo(echo a\;b)"), ("Bash:ls", "Bash:ls(ls)")]), + ("echo don\\'t; ls", [("Bash:echo", "Bash:echo(echo don\\'t)"), ("Bash:ls", "Bash:ls(ls)")]), + ("x=$(echo a && echo b); pytest", [("Bash:pytest", "Bash:pytest(pytest)")]), + ("x=$(git rev-parse HEAD; ls) && pytest", [("Bash:pytest", "Bash:pytest(pytest)")]), + ("X=$(git rev-parse HEAD) && pytest", [("Bash:pytest", "Bash:pytest(pytest)")]), + ("echo `a; b`; ls", [("Bash:echo", "Bash:echo(echo `a; b`)"), ("Bash:ls", "Bash:ls(ls)")]), + ("make & pytest", [("Bash:make", "Bash:make(make)"), ("Bash:pytest", "Bash:pytest(pytest)")]), + ("make 2>&1 && pytest", [("Bash:make", "Bash:make(make 2>&1)"), ("Bash:pytest", "Bash:pytest(pytest)")]), + ("case $x in a) make ;; b) ls ;; esac", [("Bash:make", "Bash:make(make)"), ("Bash:ls", "Bash:ls(ls)")]), + ("case $x in\n a) make ;;\n b) ls ;;\nesac", [("Bash:make", "Bash:make(make)"), ("Bash:ls", "Bash:ls(ls)")]), + ("timeout 300 pytest", [("Bash:pytest", "Bash:pytest(timeout 300 pytest)")]), + ("timeout -k 5 300 pytest", [("Bash:pytest", "Bash:pytest(timeout -k 5 300 pytest)")]), + ("nohup pytest", [("Bash:pytest", "Bash:pytest(nohup pytest)")]), + ("nice -n 10 pytest", [("Bash:pytest", "Bash:pytest(nice -n 10 pytest)")]), +]) +def test_control_syntax_and_wrappers_are_not_steps(command, steps): + assert _steps(command) == steps + + +def test_a_separator_inside_quotes_or_substitution_does_not_split(): + assert [s for s, _ in split_commands("echo 'a;b' \"c && d\" $(e; f) `g; h`")] == [ + "echo 'a;b' \"c && d\" $(e; f) `g; h`"] + + +def test_a_chained_line_reports_the_separator_after_each_command(): + assert split_commands("a && b || c; d & e\nf") == [ + ("a", "&&"), ("b", "||"), ("c", ";"), ("d", "&"), ("e", "\n"), ("f", "")] + + +@pytest.mark.parametrize("command, file", [ + ("sed -i s/a/b/ f", "f"), ("FOO=1 sed -i s/a/b/ f", "f"), ("sudo sed -i s/a/b/ f", "f"), + ("env FOO=1 sed -i s/a/b/ f", "f"), ("uv run sed -i s/a/b/ f", "f"), ("sed --in-place s/a/b/ f", "f"), + ("sed -ni s/a/b/ f", "f"), ("perl -pi -e s/a/b/ src/a.py", "src/a.py"), ("perl -Mstrict -e 'print 1' foo.pl", ""), + ("sudo tee x", "x"), ("echo hi | sudo tee x", "x"), ("echo hi | tee -a x", "x"), + ("echo tee x", ""), ("echo tee out.txt", ""), ("sed -i s/a/b/", ""), + ("echo x >| out.txt", "out.txt"), ("echo x > f", "f"), ("echo 3 > count.txt", "count.txt"), + ("seq 1 10 > out.txt", "out.txt"), ("pytest 2>/dev/null", ""), ("pytest 2> err.log", ""), + ("echo a2>/dev/null 2 > f", "f"), +]) +def test_written_file_after_prefixes_and_flags(command, file): + assert written(command) == file + + +def test_a_shell_write_is_an_edit_for_edit_matchers_only(): + ev = _events("Bash", {"command": "sed -i s/a/b/ f"})[0] + assert Matcher(tools=("Edit",))(ev) and not Matcher(tools=("Read",))(ev) + assert not Matcher(tools=("Edit",))(_events("Bash", {"command": "pytest > test.log"})[0]) + + +def test_wrapper_programs_are_found(): + assert program("timeout 300 uv run python -m pytest") == "pytest" + assert program("FOO=1 nice -n 10 nohup make") == "make" + + +def _trace(*calls, result=True, is_error=False): + lines = [] + for i, (tool, args, failed) in enumerate(calls): + lines.append({"type": "assistant", "message": {"id": f"m{i}", "content": [ + {"type": "tool_use", "id": f"t{i}", "name": tool, "input": args}]}}) + lines.append({"type": "user", "message": {"content": [ + {"type": "tool_result", "tool_use_id": f"t{i}", "content": "x", "is_error": failed}]}}) + if result: + lines.append({"type": "result", "subtype": "success", "is_error": is_error, "num_turns": len(calls)}) + return "\n".join(json.dumps(x) for x in lines) + + +LAST_OK = parse_contract({"actions": {"test": {"tool": "Bash", "command": "pytest"}}, "rules": [{"last_ok": "test"}]}) + + +@pytest.mark.parametrize("command, hidden", [ + ("pytest -q; echo done", True), ("pytest -q || true", True), ("pytest -q\ngit status", True), + ("pytest -q & echo done", True), ("pytest -q && git status", False), ("pytest -q", False), + ("pytest -q | tail -5", True), ("set -e; pytest -q; echo done", False), + ("set -e; pytest -q || true", True), ("for f in a; do pytest $f; done", False), +]) +def test_a_later_command_hides_the_status_of_an_earlier_one(command, hidden): + for grain in ("split", "call"): + run = parse_run(_trace(("Bash", {"command": command}, False)), grain=grain) + assert (check_contract(run, LAST_OK) != []) is hidden, (command, grain) + + +def test_a_chained_call_is_one_call(): + run = parse_run(_trace(("Bash", {"command": "cd x && make && make test && git diff"}, True), + ("Read", {"file_path": "/w/a"}, False))) + assert len(run.events) == 3 + 1 and run.calls() == 2 and run.failed_calls() == 1 + path = profile([run, run])["path"] + assert path["tool_calls_mean"] == 2 and path["failed_calls_mean"] == 1 + + +def test_a_run_that_made_calls_but_wrote_no_result_failed(): + died = parse_run(_trace(("Read", {"file_path": "/w/a"}, False), result=False)) + done = parse_run(_trace(("Read", {"file_path": "/w/a"}, False))) + assert died.passed is False and done.passed is True + rep = profile([done, died, died, died])["result"] + assert (rep["judged"], rep["passed"], rep["success_rate"]) == (4, 1, 0.25) + assert parse_run("").passed is None # nothing happened: nothing to judge + + +@pytest.mark.parametrize("contract, needle", [ + ({"actions": {"a": {"tool": "Bash"}}, "rules": [{"requires": 5}]}, "requires"), + ({"actions": {"a": {"tool": "Bash"}}, "rules": [{"requires": [["a"]]}]}, "requires"), + ({"actions": {"a": {"tool": "Bash"}}, "rules": [{"requires": {"a": 1}}]}, "requires"), + ({"actions": {"a": {"tool": 5}}, "rules": []}, "tool"), + ({"actions": {"a": {"tool": [["Bash"]]}}, "rules": []}, "tool"), +]) +def test_a_malformed_rule_argument_or_tool_is_a_value_error(contract, needle): + with pytest.raises(ValueError, match=needle): + parse_contract(contract) + + +def test_load_outcomes_wants_booleans(tmp_path): + o = tmp_path / "o.json" + o.write_text('{"a": 1}') + with pytest.raises(ValueError, match="true/false"): + load_outcomes(o) + + +def test_an_outcome_key_may_be_a_stem(): + runs = [parse_run("", "dir/run1.jsonl"), parse_run("", "dir/run2.jsonl")] + assert apply_outcomes(runs, {"run1": True, "run2": False}) == [] + assert [r.passed for r in runs] == [True, False] + + +def test_a_deeply_nested_trace_line_is_skipped_not_fatal(): + raw = "[" * 200000 + "\n" + _trace(("Read", {"file_path": "/w/a"}, False)) + assert [e.kind for e in parse_run(raw).events] == ["Read"] + + +def test_repeated_steps_collapse_into_one_strategy_shape(): + a = parse_run(_trace(*[("Read", {"file_path": "/w/a"}, False)] * 3, ("Edit", {"file_path": "/w/b"}, False))) + b = parse_run(_trace(("Read", {"file_path": "/w/a"}, False), ("Edit", {"file_path": "/w/b"}, False))) + assert a.shape() == b.shape() == ["Read", "Edit"] + + +def test_events_from_a_normalized_trace_split_like_a_bash_call(): + run = run_from_events([{"kind": "command", "text": "make && pytest", "rc": 1}, {"kind": "read", "path": "a"}]) + assert [e.kind for e in run.events] == ["Bash:make", "Bash:pytest", "Read"] + assert run.calls() == 2 and run.failed_calls() == 1 diff --git a/dokimi/tests/test_repo_compare.py b/dokimi/tests/test_repo_compare.py index 4282ca7c..8e334e95 100644 --- a/dokimi/tests/test_repo_compare.py +++ b/dokimi/tests/test_repo_compare.py @@ -935,13 +935,3 @@ def test_an_unreadable_batch_stamp_is_no_stamp(tmp_path): _cell(b, "fix_a", "guidance:none", 1) _cell(b, "fix_a", "guidance:raw", 1) assert compare([b], "guidance:none")["stamps"]["b1"] == {"harness_versions": ["1.0"], "model_resolved": []} - - -def test_arms_from_batches_with_different_built_in_graders_warn(tmp_path): - old, new = tmp_path / "old", tmp_path / "new" - _stamp(old, tree="repo@abc", eval_dir="e1", image="img:1", graders="g1") - _stamp(new, tree="repo@abc", eval_dir="e1", image="img:1", graders="g2") - _cell(old, "fix_a", "guidance:raw", 1) - _cell(new, "fix_a", "guidance:raw", 1) - s = compare([old, new], "old/guidance:raw", ["new/guidance:raw"]) - assert s["warnings"] == ["new/guidance:raw vs old/guidance:raw: the batches ran from different graders (g1, g2)"] diff --git a/dokimi/tests/test_repo_compare_paths.py b/dokimi/tests/test_repo_compare_paths.py new file mode 100644 index 00000000..4d063e99 --- /dev/null +++ b/dokimi/tests/test_repo_compare_paths.py @@ -0,0 +1,173 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""`repo compare --pass-k` and `--path-stability`: the gate-only and trace-based measures.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from dokimi.cli import build_parser +from dokimi.repos.compare import CompareError, compare + +from test_repo_compare import _cell, _row, _stamp + + +def _events(batch: Path, task: str, arm: str, rep: int, *commands: str) -> None: + rd = batch / f"{task}__{arm.replace(':', '-')}__m-1__r{rep}" + (rd / "events.jsonl").write_text("".join(json.dumps({"kind": "command", "text": c, "rc": 0}) + "\n" + for c in commands)) + + +def test_pass_k_says_whether_a_task_passes_every_time(tmp_path): + b = tmp_path / "b" + # Same gate rate (2/3) on t1 in both arms; the treatment passes t2 every time instead of 2/3. + for arm, t1, t2 in (("none", [1, 1, 0], [1, 1, 0]), ("guidance:raw", [1, 0, 1], [1, 1, 1])): + for rep, (x, y) in enumerate(zip(t1, t2, strict=True)): + _cell(b, "t1", arm, rep, success=bool(x)) + _cell(b, "t2", arm, rep, success=bool(y)) + summary = compare([b], "none", pass_k=3) + row = _row(summary, "pass^3") + assert row["baseline_mean"] == 0.0 and row["treatment_mean"] == 0.5 and row["p_holm"] is None + assert {t["task"]: t["delta"] for t in row["tasks"]} == {"t1": 0.0, "t2": 1.0} + two = compare([b], "none", pass_k=2) + assert _row(two, "pass^2")["baseline_mean"] == round(1 / 3, 6) + short = compare([b], "none", pass_k=4) + assert not [r for r in short["comparisons"] if r["measure"] == "pass^4"] + assert {u["reason"] for u in short["unpaired"] if u["measure"] == "pass^4"} == {"too few cells in baseline or treatment for pass^4"} + assert "pass^3" not in {r["measure"] for r in compare([b], "none")["comparisons"]} + + +def test_path_stability_compares_how_much_the_cells_paths_differ(tmp_path): + b = tmp_path / "b" + for rep in range(2): + _cell(b, "t1", "none", rep) + _cell(b, "t1", "guidance:raw", rep) + _events(b, "t1", "none", 0, "grep -r x .", "pytest -q") + _events(b, "t1", "none", 1, "git status && make check") + _events(b, "t1", "guidance:raw", 0, "grep -r x . && pytest -q") # one call, the same two steps + _events(b, "t1", "guidance:raw", 1, "grep -rn y src", "pytest -q tests") + row = _row(compare([b], "none", path_stability=True), "path_distance") + assert row["baseline_mean"] == 1.0 and row["treatment_mean"] == 0.0 and row["delta"]["mean"] == -1.0 + assert row["tasks"][0]["n_baseline"] == 2 + + +def test_path_stability_needs_two_traced_cells(tmp_path): + b = tmp_path / "b" + for rep in range(2): + _cell(b, "t1", "none", rep) + _cell(b, "t1", "guidance:raw", rep) + _events(b, "t1", "none", 0, "pytest") + summary = compare([b], "none", path_stability=True) + assert not [r for r in summary["comparisons"] if r["measure"] == "path_distance"] + assert [u["reason"] for u in summary["unpaired"] if u["measure"] == "path_distance"] == [ + "too few cells in baseline or treatment for path_distance"] + + +def test_pass_k_must_be_positive(tmp_path): + _cell(tmp_path / "b", "t1", "none", 0) + with pytest.raises(CompareError, match="at least 1"): + compare([tmp_path / "b"], "none", pass_k=0) + + +def test_cli_prints_pass_k_and_path_distance(tmp_path, capsys): + b = tmp_path / "b" + for rep in range(2): + _cell(b, "t1", "none", rep) + _cell(b, "t1", "guidance:raw", rep, success=rep == 0) + for arm in ("none", "guidance:raw"): + _events(b, "t1", arm, rep, "pytest -q") + args = build_parser().parse_args(["repo", "compare", str(b), "--baseline", "none", "--pass-k", "2", + "--path-stability"]) + args.func(args) + out = capsys.readouterr().out + assert "pass^2" in out and "path_distance" in out + + +def test_arms_from_batches_with_different_built_in_graders_warn(tmp_path): + old, new = tmp_path / "old", tmp_path / "new" + _stamp(old, tree="repo@abc", eval_dir="e1", image="img:1", graders="g1") + _stamp(new, tree="repo@abc", eval_dir="e1", image="img:1", graders="g2") + _cell(old, "fix_a", "guidance:raw", 1) + _cell(new, "fix_a", "guidance:raw", 1) + s = compare([old, new], "old/guidance:raw", ["new/guidance:raw"]) + assert s["warnings"] == ["new/guidance:raw vs old/guidance:raw: the batches ran from different graders (g1, g2)"] + + +def _raw_events(batch: Path, task: str, arm: str, rep: int, text: str) -> None: + rd = batch / f"{task}__{arm.replace(':', '-')}__m-1__r{rep}" + (rd / "events.jsonl").write_text(text) + + +def test_an_empty_or_unparseable_trace_is_not_a_steady_path(tmp_path): + b = tmp_path / "b" + for rep in range(3): + _cell(b, "t1", "none", rep) + _cell(b, "t1", "guidance:raw", rep) + for rep, cmd in enumerate(("grep x .", "git status", "make check")): + _events(b, "t1", "none", rep, cmd) + _raw_events(b, "t1", "guidance:raw", 0, "") + _raw_events(b, "t1", "guidance:raw", 1, "not json\n[1, 2]\n5\n") + _events(b, "t1", "guidance:raw", 2, "pytest") + summary = compare([b], "none", path_stability=True) + assert not [r for r in summary["comparisons"] if r["measure"] == "path_distance"] # one traced cell + + +def test_a_deeply_nested_line_skips_that_line_not_the_comparison(tmp_path): + b = tmp_path / "b" + for rep in range(2): + _cell(b, "t1", "none", rep) + _cell(b, "t1", "guidance:raw", rep) + _events(b, "t1", "none", rep, "pytest") + for rep in range(2): + _raw_events(b, "t1", "guidance:raw", rep, "[" * 200000 + "\n" + json.dumps( + {"kind": "command", "text": "pytest", "rc": 0}) + "\n") + row = _row(compare([b], "none", path_stability=True), "path_distance") + assert row["treatment_mean"] == 0.0 and row["tasks"][0]["n_treatment"] == 2 + + +def test_path_distance_is_the_mean_over_every_pair_of_three_cells(tmp_path): + b = tmp_path / "b" + for rep in range(3): + _cell(b, "t1", "none", rep) + _cell(b, "t1", "guidance:raw", rep) + _events(b, "t1", "guidance:raw", rep, "pytest") + _events(b, "t1", "none", 0, "pytest") + _events(b, "t1", "none", 1, "pytest") + _events(b, "t1", "none", 2, "git status") # the third cell differs from both others + row = _row(compare([b], "none", path_stability=True), "path_distance") + assert row["baseline_mean"] == pytest.approx(2 / 3) # pairs: 0, 1, 1 + assert row["tasks"][0]["n_baseline"] == 3 + + +def test_pass_k_is_a_gate_measure_not_a_score_measure(tmp_path): + b = tmp_path / "b" + for rep in range(2): + _cell(b, "t1", "none", rep, success=None, score=0.5) + _cell(b, "t1", "guidance:raw", rep, success=None, score=0.7) + summary = compare([b], "none", pass_k=2) + assert not [r for r in summary["comparisons"] if r["measure"] == "pass^2"] + assert "score" in {r["measure"] for r in summary["comparisons"]} + + +def test_the_optional_measures_do_not_flood_unpaired_with_tasks_they_do_not_apply_to(tmp_path): + b = tmp_path / "b" + for i in range(15): # gate tasks with one cell per arm (the `repo run` default): too few for pass^2 + _cell(b, f"g{i}", "none", 0) + _cell(b, f"g{i}", "guidance:raw", 0) + for rep in range(2): # a score task: pass^K is a gate measure + _cell(b, "s1", "none", rep, score=1.0) + _cell(b, "s1", "guidance:raw", rep, score=1.0) + _cell(b, "only_base", "none", 0) + summary = compare([b], "none", pass_k=2, path_stability=True) + reasons = sorted((u["task"], u["reason"]) for u in summary["unpaired"]) + assert reasons == [ + ("15 task(s)", "too few cells in baseline or treatment for pass^2"), + ("16 task(s)", "too few cells in baseline or treatment for path_distance"), # the score task left no trace + ("only_base", "no cells in treatment"), + ] + assert not any(u["task"] == "s1" for u in summary["unpaired"]) + assert len(compare([b], "none")["unpaired"]) == 1