[napetrov]

Summary

New context consistency: reads repeated Claude Code stream-json traces of one task and answers three questions apart. No model calls.

Axis	Question	Numbers
Result	solves it every time?	success_rate, pass@k, pass^k
Path	same steps?	mix_jsd, order_distance, action_distance, CV of calls/turns/tokens
Contract	kept the required process?	conformance_rate, violations per rule, passed_but_nonconforming

context consistency --arm A=glob --arm B=glob [--outcomes verdicts.json] [--contract contract.yaml] [--shell split|call]
  parse_run          # main-thread steps; Bash split per chained command; sed -i / > / tee carry file_path
                     # blocked Stop hook = failed Hook:Stop step
  group by order distance -> verdict per strategy (stable success / valid diversity / unstable / stable failure)
  check_contract     # requires, forbids, before, after_last, last_ok

Third commit fixes three bugs found while using it:

 tool_result is_error on a Bash call
-  mark last chained command failed      # `make check && git diff` passed last_ok
+  mark every chained command failed     # trace does not say which one
 step piped without pipefail (`pytest | tail`)
+  masked; last_ok -> "last_ok: test (status hidden by a pipe)"
 _written
-  regex over raw text                   # `echo 'a > b'` was a write
+  shlex punctuation_chars tokens

Fourth commit wires it into repo compare (item 4 of NEXT_STEPS), opt-in, descriptive, outside Holm:

 repo compare <batch> --baseline none
+  --pass-k K         # pass^K per task = C(c,K)/C(n,K); <K cells -> unpaired
+  --path-stability   # path_distance per task = mean pairwise order distance of events.jsonl paths
 measures: gate, score, [pass^K], [path_distance], cost_usd, turns, metric:*

events.jsonl is harness-agnostic, so path stability works for any harness repo run converts (consistency.run_from_events).

Docs: docs/run-consistency.md, docs/repo-evaluation-contract.md.

Evidence

• Used on 96 + 60 real runs (haiku/sonnet, skill vs script vs Stop-hook arms). Harness success was 100% everywhere; verifier outcomes and the contract separated arms (hook vs script p = 0.0006), and split mode exposed sonnet's single-Bash sed edits that call grain hid.
• Before: make check && git diff failing → check_contract == []; pytest -q | tail -5 → [].
  After: ["last_ok: test"]; ["last_ok: test (status hidden by a pipe)"]; with set -o pipefail → [].
• repo compare test: t1 2/3→2/3, t2 2/3→3/3: gate delta moves one task by 1/3; pass^3 shows baseline 0.0 vs treatment 0.5 (t2 became reliable). Path test: baseline cells grep,pytest vs git,make → 1.0; treatment grep && pytest vs grep; pytest → 0.0.
• Full suite: 6739 passed, 22 skipped; ruff, mypy clean. (Earlier run had one test_reach_timeout_kills_the_harness_process_tree[orphan] failure under -n auto load; passes alone and on main.)

Merge Danger

Door: two-way

New subcommand and module; repo compare gains two opt-in flags, default output unchanged.

Blast Radius: small

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.48263% with 63 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.48%. Comparing base (a44b3d0) to head (911a507).

Files with missing lines	Patch %	Lines
dokimi/dokimi/context/consistency.py	89.43%	24 Missing and 21 partials ⚠️
dokimi/dokimi/commands/context.py	78.31%	12 Missing and 6 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #155      +/-   ##
==========================================
- Coverage   91.54%   91.48%   -0.06%     
==========================================
  Files         222      223       +1     
  Lines       27369    27908     +539     
  Branches     4505     4632     +127     
==========================================
+ Hits        25054    25532     +478     
- Misses       1644     1678      +34     
- Partials      671      698      +27     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	92.50% <100.00%> (+0.05%)	⬆️
dokimi/dokimi/repos/compare.py	99.66% <100.00%> (+0.03%)	⬆️
dokimi/dokimi/commands/context.py	87.75% <78.31%> (-12.25%)	⬇️
dokimi/dokimi/context/consistency.py	89.43% <89.43%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 8b8bd09. The metrics (pass@k, pass^k, JSD, order distance, the C(c,K)/C(n,K) form in repo compare) are right, and the default repo compare output is unchanged. The problems are in the shell-step parsing, in how empty or unreadable input is reported, and in the strength of the tests. Inline comments cover the first ones; the test gap is below.

What I ran, on a Linux review host at 8b8bd09:

• make ci-local: 24 passed, 2 failed, 6 skipped. 6759 tests passed, 1 failed: test_unbalanced_quote_in_hook_completes_quickly (1.25 s against a 1 s limit, under 8-worker load). pip-audit reports virtualenv 21.14.7 (CVE-2026-113722) from the existing locks, fixed on main by #152. Neither comes from this PR. The changed-line coverage gate passed.
• Reproductions for the code findings, with the LLM transport mocked (none of this code calls a model).
• A mutation run over the two new test files: 49 mutants applied, 21 pass the full suite.

Tests do not pin several rules. tests/test_context_consistency.py and tests/test_repo_compare.py survive these one-line mutants:

• forbids: ok = not a changed to ok = True passes. The test contract has forbids: push, but no fixture run pushes.
• before / after_last: a[0] < b[0] changed to <=, and a[-1] changed to a[0], both pass.
• passed_but_nonconforming: dropping the r.passed and condition passes.
• Metric denominators: success_rate and pass^k over n instead of judged pass, because no fixture has an unjudged run.
• run_from_events: rc != 0 -> False and denied -> False both pass, so failure marking for --path-stability is untested.
• repo compare: if pass_k and not score_task changed to if pass_k passes (no score-task test); order_distance(runs[:2]) instead of all runs passes (the tests use two cells per arm); except ValueError: continue changed to raise passes.
  Add a fixture per rule where the rule is violated, one arm with an unjudged run, and a three-cell path case.

Checked, no issues

• Metrics: pass_at / pass_hat are guarded by k <= judged, JSD stays in 0..1, and the single-linkage grouping and CV are safe on zero or one sample.
• Holm: the new measures are outside PRIMARY, so the family size is unchanged. With the flags absent no events.jsonl is read.
• Repo rules: the new page is in mkdocs.yml nav and the docs index, the changelog fragment has the scriv format, no schema or SCHEMA_VERSIONS entry applies, and no generated files are committed.
• Docs vs code: every flag and metric name in the docs matches the CLI help and the code.

[antonwolfy]

Shell control syntax becomes path steps, and two _written checks are wrong. _segments splits on ; and && without parsing compound commands, so keywords turn into steps (outputs of parse_run on one Bash call):

• for f in a b; do black $f; done -> Bash:for, Bash:do, Bash:done. There is no Bash:black.
• if [ -f x ]; then make; fi -> Bash:if, Bash:then, Bash:fi. There is no Bash:make.
• (cd sub && pytest) -> Bash:(cd, Bash:pytest).
• # note && x + newline + pytest -> Bash:x, Bash:pytest.

A contract rule that names Bash:make cannot match, and order_distance / action_distance / mix_jsd count the keywords, so identical strategies split.

In _written (consistency.py:274, outside this hunk's anchor): the words[i-1].isdigit() guard meant to skip 2> also skips echo 3 > count.txt and seq 1 10 > out.txt (no file_path). The in-place regex counts perl -Mstrict -e 'print 1' foo.pl as an edit of foo.pl. sed --in-place 's/a/b/' f.py is not seen as an edit.

Suggestion: drop the shell reserved words and leading ( / trailing ) in _program, treat only 2>, 2>> and &>-style fd redirects as non-writes, and match -i / --in-place flags exactly.

[napetrov]

Fixed in 911a507. Compound syntax is stripped from steps (for/do/done, if/then/else/fi, while, leading ( and trailing ), [ tests) and comments are dropped before splitting on && and ;. _written now treats only a digit glued to the operator (2>, 2>>) as an fd redirect, so echo 3 > count.txt writes, and the in-place flag is matched per tool: sed -i/-i.bak/-ni/--in-place, perl -pi, but not perl -Mstrict. Tests: test_control_syntax_is_not_a_step, test_written_review_cases.

[antonwolfy]

Thanks, the reserved words are no longer steps or programs, and _written is right now. The action and command text still keep them, though: for f in a; do black x; done gives Bash:black(do black x) against Bash:black(black x) for the bare command, seq_distance 1.0, and a command: ^black matcher misses the loop step. Details in the review body.

[antonwolfy]

@napetrov benchmark now fails on this PR:

ValueError: golden_manifest violation: matched 32 document(s) under .../dokimi, more than max_docs=31. include=['docs/**/*.md'] exclude=['docs/archive/**']

It passed at 8b8bd09 because main was older then. Since that run, #139 has added docs/research-register.md and set max_docs: 31 for it. This PR adds docs/run-consistency.md, and GitHub tests the PR merged into current main, so the count is 32. Merged as is, it would break benchmark on main.

Suggestion: in dokimi/benchmarks/spec.v1.yaml, set max_docs: 32 and change the comment's "Measured selection at the time of writing: 31 files" to 32. That file asks for the bump in the same change that adds the doc.

[napetrov]

Review at 8b8bd09, test gaps, all pinned in 911a507 (each checked by applying the one-line mutant and seeing a test fail):

• forbids: test_forbids_fires_on_a_run_that_pushes
• before strictness (a[0] < b[0]): test_before_needs_a_strictly_earlier_action
• after_last uses the last action: test_after_last_looks_past_the_last_action_not_the_first
• passed_but_nonconforming: test_passed_but_nonconforming_counts_passes_only
• success_rate and pass^k over judged runs, with one unjudged run: test_result_rates_count_judged_runs_only
• run_from_events rc and denied marking: test_run_from_events_marks_failed_commands_and_denied_tools
• pass_k only on gate tasks: test_pass_k_is_a_gate_measure_not_a_score_measure
• order_distance over all cells (three-cell case): test_path_distance_is_the_mean_over_every_pair_of_three_cells
• bad lines skipped, not raised: covered by the empty/unparseable and nested-line tests.
  Also: merged intel/main (a5ea88b), and benchmarks/spec.v1.yaml max_docs 31 -> 32 after recounting docs/**/*.md minus docs/archive (32), in 6148101; make benchmark-run passes locally.

[antonwolfy]

Review at 911a507. The fixes for the earlier threads hold: the "no runs" verdict, the clean error for malformed --outcomes, the refusal of a shared outcome key, the empty-trace and nested-line cases in repo compare, the max_docs bump and the --reps wording. The shell-step parser is still the weak part, and the new shell-edit rule has a side effect. Six new points below, plus one thread still open.

What I ran, on a Linux review host at 911a507:

• make ci-local: 27 passed, 1 failed, 7 skipped. 7220 tests passed. The one failure is pip-audit (gitpython 3.2.0, CVE-2026-114145, fixed in 3.2.1) from the existing locks; the PR changes no lock file. The changed-line coverage gate passed.
• Reproductions for the code findings, with no model involved (none of this code calls one).
• A mutation run over the two new test files: 72 mutants, 29 pass the full suite, 6 of those equivalent. The ones that matter are in point 2 below.

Still open from my earlier review:

• dokimi/dokimi/context/consistency.py:207: the program name is cleaned, but the step's action and command still carry the syntax words. for f in a; do black x; done gives the action Bash:black(do black x), a plain black x gives Bash:black(black x), and seq_distance between them is 1.0. So action_distance still splits identical strategies, and a matcher command: ^black does not match the loop step ([False] against [True] for the bare command). The same holds for if pytest -q; then ...; fi (Bash:pytest(if pytest -q)) and (cd x && pytest -q) (Bash:pytest(pytest))). Suggestion: build the action and command text from the words after _strip_syntax, not from the raw segment.

Findings that cannot be anchored inline:

1. repo compare --pass-k / --path-stability flood unpaired with entries that are not unpaired tasks. compare.py:292-296 adds an entry for every task that lacks the measure: score tasks (pass^K is gate-only), tasks with fewer than K cells, and tasks with fewer than two traced cells. The reason reads "no pass^2 in baseline or treatment", which is wrong for a score task. repo run defaults to --reps 1, so --path-stability on a default batch lists every task. Repro: 15 tasks with one cell per arm plus one task only in the baseline; compare(..., pass_k=2, path_stability=True) returns 31 unpaired entries, and the text shows ten "no pass^2 in baseline or treatment" lines and "and 21 more". Without the flags the same batch prints one line, zz_only_base (m-1, no cells in treatment), the only real one. Suggestion: keep a task out of unpaired when the measure does not apply to it, and report the count of tasks without the measure in one line.

2. Tests still do not pin several rules added in the fixes. Mutation check, one-line mutants that pass both new test files (140 tests):

• Matcher.__call__: dropping and set(self.tools) & _WRITE_TOOLS (so tool: Read also matches a shell write) passes.
• parse_run: except (ValueError, RecursionError) changed to except ValueError passes; only _cell_path has the nested-line test. A trace line of "[" * 100000 still aborts context consistency.
• _written: dropping prev != "1" or the (?<!\S) lookbehind in _fd_redirect, removing ">|" from the operators, len(files) >= 2 changed to >= 1 (so sed -i 's/a/b/' with no file reports a file), and tee without the pipe-or-start condition (echo tee out.txt becomes a write) all pass.
• apply_outcomes: removing the stem match passes. load_outcomes: accepting int values passes.
• Run.shape: if not out or out[-1] != k changed to if True passes, so nothing checks that repeats collapse.
  Add one case per line: _written("echo tee out.txt") == "", _written("sed -i 's/a/b/'") == "", _written("echo x >| f") == "f", a Read matcher against a sed -i step, a stem key for dir/run1.jsonl, load_outcomes on {"a": 1}, parse_run on a 200000-bracket line, and runs Read Read Read Edit / Read Edit as one strategy.

[antonwolfy]

_segments still mis-splits common shell: \;, $(...), backticks, & and case. It tracks quotes but not escapes outside them, nesting, or &. Outputs of _events("Bash", {"command": ...}):

• find . -name '*.pyc' -exec rm {} \; -print gives Bash:find(find ... {} \) with a dangling backslash, plus a bogus step Bash:-print.
• echo a\;b; ls gives Bash:echo(echo a\), Bash:b(b), Bash:ls.
• echo don\'t; ls takes \' as an opening quote, so ls is swallowed: one step.
• x=$(echo a && echo b); pytest gives Bash:a(x=$(echo a), Bash:echo(echo b)), Bash:pytest. x=$(git rev-parse HEAD; ls) && pytest gives the program rev-parse.
• make & pytest is one make step and pytest is lost.
• case $x in a) make ;; b) ls ;; esac gives one step, Bash:b); the make branch is gone. The multi-line form gives Bash:a) and Bash:b).

The bogus steps change order_distance / action_distance and break requires / before for a rule that names the real command. find -exec ... \; is in most agent traces.

Suggestion: walk the command once with a small state machine (backslash outside quotes, $(/backtick depth, & as a separator when not part of &&, >&, &>) and treat case labels like the other reserved words. Add one test per case above.

[antonwolfy]

Any shell redirect or tee now counts as an edit, so a contract that forbids or orders edit fires on test runs. The rule added for the edit action matches every Bash step with a file_path, and _written sets file_path for any > / tee target, not only for source edits. Matcher(tools=("Edit",)) returns True for:

• pytest > test.log
• pytest 2>&1 | tee test-output.txt
• git diff > changes.patch

So forbids: edit reports a violation for a run that only logged its output, and before: [guide, edit] is judged against the log write. The step's action is also rewritten to Bash:pytest(test-output.txt), which hides the command. The opposite miss: make &> build.log and make &>/dev/null have &> in _OPERATORS but are not seen by the write detection.

Suggestion: keep two things apart. Let file_path mean an in-place edit (sed -i, perl -pi) or a redirect to a path that is not a log or /dev/*, or give the step a separate writes field and let the edit rule use only the in-place forms; document the choice in docs/run-consistency.md. Add a test that pytest > test.log does not match edit.

[antonwolfy]

A scalar or nested rule argument still ends in a traceback, so the clean-error fix for --contract is incomplete. parse_contract calls list(arg or []) here and tuple(str(t) for t in tools) at line 424. The CLI catches only OSError and ValueError.

• rules: [{requires: 5}] raises TypeError: 'int' object is not iterable.
• rules: [{requires: [[a]]}] raises TypeError: unhashable type: 'list'.
• actions: {a: {tool: 5}} raises TypeError: 'int' object is not iterable.
• requires: {a: 1} is accepted and uses the dict keys as names.

It is the same failure class as the earlier thread (the traceback instead of ✗ ...), reached through a different input.

Suggestion: check that arg is a string or a list of strings, and that tool is a string or a list of strings, and raise ValueError(f"{where}: ...") otherwise. Add the three inputs above to test_malformed_outcomes_and_contract_are_value_errors.

[antonwolfy]

A run with no result event (crashed, timed out, truncated) drops out of success_rate, pass@k and pass^k instead of counting as a failure. parse_run leaves passed=None without a result event, and profile keeps only judged runs. The --outcomes path warns about unjudged runs; this path does not, and the text table shows runs but not judged.

Repro: four traces, one ending with a result success, three with tool calls and no result event. profile(...)["result"] gives {'runs': 4, 'judged': 1, 'passed': 1, 'success_rate': 1.0, 'pass@1': 1.0, 'pass^1': 1.0}. An arm where three of four runs died reads as 100% reliable. That is survivorship bias in the metric the PR is built to report.

Suggestion: when no --outcomes is given, count a trace that has events but no result event as failed (passed = False), or at least print a warning with the number of unjudged runs and show judged in the table. Add a test with a truncated trace.