feat(repo): show result.json process counts in repo report and repo compare by napetrov · Pull Request #177 · intel/dokimi

napetrov · 2026-10-22T05:54:35Z

Part of #41 (follow-up to D4, #157).

Summary

repo report and repo compare now show the result.json process counts. #157 adds those counts; this PR is useful once #157 lands. It does not stack on #157: the code reads process defensively, so an absent or null process is skipped and older batches report and compare as they do today.

 dokimi/repos/
+├── process_counts.py   # process_values / process_means / process_order / process_line
 ├── report.py           # row["process"] = per-arm means; one "process ..." line under the row
 └── compare.py          # measures += process:<key>, counter:<name> (descriptive, not in Holm)

Measure names: process:{tool_calls,commands,failed_commands,denied_commands,guidance_read} (guidance_read is the number of paths read), plus counter:<name> for each manifest process_metrics counter.
In compare, a counter becomes a measure as soon as any compared cell reports it. Values are per-task means over the cells that have the counter, with a t interval and a raw p, the same way as cost and turns.
Docs: contract §9 (the report and compare paragraphs), architecture.md, and a scriv fragment.

Evidence

Before: repo compare showed only gate/score/cost_usd/turns/metric rows.
After: output on a synthetic batch:

  process:tool_calls           3     14.000     10.000  -4.000 [-4.000, -4.000]      0.250       -  0/3/0
  process:failed_commands      3      2.000      0.000  -2.000 [-2.000, -2.000]      0.250       -  0/3/0
  process:guidance_read        3      0.000      1.000  +1.000 [+1.000, +1.000]      0.250       -  3/0/0
  counter:bazel                3      1.000      3.000  +2.000 [+2.000, +2.000]      0.250       -  3/0/0

tests/test_repo_process_counts.py (synthetic result.json with and without process): measure order and deltas, p_holm None, null/absent skipped (not zero), non-counts ignored, report means and the single line. Full pytest -q: 7072 passed; ruff check . and mypy are clean. No function's radon CC went up.

Merge Danger

Door: two-way

The change only adds new keys and rows, and nothing is written to disk.

Blast Radius: narrow

A consumer that expects a fixed measure list in repo compare --json will see extra rows when cells carry process. There is a one-line import conflict with #153 in compare.py; it does not conflict with #155 or #157.

🤖 Generated with Claude Code

…ompare repo report gives each row the mean of every process count over the cells that have one; repo compare adds them as descriptive measures (process:<key>, counter:<name>), outside Holm's family. An absent or null process is skipped, so older batches compare as before. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

codecov-commenter · 2026-10-22T06:14:09Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.67%. Comparing base (5064dc2) to head (2614d0b).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #177      +/-   ##
==========================================
+ Coverage   91.64%   91.67%   +0.03%     
==========================================
  Files         222      223       +1     
  Lines       27409    27459      +50     
  Branches     4508     4517       +9     
==========================================
+ Hits        25118    25173      +55     
+ Misses       1620     1617       -3     
+ Partials      671      669       -2

Files with missing lines	Coverage Δ
dokimi/dokimi/repos/compare.py	`99.64% <100.00%> (+0.01%)`	⬆️
dokimi/dokimi/repos/process_counts.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/repos/report.py	`95.27% <100.00%> (+0.87%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

antonwolfy

Review at c301aa6. The measure plumbing is clean: Holm is untouched, null and absent process are skipped, not zeroed, and the report JSON only gains a key. One bug at the edge, where only one arm carries process, and one test gap. One nit on the docs.

What I ran, on a Linux review host at c301aa6:

make ci-local: 20 passed, 8 failed, 7 skipped. The 8 failures (ruff, zizmor, pre-commit, bandit, mypy, deptry, mkdocs, pip-audit) exited 127, tools missing from my venv: make dev stops at the tools install on this base, which #179 fixed on main. Static checks, mypy and the strict docs build pass on GitHub's CI at this head. The test step passed: 7073 passed, 21 skipped. Changed-line coverage 100% (48 lines).
A reproduction of the unpaired finding, with no model or network access.
A mutation run on the new tests: 34 mutants, 10 of which pass the 4 new tests (see the inline comment on the test).

Checked, no issues

Holm: PRIMARY is still gate and score, and the process rows keep p_holm None.
Scoping: only arms named in the compared selectors add process measures; there is no name clash with metric:, score or cost_usd.
Values: bool, NaN, strings and a null process are skipped; guidance_read counts the list length.
Report: summarize only adds a process key per row; a row without it prints no extra line.

antonwolfy · 2026-10-22T14:50:22Z

    base_vals = _task_values([c for c in cells if selector_matches(base_sel, c)], metrics)
    rows: list[dict[str, Any]] = []
    unpaired: list[dict[str, Any]] = []
+    measures += _process_measures(cells, [baseline, *treatments])


When only one arm carries process, every shared task becomes an unpaired entry for every process measure, and the real unpaired tasks scroll out of view.

Setup: 12 tasks, with process only on the treatment arm (the baseline comes from a batch written before process existed, the case the PR description says compares "as they do today"), plus one task zz that has no baseline cells.

compare([batch], "guidance:none")["unpaired"] has 73 entries, 1 of them real (zz, no cells in baseline). The other 72 are no process:tool_calls in baseline, no process:commands in baseline, and so on, 6 per task.

format_compare prints the first 10 and then and 63 more: unpaired: zz (m-1, no cells in baseline), a (m-1, no process:tool_calls in baseline), b (m-1, no process:tool_calls in baseline), ... and 63 more. With a few more tasks, zz is cut off.

--json unpaired grows by about 6 entries per task.

The PR introduces this: on the base the list holds only tasks that are really missing. test_process_absent_or_null_is_skipped (test_repo_process_counts.py:65) pins the per-task entry for the partial case, but not this all-tasks case.

Suggestion: in _process_measures, keep a measure only when both the baseline and at least one treatment report it for some task. Or keep a process measure's missing-side entries out of unpaired, since a descriptive measure absent from an arm is not a left-out task.

Fixed in 75bd017: _compare_model skips a process:/counter: measure when the baseline or the treatment reports it for no shared task, so an older baseline batch adds no unpaired entries. Test: test_process_on_one_arm_only_does_not_flood_unpaired (unpaired is just zz; fails on the old code). The partial case (one task null) still lists that task.

antonwolfy · 2026-10-22T14:50:22Z

+    summary = compare([tmp_path], "guidance:none")
+    assert [r["measure"] for r in summary["comparisons"]][3:] == ["process:tool_calls"]
+    assert _row(summary, "process:tool_calls")["n_tasks"] == 1
+    assert {(u["task"], u["measure"]) for u in summary["unpaired"]} == {("a", "process:tool_calls")}


The tests do not pin "a missing value is skipped, not counted as zero" within a task, or the selector scoping of _process_measures.

Mutation check: each of these passes all 4 tests in this file.

_process_lists: v.get(name) changed to v.get(name, 0.0), so a rep without the counter counts as 0 and drags the per-task mean down.

_process_measures: the selectors [baseline, *treatments] changed to [baseline], or to treatments, or the selector_matches filter changed to if True.

The fixtures give every rep of a task the same value, and the null case nulls a whole task, so the mean over reps is never checked. Only two arms are used, so scoping is never checked.

Test to add: one task with three reps where rep 3 has no process (expect the mean of the other two in compare's process:tool_calls row), and a batch with a third arm that carries a counter the compared arms lack (expect no counter:<name> row).

Added in 75bd017: test_a_missing_process_value_in_a_task_is_not_zero (rep 3 without process, mean of the other two) and test_process_measures_come_only_from_the_compared_arms (a third arm with its own counter yields no row). I did not rerun the full mutant set.

antonwolfy · 2026-10-22T14:50:22Z

 - The report gives, per (task, arm, model), passes over finished cells with a
  Wilson 95% interval (score-only tasks: the mean of `score.metric`), grader
  failures (counted as failures too), missing cells, and mean cost and turns.
+  When cells carry `result.json` `process` (§4.1), a line under the row gives


Nit: this sentence points to §4.1 for result.json process, but §4.1 at this PR's base describes events.jsonl and does not define process, its keys (tool_calls, failed_commands, guidance_read, counters) or the manifest process_metrics. Nothing at the base writes the field either, so until #157 lands the report and compare never show it.

Suggestion: merge after #157, so the §4.1 reference resolves and the field is written.

Agreed, and left as is on purpose: #157 is where §4.1 gains the process fields and where result.json starts writing them, and this PR only reads them. It should merge after #157; I did not reword the reference to avoid describing fields that are not in the contract at this base.

…are's unpaired list When the baseline batch predates result.json process, every shared task became an unpaired entry for every process measure and hid the tasks that were really missing. A process measure an arm reports for no task is now left out. Pins the missing-value-is-not-zero mean and the selector scoping. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov requested review from antonwolfy, jharlow-intel, vchamarthi and xaleryb as code owners October 22, 2026 05:54

napetrov mentioned this pull request Oct 22, 2026

ci: no-growth ratchet for file length and function complexity #93

Merged

5 tasks done

antonwolfy reviewed Oct 22, 2026

View reviewed changes

napetrov added 2 commits October 22, 2026 11:40

Merge remote-tracking branch 'intel/main' into process-in-report

2614d0b

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Navigation Menu

Repository navigation

feat(repo): show result.json process counts in repo report and repo compare - #177

feat(repo): show result.json process counts in repo report and repo compare#177
napetrov wants to merge 3 commits into
intel:mainfrom
napetrov:process-in-report

napetrov commented Oct 22, 2026

Uh oh!

codecov-commenter commented Oct 22, 2026 •

edited

Loading

Uh oh!

antonwolfy left a comment

Uh oh!

antonwolfy Oct 22, 2026

Uh oh!

napetrov Oct 22, 2026

Uh oh!

antonwolfy Oct 22, 2026

Uh oh!

napetrov Oct 22, 2026 •

edited

Loading

Uh oh!

antonwolfy Oct 22, 2026

Uh oh!

napetrov Oct 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

napetrov commented Oct 22, 2026

Summary

Evidence

Merge Danger

Uh oh!

codecov-commenter commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

antonwolfy Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

antonwolfy Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

antonwolfy Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Oct 22, 2026 •

edited

Loading

napetrov Oct 22, 2026 •

edited

Loading