feat(repo): process metrics in result.json, manifest process_metrics, grade optional for dokimi: graders by napetrov · Pull Request #157 · intel/dokimi

napetrov · 2026-10-22T04:47:16Z

Part of #41 (step D4).

Summary

 run_cell
   convert(trace) -> events.jsonl
+  run_process_metrics(events, guidance.globs, manifest.process_metrics)
   run_entry("grade") -> grade
-  result.json {agent, grade}
+  result.json {agent, process, grade}

# result.json
process: {tool_calls: 10, commands: 4, failed_commands: 1, denied_commands: 1,
          guidance_read: [AGENTS.md, src/AGENTS.md], counters: {bazel_cmds: 2, icpx_missing: 1}}
# repo-eval.yaml
process_metrics:
  bazel_cmds: '\bbazel(?:isk)?\b'                             # command text (default)
  icpx_missing: {regex: "icpx is not found", match: output}   # text | output | any

New dokimi/repos/process_metrics.py; counts all events (parent and subagents). failed_commands = rc not 0/null; denied commands still count for counters (the agent asked for them), as in oneDAL's metrics.py.
repo validate refuses a regex that does not compile, a bad match, a bad name; process_metrics needs a runnable manifest.
entry_points.grade no longer schema-required (widening, v1 in place); check_repo requires it only when some task's grader does not start with dokimi:, and names those tasks. Generic, so it holds whether or not feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay #120 lands first.
Task host: blocks were already validated by repo validate (feat(repo): task host requirements, unsupported instead of a false pass (#41 D3) #112); added a test that pins it.
Contract: §2 manifest rules, §4.1 result.json process table. Nothing in §3/§5/§8.

Evidence

Before: result.json has no process; oneDAL recomputes counts in its own metrics.py; a manifest without entry_points.grade fails the schema.
After: tests/test_repo_process_metrics.py (each metric, counter fields, 6 regex/shape refusals, grade-optional rule, host validation) and test_repo_run.py asserting result["process"] on a real cell. make ci-local --offline: 0 failed, 15 passed, 11 skipped (network/docker). git merge-tree against repo scan: L0 rules for parity with oneDAL static_check (D6) #115 and feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay #120 heads: clean.

Merge Danger

Door: two-way

Additive key in result.json; schema widened only.

Blast Radius: small

Readers of result.json that reject unknown keys would see process; none in-tree.

Checklist

DCO sign-off
scriv fragment
contract doc updated
pytest, ruff, mypy pass

🤖 Generated with Claude Code

… grade optional for dokimi: graders result.json gains `process`, counted by the framework from events.jsonl: tool_calls, commands, failed_commands, denied_commands and guidance_read (read paths the guidance globs match). The manifest's process_metrics maps a name to a regex over command text, output or both; repo validate refuses a regex that does not compile. entry_points.grade is required only when some task's grader is not a built-in dokimi: one. Part of intel#41 (step D4). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

codecov-commenter · 2026-10-22T05:04:16Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.55%. Comparing base (a44b3d0) to head (3c8a0cc).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #157      +/-   ##
==========================================
+ Coverage   91.54%   91.55%   +0.01%     
==========================================
  Files         222      223       +1     
  Lines       27369    27423      +54     
  Branches     4505     4510       +5     
==========================================
+ Hits        25054    25108      +54     
  Misses       1644     1644              
  Partials      671      671

Files with missing lines	Coverage Δ
dokimi/dokimi/repos/manifest.py	`94.50% <100.00%> (+0.31%)`	⬆️
dokimi/dokimi/repos/process_metrics.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/repos/runner.py	`93.86% <100.00%> (+0.14%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

antonwolfy

Review at 343f7ac. The process-metrics code is clean and the counting rules match the contract. Two things are worth fixing before merge: the optional grade entry point has no runtime behind it at this head, and the one runner test cannot tell a working guidance_read / counters wiring from a broken one.

What I ran, on a Linux review host at 343f7ac:

make ci-local: 25 passed, 1 failed, 6 skipped. 6,731 tests, changed-line coverage passes. The failure is pip-audit on virtualenv 21.14.7 (CVE-2026-113722) from the existing locks, fixed on main by #152.
A reproduction of the first finding through repo run with the container runner faked.
A mutation run: 48 mutants of process_metrics.py, manifest.py and runner.py against the three test files that cover them; 18 survive, 17 of them real gaps (the second finding).

Checked, no issues

Schema: dropping required: ["grade"] only widens repo_manifest.v1; process_metrics is validated by both the schema and parse_counter.
Counting rules: failed_commands ignores rc null, denied commands still reach the counters, bad regex / bad match / bad name are refused with the metric named.
Offline: the new tests need no network or Docker.

antonwolfy · 2026-10-22T14:14:41Z

    return report


+def grade_problems(manifest: RepoManifest, tasks: list[TaskSummary]) -> list[str]:


A manifest without entry_points.grade and with only dokimi: graders passes repo validate, then every cell of repo run fails as a grader failure.

grade_problems returns [] for such a manifest, but nothing at this head runs a dokimi: grader. run_cell still calls run_entry(ctx, "grade", ...) (runner.py:216), and run_entry raises when the entry point is missing (grading.py:107). grep -rn 'dokimi:' dokimi/dokimi --include=*.py finds only the new BUILTIN_GRADER_PREFIX.

Reproduction: take the test_repo_run.py fixture, remove entry_points.grade, change the task graders to dokimi:calc, and run repo run --task fix_add. The cell's result.json grade is:

{"pass": null, "rc": 125, "status": "grader_failure", "success": false, "reasons": ["grade run failed: the manifest declares no `grade` entry point"]}

The description says the rule holds whether or not #120 lands first. It holds for validate, but not for run: a repository can pass validation and then pay for agent runs that all fail grading.

Suggestion: land this after the PR that adds the built-in grader dispatch, or keep grade required until then. If the PR stays as it is, add a test that runs a cell with a dokimi: grader and no grade entry point, so the gap is visible. The contract text at repo-evaluation-contract.md:197 also describes a built-in grader that does not exist at this head.

Resolved by merging intel/main (d9c92f4), which brings the #120 built-in grader dispatch: run_entry runs a dokimi: grader without entry_points.grade, so no runner change is needed. Pinned in 3c8a0cc by test_a_cell_graded_by_a_builtin_grader_needs_no_grade_entry_point (repo run on a manifest with no entry points, cell status graded).

antonwolfy · 2026-10-22T14:14:41Z

    assert result["grade"]["status"] == "graded" and result["grade"]["success"] is True
    assert result["agent"]["cost_usd"] == 0.01 and result["agent"]["num_turns"] == 3
    assert result["model_resolved"] == "m-1"
+    assert result["process"] == {"tool_calls": 1, "commands": 0, "failed_commands": 0, "denied_commands": 0,


The runner test cannot tell a working guidance_read / counters wiring from a broken one.

The fixture manifest has no counters and its trace reads no guidance file, so the assertion expects "guidance_read": [] and "counters": {}. run_process_metrics (runner.py:236) is therefore unpinned.

Mutation check: each of these passes test_repo_process_metrics.py, test_repo_manifest.py and test_repo_run.py (194 tests):

is_guidance as lambda path: False, and as lambda path: True;

globs = ();

ctx.manifest.process_metrics replaced by {};

process = {} in place of None for an unreadable trace.

Three smaller gaps in test_repo_process_metrics.py also survive:

match: any that searches only the text (x_anywhere, line 58, already matches both commands by text, so the output half is never needed);

counters run over all events instead of command events, since no non-command event in the fixture carries text or output;

failed_commands counting only rc == 1 or rc > 0, since no test uses another non-zero rc.

Suggestion: give the fixture manifest one process_metrics entry and one guidance glob, add a trace that reads a matching file, and assert non-empty guidance_read and counters in result.json. Make x_anywhere match only the output. Add a read event with a text field, an rc of -9, and a trace that fails to convert (process is None).

Added in 3c8a0cc: the fixture manifest has process_metrics (text and output counters) and the guidance glob; test_process_metrics_reach_result_json asserts non-empty guidance_read and counters in result.json, and test_an_unreadable_trace_leaves_process_null_and_still_grades covers process: null. Unit tests: test_any_matches_the_output_when_the_text_does_not, test_counters_search_command_events_only, test_failed_commands_counts_every_non_zero_rc (rc 2, -9, 137). I did not re-run the mutation set.

… cell without a grade entry point The runner fixture now has process_metrics and a guidance glob, with a trace that reads a guidance file and runs a failing command; a null process for an unreadable trace is covered. Process-metric unit tests gain output-only 'any' matches, command-only counting and non-1 rc values. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov requested review from antonwolfy, jharlow-intel, vchamarthi and xaleryb as code owners October 22, 2026 04:47

antonwolfy reviewed Oct 22, 2026

View reviewed changes

napetrov added 2 commits October 22, 2026 08:07

Merge remote-tracking branch 'intel/main' into sweep1022-157

d9c92f4

Navigation Menu

Repository navigation

feat(repo): process metrics in result.json, manifest process_metrics, grade optional for dokimi: graders - #157

feat(repo): process metrics in result.json, manifest process_metrics, grade optional for dokimi: graders#157
napetrov wants to merge 3 commits into
intel:mainfrom
napetrov:d4-process-metrics

napetrov commented Oct 22, 2026

Uh oh!

codecov-commenter commented Oct 22, 2026 •

edited

Loading

Uh oh!

antonwolfy left a comment

Uh oh!

antonwolfy Oct 22, 2026

Uh oh!

napetrov Oct 22, 2026

Uh oh!

antonwolfy Oct 22, 2026

Uh oh!

napetrov Oct 22, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return report


		def grade_problems(manifest: RepoManifest, tasks: list[TaskSummary]) -> list[str]:

Conversation

napetrov commented Oct 22, 2026

Summary

Evidence

Merge Danger

Checklist

Uh oh!

codecov-commenter commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

antonwolfy left a comment

Choose a reason for hiding this comment

Uh oh!

antonwolfy Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

antonwolfy Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

napetrov Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Oct 22, 2026 •

edited

Loading