feat(repo): process metrics in result.json, manifest process_metrics, grade optional for dokimi: graders

dokimi/changelog.d/20261021_140000_napetrov_d4-process-metrics.md

@@ -0,0 +1,15 @@
+### Added
+
+- **Process metrics in `result.json`** — every `repo run` cell's `result.json` has a
+  `process` object counted from `events.jsonl`: `tool_calls`, `commands`,
+  `failed_commands` (not denied, `rc` neither 0 nor null), `denied_commands`, and `guidance_read` (the
+  workspace paths read, not denied, that the manifest's guidance globs match). A manifest's new `process_metrics:` maps
+  a name to a regex, searched in each command's text, or its `output` (`match: output|any`),
+  and `process.counters` holds each count. `repo validate` refuses a regex that does not
+  compile, and a name that is not a string (a bare YAML `404:` or `on:`).
+
+### Changed
+
+- **`entry_points.grade` is optional** when every task's grader is a built-in `dokimi:` one;
+  `repo validate`, `repo run` and `repo tasks check` name the tasks that need it otherwise,
+  the last two before any agent runs.

dokimi/docs/repo-evaluation-contract.md

@@ -191,6 +191,10 @@ tasks: dev/agent_evals/tasks
 explore_tasks: null                       # optional ExploreBench YAML directory
 
 build_systems: [bazel, make]              # selects L0 claim-check adapters
+
+process_metrics:                          # optional repository counters, result.json `process.counters` (§4.1)
+  bazel_cmds: '\bbazel(?:isk)?\b'          # a plain regex searches the command text
+  icpx_missing: {regex: "icpx is not found", match: output}   # match: text (default), output or any
 ```
 
 Rules:
@@ -199,7 +203,17 @@ Rules:
 - `eval_dir`, `tasks` and `entry_points` come together or not at all. Without them the
   manifest is *guidance-only*: `scan` and `reach` run on it and `validate` checks its
   globs, while `tasks check` and `run` refuse it. `image` and `explore_tasks` need
-  `eval_dir`.
+  `eval_dir`, and so does `process_metrics`.
+- `entry_points.grade` is required only when some task's `grader` does not
+  start with `dokimi:` (a grader built into the framework); `repo validate`
+  names the tasks that need it.
+- `process_metrics` maps a name (letters, digits, `_`; not
+  starting with a digit) to a Python regex,
+  given as a string or as `{regex: <regex>, match: text|output|any}`. For each
+  name the framework counts the `command` events of the run whose `text`
+  (default), `output` or either contains a match (`re.search`); denied commands
+  count too, since the agent asked for them. `repo validate` refuses a regex
+  that does not compile.
 - `eval_dir` is a proper subdirectory, and `tasks` and `explore_tasks` sit inside it: it is the
   one directory deleted from every workspace (§3).
 - `{eval_dir}`, `{source_dir}`, `{run_dir}`, `{out}` and `{task}` are the only
@@ -384,6 +398,20 @@ Code's `modelUsage`), `main` when the harness reported the parent's only, and
 `null` when it reported none. `subagent_calls` counts the parent's and
 subagents' `Agent`/`Task` calls; `null` when the harness wrote no result record.
 
+`result.json` `process` holds what the framework counts in the events, the
+parent's and subagents' together, so a grader need not:
+
+| Key | Meaning |
+|---|---|
+| `tool_calls` | every event: each tool call the agent made |
+| `commands` | `command` events |
+| `failed_commands` | commands not denied whose `rc` is neither `0` nor `null` |
+| `denied_commands` | commands with `denied: true` (a hook or the harness's permissions refused them) |
+| `guidance_read` | sorted, distinct workspace paths (normalized: `./a`, `a/../b`; not `~`) of `read` events, neither denied nor failed (`rc` 1: the file does not exist), that `guidance.globs` match |
+| `counters` | each `process_metrics` name (§2) and its count |
+
+`process` is `null` when the trace could not be read, or the agent left none (no event and no result record).
+
 For a `patch:` arm with `added` files or `mcp_servers`, `result.json` has an
 `uptake` object: whether the agent used what the arm added. `files` and
 `mcp_servers` list what it added (sorted); `commands` counts
@@ -524,8 +552,8 @@ A review task with `pass: null` is scored, not gated: give it `score`. A `gate`
 on its metrics is not applied (`success` is null), and without `score` the run
 is a grader failure. A clean-diff control (no seeds) has `recall: null`, so it
 takes no `score`; its gate is the default `["pass"]`. The manifest's
-`entry_points.grade` is still required (`repo_manifest.v1`), even when every
-task uses a built-in grader.
+`entry_points.grade` is optional when every task's `grader` starts with
+`dokimi:`.
 
 Both review graders read the answer the same way. Its last parseable
 ` ```json ` fence (one nesting arrays or objects more than 100 deep counts

dokimi/dokimi/commands/repo_run.py

@@ -49,6 +49,7 @@ def _context(args: argparse.Namespace, out: Path, pin: bool = False):
     content = out / "_eval" / "content"
     # a pinned batch keeps its content until the pin is checked
     staged = export_eval_dir(tree, manifest, content.with_name("content.next") if pin else content)
+    _require_grade(staged, manifest, args.task)  # before the pin and the image build
     runner = DockerRunner(args.docker)
     image, ctx_dir = args.image, None
     if not image:
@@ -109,6 +110,21 @@ def context_digest(root: Path) -> str:
     return h.hexdigest()
 
 
+def _require_grade(content: Path, manifest: RepoManifest, ids: list[str] | None) -> None:
+    """Refuse the selected tasks when one needs ``entry_points.grade`` and the manifest has none."""
+    from dokimi.repos.manifest import ManifestError, TaskSummary, grade_problems
+    from dokimi.repos.workspace import PrepError, load_tasks
+
+    try:
+        tasks = load_tasks(content, manifest, ids)
+    except (OSError, ValueError, PrepError):
+        return  # reported where the run loads its tasks
+    graders = [(t.id, str(t.raw.get("grader") or "") if isinstance(t.raw, dict) else "") for t in tasks]
+    problems = grade_problems(manifest, [TaskSummary(i, "", "", g, ()) for i, g in graders])
+    if problems:
+        raise ManifestError(manifest.path, problems)
+
+
 def _uses_builtin_graders(content: Path, manifest: RepoManifest) -> bool:
     """Whether a task of the exported eval dir names a ``dokimi:`` grader; an unreadable task.json counts."""
     from dokimi.repos.graders import builtin_name

dokimi/dokimi/repos/events.py

@@ -150,6 +150,9 @@ def _apply(out: Transcript, rec: dict[str, Any], t: float | None,
                 denied = bool(block.get("is_error")) and bool(_HOOK_DENIAL.match(output or ""))
                 if denied:
                     result_of["denied"] = True
+                if result_of["kind"] == "read" and not denied:
+                    # A read that failed (the file does not exist) shows nothing of the file.
+                    result_of["rc"] = 1 if block.get("is_error") else 0
                 if result_of["kind"] == "command":
                     # Claude Code reports success or failure only, with stdout
                     # and stderr merged into one text.

dokimi/dokimi/repos/manifest.py

@@ -33,6 +33,7 @@
 
 from .graders import DATA_FILES, NAMES as BUILTIN_NAMES, PREFIX as BUILTIN_PREFIX
 from .host_spec import HostSpecError, parse_host
+from .process_metrics import Counter, parse_counter
 from .source import RepoTree
 
 SCHEMA_VERSION = "repo_manifest.v1"
@@ -53,7 +54,6 @@
 TASK_REQUIRED = ("id", "family", "base", "prompt", "grader")
 _PLACEHOLDER_RE = re.compile(r"\{([^{}]*)\}")
 
-
 # Run-dir entries the framework creates itself and the agent's container does not mount. An `image.env`
 # scratch directory with one of these names would hand the agent the framework's own entry. The
 # mounted ones (workspace, home, config) are not scratch and stay allowed: nothing new is mounted.
@@ -131,6 +131,7 @@ class RepoManifest:
     image: ImageSpec | None = None
     explore_tasks: str | None = None
     build_systems: tuple[str, ...] = ()
+    process_metrics: dict[str, Counter] = field(default_factory=dict)
 
     @property
     def runnable(self) -> bool:
@@ -167,6 +168,8 @@ def to_dict(self) -> dict[str, Any]:
             "tasks": self.tasks,
             "explore_tasks": self.explore_tasks,
             "build_systems": list(self.build_systems),
+            "process_metrics": {name: {"regex": c.pattern.pattern, "match": c.field}
+                                for name, c in self.process_metrics.items()},
         }
 
 
@@ -236,7 +239,7 @@ def parse_manifest(raw: Any, path: str = "<manifest>") -> RepoManifest:
         missing = [k for k in _RUN_KEYS if k not in raw]
         raise ManifestError(path, [f"{', '.join(run_keys)} without {', '.join(missing)}: a manifest "
                                    "that runs tasks needs all of eval_dir, tasks and entry_points"])
-    needs_eval = [k for k in ("image", "explore_tasks")
+    needs_eval = [k for k in ("image", "explore_tasks", "process_metrics")
                   if isinstance(raw, dict) and k in raw and not run_keys]
     if needs_eval:
         raise ManifestError(path, [f"{', '.join(needs_eval)} without eval_dir, tasks and entry_points: "
@@ -272,6 +275,7 @@ def parse_manifest(raw: Any, path: str = "<manifest>") -> RepoManifest:
     duplicates = sorted({n for n in cache_names if cache_names.count(n) > 1})
     if duplicates:
         errors.append(f"image/caches: duplicate cache name(s) {duplicates}")
+    counters = _parse_counters(raw.get("process_metrics") or {}, errors)
     if errors:
         raise ManifestError(path, errors)
 
@@ -292,9 +296,24 @@ def parse_manifest(raw: Any, path: str = "<manifest>") -> RepoManifest:
         ),
         explore_tasks=_norm(raw["explore_tasks"]) if raw.get("explore_tasks") else None,
         build_systems=tuple(raw.get("build_systems") or ()),
+        process_metrics=counters,
     )
 
 
+def _parse_counters(raw: dict[Any, Any], errors: list[str]) -> dict[str, Counter]:
+    """``process_metrics`` compiled; each one that does not compile adds to ``errors``."""
+    counters: dict[str, Counter] = {}
+    for name, value in raw.items():
+        if not isinstance(name, str):  # the schema's name pattern skips a non-string key
+            errors.append(f"process_metrics/{name!r}: a name is a string; YAML reads a bare 404, on or yes as another type")
+            continue
+        try:
+            counters[name] = parse_counter(name, value)
+        except ValueError as exc:
+            errors.append(str(exc))
+    return counters
+
+
 def check_repo(tree: RepoTree, manifest: RepoManifest,
                open_base: Callable[[str], RepoTree] | None = None) -> RepoCheckReport:
     """Check ``manifest`` against the snapshot in ``tree``.
@@ -336,6 +355,7 @@ def check_repo(tree: RepoTree, manifest: RepoManifest,
 
     if tree.is_dir(manifest.tasks):
         _check_tasks(tree, manifest, files, report)
+        report.errors.extend(grade_problems(manifest, report.tasks))
     else:
         report.errors.append(f"tasks directory does not exist: {manifest.tasks}")
     if open_base is None:
@@ -347,6 +367,17 @@ def check_repo(tree: RepoTree, manifest: RepoManifest,
     return report
 
 
+def grade_problems(manifest: RepoManifest, tasks: list[TaskSummary]) -> list[str]:
+    """``entry_points.grade`` is required only when some task's grader is not a built-in ``dokimi:`` one."""
+    if manifest.entry_points.get("grade"):
+        return []
+    own = sorted(t.id for t in tasks if not t.grader.startswith(BUILTIN_PREFIX))
+    if not own:
+        return []
+    return [f"entry_points/grade: required, since these tasks use the repository's own grader "
+            f"(not {BUILTIN_PREFIX}*): {', '.join(own)}"]
+
+
 def require_runnable(manifest: RepoManifest) -> None:
     """Refuse a guidance-only manifest where tasks are about to be prepared or run."""
     manifest.run_paths()

dokimi/dokimi/repos/process_metrics.py

@@ -0,0 +1,88 @@
+# Copyright (C) 2026 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+"""Process metrics of one run, from its ``events.jsonl`` events, for ``result.json`` ``process`` (contract §4.1).
+
+The framework counts what every repository asks of a trace:
+
+- ``tool_calls``: every event, so every tool call of the parent and its subagents;
+- ``commands``: the ``command`` events;
+- ``failed_commands``: commands not denied whose ``rc`` is neither ``0`` nor ``null`` (``null`` means
+  no result came back, which is not a failure; a permission denial carries ``rc`` 1 but never ran);
+- ``denied_commands``: commands a hook or the harness's permissions refused (``denied``);
+- ``guidance_read``: the sorted workspace paths of ``read`` events not denied that the manifest's guidance
+  globs match; a path outside the workspace (absolute, or with ``..``) is no repository file.
+
+``counters`` holds the manifest's ``process_metrics``: for each name, how many ``command`` events
+its regex finds a match in (``re.search``). A counter matches the command text unless it names
+``output`` (the command's output, as clipped in the event) or ``any`` (either).
+"""
+
+from __future__ import annotations
+
+import posixpath
+import re
+from collections.abc import Callable, Iterable, Mapping
+from dataclasses import dataclass
+from pathlib import PurePosixPath
+from typing import Any
+
+COUNTER_FIELDS = ("text", "output", "any")
+
+
+@dataclass(frozen=True)
+class Counter:
+    """One ``process_metrics`` entry: a compiled regex and the command field it searches."""
+
+    pattern: re.Pattern[str]
+    field: str = "text"
+
+    def hits(self, event: Mapping[str, Any]) -> bool:
+        fields = ("text", "output") if self.field == "any" else (self.field,)
+        return any(self.pattern.search(str(event.get(f) or "")) for f in fields)
+
+
+def parse_counter(name: str, value: Any) -> Counter:
+    """A ``process_metrics`` value, ``<regex>`` or ``{regex: <regex>, match: text|output|any}``.
+
+    Raises ``ValueError`` naming the metric when the regex does not compile or the shape is wrong."""
+    regex, where = (value, "text") if isinstance(value, str) else (
+        (value.get("regex"), value.get("match", "text")) if isinstance(value, dict) else (None, None))
+    if not isinstance(regex, str) or not regex:
+        raise ValueError(f"process_metrics/{name}: give a regex, or {{regex: <regex>, match: text|output|any}}")
+    if where not in COUNTER_FIELDS:
+        raise ValueError(f"process_metrics/{name}: match must be one of {', '.join(COUNTER_FIELDS)}, got {where!r}")
+    try:
+        return Counter(re.compile(regex), where)
+    except re.error as exc:
+        raise ValueError(f"process_metrics/{name}: not a valid regex: {exc}") from exc
+
+
+def _workspace_path(path: str) -> str | None:
+    """``path`` normalized (``./a``, ``a//b``, ``a/./b``), or None when it is not inside the workspace.
+
+    A ``..`` that stays inside (``src/../AGENTS.md``) is resolved; ``~`` is a home directory, not the repository."""
+    if path.startswith("~"):
+        return None
+    p = PurePosixPath(posixpath.normpath(path))
+    if p.is_absolute() or ".." in p.parts:
+        return None
+    return str(p)
+
+
+def process_metrics(events: Iterable[Mapping[str, Any]], is_guidance: Callable[[str], bool],
+                    counters: Mapping[str, Counter] | None = None) -> dict[str, Any]:
+    """The ``process`` object of ``result.json`` for one run's events; ``is_guidance`` tells a guidance path."""
+    events = list(events)
+    commands = [e for e in events if e.get("kind") == "command"]
+    # A read that failed (rc 1: the file does not exist) read no guidance.
+    reads = {w for e in events if e.get("kind") == "read" and e.get("path") and not e.get("denied")
+             and e.get("rc") in (0, None) and (w := _workspace_path(str(e["path"]))) is not None}
+    return {
+        "tool_calls": len(events),
+        "commands": len(commands),
+        "failed_commands": sum(e.get("rc") not in (0, None) and not e.get("denied") for e in commands),
+        "denied_commands": sum(bool(e.get("denied")) for e in commands),
+        "guidance_read": sorted(p for p in reads if is_guidance(p)),
+        "counters": {name: sum(c.hits(e) for e in commands) for name, c in (counters or {}).items()},
+    }

dokimi/dokimi/repos/runner.py

@@ -64,7 +64,8 @@
 from .grading import EvalContext, failed_outcome, outcome, run_entry
 from .harness_profile import HarnessProfile
 from .host import probe_host, unmet, validate_tasks
-from .manifest import scratch_name
+from .manifest import match_glob, scratch_name
+from .process_metrics import process_metrics
 from .workspace import PrepError, Task, prepare_workspace
 
 SUPPORT_HOST = Path(__file__).resolve().parent / "support"
@@ -193,6 +194,7 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path,
         "input_tokens": None, "output_tokens": None, "cache_read_tokens": None,
         "cache_write_tokens": None, "usage_scope": None, "subagent_calls": None,
         "plugins": [], "over_budget": False, "caps": caps, "network": network, "trace_errors": []}
+    process: dict[str, Any] | None = None  # null when the trace could not be read
     added = meta.get("arm_patch") or {}
     uptake_of = (added.get("added") or [], added.get("mcp_servers") or [])
     events = None  # the trace's events once it parsed; none: the uptake is unmeasured
@@ -207,6 +209,7 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path,
         _write_json(rd / "meta.json", meta)
         if transcript.events or transcript.num_turns is not None:  # else the agent never started
             events = transcript.events
+            process = run_process_metrics(ctx, events)
         cost = transcript.cost_usd
         agent.update(
             cost_usd=cost, charged_usd=cost if cost is not None else caps["budget_usd"],
@@ -227,7 +230,8 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path,
                               "model_resolved": meta.get("model_resolved"), "rep": rep,
                               "harness": opts.profile.name, "harness_version": meta["harness_version"],
                               "base_date": meta["base_date"], "score_task": bool(task.score),
-                              "score_max": task.score["max"] if task.score else None, "agent": agent, "grade": grade}
+                              "score_max": task.score["max"] if task.score else None, "agent": agent,
+                              "process": process, "grade": grade}
     if any(uptake_of):  # an arm that added something to use: did the agent use it (#41 D4)
         record["uptake"] = arm_uptake(events, *uptake_of, WORKSPACE)
     try:
@@ -238,6 +242,13 @@ def run_cell(ctx: EvalContext, task: Task, arm: str, rep: int, batch_dir: Path,
     return record
 
 
+def run_process_metrics(ctx: EvalContext, events: list[dict[str, Any]]) -> dict[str, Any]:
+    """``result.json`` ``process``: the framework's counts and the manifest's ``process_metrics`` (contract §4.1)."""
+    globs = ctx.manifest.guidance_globs
+    return process_metrics(events, lambda path: any(match_glob(g, path) for g in globs),
+                           ctx.manifest.process_metrics)
+
+
 class CellCrash(Exception):
     """A cell whose agent ran but whose ``result.json`` could not be written: its cost still counts."""
 

dokimi/dokimi/schemas/repo_manifest.v1.json

@@ -62,7 +62,6 @@
     "entry_points": {
       "type": "object",
       "additionalProperties": false,
-      "required": ["grade"],
       "properties": {
         "grade": {"type": "string", "minLength": 1},
         "oracle": {"type": "string", "minLength": 1},
@@ -75,6 +74,24 @@
       "type": "array",
       "uniqueItems": true,
       "items": {"enum": ["bazel", "make", "cmake", "pytest", "pre-commit"]}
+    },
+    "process_metrics": {
+      "type": "object",
+      "propertyNames": {"pattern": "^[A-Za-z_][A-Za-z0-9_]*$"},
+      "additionalProperties": {
+        "oneOf": [
+          {"type": "string", "minLength": 1},
+          {
+            "type": "object",
+            "additionalProperties": false,
+            "required": ["regex"],
+            "properties": {
+              "regex": {"type": "string", "minLength": 1},
+              "match": {"enum": ["text", "output", "any"]}
+            }
+          }
+        ]
+      }
     }
   }
 }