feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay

dokimi/changelog.d/20261018_221020_napetrov_builtin-review-graders.md

@@ -0,0 +1,3 @@
+### Added
+
+- **Built-in review graders, `task.json` `grader: "dokimi:review_seeds"` and `"dokimi:review_replay"`** — a review task is now data only: `manifest.json` (seeds and decoys) or `gold.json` (human review threads) next to `task.json`, graded by framework code instead of the repository's `grade` and `oracle` entry points. `repo tasks check` and `repo run` use them with no other change. They run in the grading container on the image's `python3` (standard library only, 3.8 or newer), from a read-only mount at `/eval/graders` the agent's container never gets; an image without `python3` is a grader failure that says so. The matching rules are oneDAL's `review` and `review_replay` graders', with the same metrics for the same answers; `grade.json` records `details.grader_version`. An answer with no findings JSON is a failed run with `recall` 0 and `parse_error`, not a grader failure. Any other grader name goes to the repository's entry points as before, and the `dokimi:` prefix is reserved (contract §4.5).

dokimi/changelog.d/20261021_120000_napetrov_batch-pins-graders.md

@@ -0,0 +1,3 @@
+### Changed
+
+- **A batch is pinned to the built-in grader sources too.** When a task names a `dokimi:` grader, `repo run` hashes the built-in graders into `_eval/batch.json` (`graders`) next to `eval_dir` and `image`, and refuses to resume a batch after they change, as it does for an edit to `.agent-eval/grade.py`; `repo compare` warns when two batches ran from different graders. Before, a grader change moved scores (for example `replay_mul`) between cells of one batch with no warning. A batch stored before this change with such a task has no `graders` pin and is refused on resume: start a fresh `--batch` dir. A batch with no such task is not pinned to them and resumes as before.

dokimi/docs/repo-evaluation-contract.md

@@ -126,7 +126,7 @@ outside every workspace and passes its absolute path to entry points as
   `answer.txt` are written to `{run_dir}` after the agent's container exits.
 - `grade` and `oracle` run in a separate container from the same image, after
   the agent's container has exited. That container mounts `{eval_dir}` and
-  `{source_dir}` read-only and `{run_dir}` read-write. It gets a `HOME` of its
+  `{source_dir}` read-only and `{run_dir}` read-write (and, for a built-in grader (§4.5), the framework's grader code at `/eval/graders`, read-only). It gets a `HOME` of its
   own (`{run_dir}/grade-home`, made empty before each run; the agent never had
   it), `PYTHONNOUSERSITE=1`, and git configuration overrides that disable
   `core.fsmonitor` and hooks (`core.hooksPath=/dev/null`); the workspace's
@@ -139,8 +139,9 @@ outside every workspace and passes its absolute path to entry points as
   containers share. A grader that imports or runs workspace code runs the
   agent's code.
 - **Fixed paths.** Inside every container `{run_dir}` is `/eval/run`,
-  `{eval_dir}` is `/eval/content` and `{source_dir}` is `/eval/source`. The
-  agent container mounts the workspace at `/eval/run/workspace`, the same path
+  `{eval_dir}` is `/eval/content`, `{source_dir}` is `/eval/source`, and
+  the built-in graders' code is `/eval/graders` (grading container only, §4.5).
+  The agent container mounts the workspace at `/eval/run/workspace`, the same path
   graders see. An absolute path in an answer or grade therefore means the same
   thing after a run directory is archived or regraded on another host. On
   oneDAL, 14 of 308 copied runs regraded differently because build answers
@@ -243,15 +244,15 @@ Rules:
 ## 3. `task.json`
 
 The framework interprets the **core** and **workspace** keys. Every other key
-belongs to the repository's grader and is passed through unread.
+belongs to the repository's grader and is passed through unread, except `grader`, which the framework reads to tell a `dokimi:` name (§4.5) from a repository's.
 
 | Key | Required | Read by | Meaning |
 |---|---|---|---|
 | `id` | yes | framework | equals the directory name, and contains no whitespace or comma: the workflow's `tasks`, `smoke-tasks` and `exclude-tasks` inputs are lists of ids split on whitespace and commas, so `repo validate` refuses an id containing either |
 | `family` | yes | framework (reporting) | free string; common values: `build`, `fix`, `feature`, `test`, `review` |
 | `base` | yes | framework | commit to snapshot: a hex SHA, optionally followed by `^` (the parent, the usual form for a mined fix). Branch names, tags and `HEAD` are refused, because repetitions of one task must get one snapshot. Use the full 40-character SHA; a shorter one is accepted with a warning. It must resolve in the source. `meta.json` records it as `base_rev` and the resolved full SHA as `base_sha` |
 | `prompt` | yes | framework | what the agent is told |
-| `grader` | yes | repository | grader name, interpreted by the `grade` entry point |
+| `grader` | yes | framework for `dokimi:<name>` (§4.5); repository otherwise | grader name; any other name is interpreted by the `grade` entry point |
 | `setup_patches` | no | framework | applied after the snapshot, before the arm (for example an injected bug) |
 | `review_patch` | no | framework | applied with `git am` as the HEAD commit under review |
 | `gate` | no | framework | what counts as success; default `["pass"]`. Each entry is a boolean metric name, which must be `true`, or `{"metric": <name>, "min": <number>}` for a numeric metric |
@@ -468,27 +469,115 @@ against the repository. Every other key, per claim and at the top level, is
 free-form and stored with the scan. A fixed minimum lets L0 merge any
 repository's checks with its own without knowing their rules.
 
+### 4.5 Built-in graders
+
+A task whose `grader` is `dokimi:<name>` is graded by the framework, not by
+the repository's `grade` and `oracle` entry points; any other name goes to
+those entry points. Such a task is data only: its directory holds
+`task.json`, the patch and the JSON files the grader reads. The `dokimi:`
+prefix is the framework's: an unknown `dokimi:` name fails `tasks check` and
+the run's grade, and is never handed to the repository.
+
+A built-in grader runs where `grade` and `oracle` run: in the grading
+container, with the same mounts, network and environment (§4.2). The
+framework's grader code is mounted read-only at `/eval/graders` (never into
+the agent's container) and runs as
+`python3 -I -B /eval/graders/main.py grade|oracle <name> <task dir> /eval/run`,
+where `<task dir>` is the task's directory under `{eval_dir}`. It uses the
+standard library only and the image's `python3` (3.8 or newer). `-I` keeps
+`PYTHONPATH` from `image.env` and the workspace off `sys.path`; `-B` prevents
+writing `.pyc` files to the read-only mount. An image with no `python3` on
+`PATH` is a grader failure that says so: the grader exits 127 and its
+`<kind>.err` file shows `exec python3 failed`. Any
+other exit 127 is reported as a plain `grade exited 127`.
+
+- `grade` reads `task.json` and `answer.txt` from `{run_dir}` and the data
+  files from the task's directory, and writes `grade.json` (§4.2): numbers,
+  booleans and `null` under `metrics`; strings, lists and objects under
+  `details`, with `details.grader_version`. A grader that raises writes
+  `pass: null` with `errors`. A metric the task's `gate` names that the grader
+  did not produce is `null`.
+- `oracle` writes the reference answer to `answer.txt` and leaves the
+  workspace alone.
+
+A review task with `pass: null` is scored, not gated: give it `score`. A `gate`
+on its metrics is not applied (`success` is null), and without `score` the run
+is a grader failure. A clean-diff control (no seeds) has `recall: null`, so it
+takes no `score`; its gate is the default `["pass"]`. The manifest's
+`entry_points.grade` is still required (`repo_manifest.v1`), even when every
+task uses a built-in grader.
+
+Both review graders read the answer the same way. Its last parseable
+` ```json ` fence (one nesting arrays or objects more than 100 deep counts
+as unparseable, whatever `json.loads` accepts) must be an array of findings
+`{"file", "line", "severity", "rule", "message"}`; entries that are not
+objects are dropped. `file` is made repository-relative: a path is cut after
+`/eval/run/workspace/`, then after the last `/repo/` (a standalone oneDAL
+run's workspace; a relative `a/repo/b` becomes `b`, the seed's or thread's
+`file` as well as the finding's), then leading `./` and `/` go. `line` is
+read with Python's `int()` (`"6"` is 6, `5.9` is 5); a missing or falsy line
+(`null`, `0`, `""`), or a value `int()` cannot read, is -99, near nothing. `severity` `blocker` or `major` counts toward
+`n_severe`. An answer with no such array gives only `pass: false`,
+`parse_error: true` and every recall 0 (`recall`; replay also `recall_all`
+and `recall_addressed`; `null` when there is nothing to recall): the agent
+failed the task, and on a task with `score` that is a failed run with score
+0, not a grader failure.
+
+| `grader` | data next to `task.json` (key that renames it) | `pass` | `metrics` | `details` |
+|---|---|---|---|---|
+| `dokimi:review_seeds` | `manifest.json` (`manifest`); oracle answer `oracle.json` (`oracle_answer`), a list of findings | `null`; for a manifest with no seeds (a clean-diff control), `true` when no finding is `blocker` or `major` | `n_findings`, `n_severe`, `n_decoys`, `unmatched`, `recall` (`null` with no seeds) | `seeds_hit`, `decoys_flagged`, `recall_by_class` (`"hit/total"`) |
+| `dokimi:review_replay` | `gold.json` (`gold`); `task.json` `window`, default 5 | `null` (score-only) | `n_findings`, `n_severe`, `n_threads`, `recall` (= `recall_all`), `recall_addressed`, `precision_proxy` | `threads_hit`, `recall_by_kind`, `decoys_flagged` (always `[]`) |
+
+`manifest.json` is `{"seeds": [{id, class, file, lines, kw}], "decoys":
+[{id, file, lines, kw}]}`; `file` `"*"` matches any file, and `kw` is a
+regular expression searched case-insensitively in the finding's `rule`,
+`message` and `evidence`.
+
+- A finding on a seed's file hits every seed whose `kw` matches. When none
+  matches, it hits the one seed (not `"*"`) whose `lines` are nearest, if
+  within 1 line.
+- A finding that hits no seed is a false positive when it is on a decoy's
+  file and matches its `kw` or one of its `lines`; it is listed in
+  `decoys_flagged`. Otherwise it counts as `unmatched`.
+- `recall` is seeds hit / seeds.
+
+`gold.json` is `[{id, file, line, kind, comment, addressed}]`: one human
+review thread of a merged PR, anchored on a line of the reviewed revision.
+
+- A finding on a thread's file within `window` lines of its anchor can hit
+  it. Matching is one-to-one and nearest first: one finding credits at most
+  one thread, and one thread at most one finding.
+- `recall_addressed` counts only threads with `addressed: true` (the code
+  changed in response), `recall_by_kind` groups by `kind`, and
+  `precision_proxy` is the share of findings within the window of any
+  thread, a lower bound on precision. Humans miss problems and raise
+  non-defects, so neither is a verdict.
+- The oracle answer is one finding per thread, at its anchor.
+
 ## 5. Aligning oneDAL `dev/agent_evals` (uxlfoundation/oneDAL#3818)
 
-The harness stays in oneDAL and keeps working standalone. These changes make
-the same content runnable by the framework.
+oneDAL drops its standalone mode: the generic parts of its harness move into
+dokimi (#41), and `dev/agent_evals` keeps only its tasks, its image and the
+graders that know oneDAL. These changes make that content runnable by the
+framework.
 
 | Today | Change |
 |---|---|
 | guidance globs in `config.py` | move to `repo-eval.yaml` `guidance.globs` |
-| `task.json` keys | already match the core keys. Add `gate: ["strict_pass"]` to fix tasks, so a hidden-test pass that breaks a regression suite is not counted as success. Add `score: {metric: recall, max: 1.0}` to review tasks |
+| `task.json` keys | already match the core keys. Add `gate: ["strict_pass"]` to fix tasks, so a hidden-test pass that breaks a regression suite is not counted as success. Add `score: {metric: recall, max: 1.0}` to review tasks with seeds or threads, not to clean-diff controls (§4.5) |
 | `base` is a 8–12 character SHA | expand to the full 40-character SHA (keep a trailing `^`) |
 | entry points call `dev/agent_evals/run.py` | call `{eval_dir}/run.py`; the graders must not assume they sit inside the workspace (§1.1) |
-| `prep()` in `common.py` | unchanged for standalone use; under the framework the same steps run framework-side (§3) |
+| `prep()` in `common.py` | goes: the framework prepares every workspace (§3) |
 | arms `none` / `main-raw` | become `guidance:none` / `guidance:raw`. `guidance:bridge` is framework-only (§3); oneDAL keeps no bridge arm |
-| graders read history from `ONEDAL_EVAL_SRC` | read `{source_dir}` under the framework; the variable stays for standalone runs |
-| `contract.py` derives the base when `meta.json` has none | diff from `meta.json` `start_sha`, and read `base_sha` / `base_rev` for the upstream commit (§4); derive only standalone |
+| graders read history from `ONEDAL_EVAL_SRC` | read `{source_dir}`, the only full-history source; the variable goes |
+| `contract.py` derives the base when `meta.json` has none | diff from `meta.json` `start_sha`, and read `base_sha` / `base_rev` for the upstream commit (§4); nothing derives it |
 | `build_make_gnu` pip-installs MKL and TBB | install them in the image, or give the task `egress: [pypi.org, files.pythonhosted.org]` |
 | Bazel install base in the container's `HOME` | put it on a declared cache (`bin/bazel --output_user_root`) |
 | `static_check.py --out` writes `files` / `counts` / `claims` | add `"schema": "repo_static.v1"`; the claim fields already fit §4.4 |
-| `run.py grade <run_dir>` writes a flat dict | add `--contract` mode that reads §4 inputs and writes `repo_grade.v1` (scalar fields move under `metrics`, lists and objects under `details`) |
+| `run.py grade <run_dir>` writes a flat dict | `run.py grade --contract` reads the §4 inputs and writes `repo_grade.v1` (scalar fields under `metrics`, lists and objects under `details`); it stays for the graders that are not built in: `build.py` and `make.py`, which know oneDAL, and the fix and mutation graders until `dokimi:hidden_tests` and `dokimi:mutation` exist (#41) |
+| `graders/review.py` and `replay.py` (task `grader` `review`, `review_replay`) | review tasks name `dokimi:review_seeds` or `dokimi:review_replay` (§4.5) and keep `manifest.json`, `oracle.json` or `gold.json`; both files go. The built-in graders give the same metrics for the same answers |
 | graders read Claude stream-json via `traces.py` | read `events.jsonl` instead; `traces.py` becomes the Claude Code → events converter, which moves to the framework's harness profile |
-| oracle logic spread across `run.py check` | expose it as the `oracle` entry point |
+| oracle logic spread across `run.py check` | expose it as the `oracle` entry point; review tasks need none (§4.5) |
 | `bin/bazel` wrapper, shared caches, `MKLROOT` removal | expressed in the Dockerfile and `image.caches` / `image.env` |
 | clean-ancestor guard, no-background hook, per-run config dir | done by the framework for every harness |
 | fix and feature tasks have no review-level grading | add `rubric.json` per task ([mergeability](repo-evaluation-mergeability.md) §2), starting from the tasks mined from reviewed PRs; maintainers approve each rubric |
@@ -829,7 +918,8 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo
   on a rerun unless `--force`; a cell without one is reported as missing, not
   failed, and the next run of the batch fills it. A batch stays on the eval
   tree it started with: its revision, or for a working tree the digests of
-  its `eval_dir` and of the image build context.
+  its `eval_dir` and of the image build context. A batch with a task that
+  names a `dokimi:` grader (or a task directory that cannot be read) also stays on the built-in graders' sources.
 - The agent's container mounts the workspace, the per-run `home`, `config` and
   scratch directories (each `image.env` value under `{run_dir}/`), the caches,
   the harness executable (`--bin`, read-only at `/eval/harness`) and the
@@ -868,10 +958,10 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo
   arm and score-only in the other, or a measure (such as a metric) one arm
   lacks. A metric's `n_baseline`/`n_treatment` count the cells that report it.
   The output names the batches behind each selector and each batch's pin
-  (`tree`, `eval_dir`, `image` from `_eval/batch.json`) with its cells'
+  (`tree`, `eval_dir`, `image`, `graders` from `_eval/batch.json`) with its cells'
   harness versions, and warns when a bare `ARM` pools several batches, when a
   batch of a comparison has no pin, or when the two arms ran from different
-  `eval_dir` or `image` pins or harness versions. An image `repo run` tagged
+  `eval_dir`, `image` or `graders` pins or harness versions. An image `repo run` tagged
   from the tree (`dokimi/<id>:<sha>`) follows the tree, so two revisions are
   not warned about it; an `--image` of its own is compared. `repo report` warns
   (`model_mismatch` in its summary) when a cell's `model_resolved` is another

dokimi/dokimi/commands/repo_run.py

@@ -15,6 +15,7 @@
 from typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
+    from dokimi.repos.manifest import RepoManifest
     from dokimi.repos.workspace import Arm
 
 
@@ -29,6 +30,7 @@ def _context(args: argparse.Namespace, out: Path, pin: bool = False):
     from dokimi.repos import RepoSource, open_source
     from dokimi.repos.container import DockerRunner
     from dokimi.repos.grading import EvalContext
+    from dokimi.repos.graders import graders_digest
     from dokimi.repos.runner import slug
     from dokimi.repos.source import default_cache_dir
     from dokimi.repos.workspace import export_build_context, export_eval_dir, source_git_dir
@@ -60,7 +62,10 @@ def _context(args: argparse.Namespace, out: Path, pin: bool = False):
     elif not runner.image_exists(image):
         raise ValueError(f"--image {image} does not exist")
     if pin:
-        _pin_batch(out, {"tree": tree.label, "eval_dir": context_digest(staged), "image": image})
+        stamp = {"tree": tree.label, "eval_dir": context_digest(staged), "image": image}
+        if _uses_builtin_graders(staged, manifest):  # else no score depends on them: an upgrade keeps the batch
+            stamp["graders"] = graders_digest()
+        _pin_batch(out, stamp)
         shutil.rmtree(content, ignore_errors=True)
         staged.rename(content)
     if not args.image and (args.rebuild_image or not runner.image_exists(image)):
@@ -104,6 +109,18 @@ def context_digest(root: Path) -> str:
     return h.hexdigest()
 
 
+def _uses_builtin_graders(content: Path, manifest: RepoManifest) -> bool:
+    """Whether a task of the exported eval dir names a ``dokimi:`` grader; an unreadable task.json counts."""
+    from dokimi.repos.graders import builtin_name
+    from dokimi.repos.workspace import load_tasks
+
+    try:
+        tasks = load_tasks(content, manifest)
+    except (OSError, ValueError):
+        return True
+    return any(isinstance(t.raw, dict) and builtin_name(t.raw.get("grader")) is not None for t in tasks)
+
+
 def _pin_batch(batch_dir: Path, pin: dict[str, str]) -> None:
     path = batch_dir / "_eval" / "batch.json"
     if path.is_file():