[napetrov]

Part of #41 (step D2).

Summary

A task names a built-in grader and is data only; run_entry picks framework code over the repository's entry point:

 run_entry(ctx, kind, task, run_dir)          # grade | oracle; tasks check and repo run unchanged
+  if task.grader starts with "dokimi:"
+    argv  = python3 -I -B /eval/graders/main.py <kind> <name> /eval/content/<task dir> /eval/run
+    mount repos/graders/ -> /eval/graders (read-only, grading container only)
+    exit 127 -> ContainerError "dokimi:<name> runs on the image's python3, and the image has none on PATH"
+  else
     argv = manifest entry_points[kind]       # as before

dokimi/dokimi/repos/graders/
├── __init__.py   # host side: dokimi: prefix, names, mount source
├── main.py       # container entry: grade -> grade.json (repo_grade.v1), oracle -> answer.txt
└── review.py     # review_seeds, review_replay; stdlib only, Python 3.8+

• Matching is ported from oneDAL's graders/review.py and replay.py: seeds and decoys with a keyword match and a fallback to the nearest line, one-to-one nearest-first replay matching within window, recall_by_class / recall_by_kind, precision_proxy, and the clean-diff control. split_result is the same, and so is the rule that sets a gate metric the grader never produced to null.
• An answer with no findings JSON now reports recall 0 (replay: recall_all too) next to pass: false, parse_error: true. Before, tasks check marked every score review task broken on the null workspace ("no numeric metrics.recall"). The oneDAL graders are making the same change, so the two stay identical.
• -I: a PYTHONPATH set in image.env cannot put the agent's files ahead of the standard library. The grader code is never mounted into the agent's container.
• grade.json records details.grader_version.
• Contract: new §4.5. §5 now says the generic parts of oneDAL's harness move into dokimi, and its grader rows are updated.

Evidence

Equivalence with oneDAL. For each of oneDAL's 9 review tasks (3 review, 6 review_replay) I ran oneDAL's grader in-process and dokimi's main.py as the container runs it, on the same answers:

answers	n
oracle (the two oracle texts are also identical)	9
empty and garbage (no fence, broken JSON, a dict, non-objects, odd line/severity/file)	117
oracle answers perturbed (line shifts ±1..7, text removed so only line matching applies, */.//absolute/backslash//eval/run/workspace/ and nested repo/ paths, duplicates, halves)	225
every other review task's oracle answer	72
stored agent answers from earlier oneDAL batches (answer.txt, or the last result of a stream-json trace); 640 unique	991
total	1414

Compared per answer: pass, metrics, details (without grader_version) and the first error line.

oneDAL dev/agent-evals	mismatches
c4e2667 (before the parse_error change)	124 + review fixes: all on parse_error answers (added recalls) or /eval/run/workspace/ paths
6dc140de (has recall: 0 on parse_error; repo tasks check 27/27 ok)	108: 84 replay parse_error answers that differ only by recall_addressed, and 24 /eval/run/workspace/ paths
82ec2e13 (both review fixes below; oneDAL repo tasks check 27/27 ok)	0

The two review fixes (Copilot) change results only where oneDAL's graders were wrong under dokimi. A path in the agent's workspace (/eval/run/workspace/src/x.cpp) is cut there before oneDAL's /repo/ rule; none of the 991 stored answers uses one. A replay answer with no findings also reports recall_addressed 0, so a task scored on it is not a grader failure on the null workspace. A mutated port (line fallback widened from 1 to 2) gives 7 mismatches, so the comparison does catch a change in semantics. Process metrics (oneDAL's metrics.py) belong to D4 and are not compared.

comparison script (not in the repo)

"""Equivalence check: oneDAL's review graders (in-process, from a checkout of dev/agent-evals) vs dokimi's
built-in graders (run as the grading container runs them: python3 -I -B main.py grade|oracle ...).

usage: python3 compare.py <oneDAL checkout> <dokimi graders dir> <runs root> [--out report.json]

For every task under dev/agent_evals/tasks whose grader is `review` or `review_replay`:
  - oracle answers: both oracles' texts must be identical, then both graders grade it
  - empty answer, garbage answers, perturbed oracle answers, and every other review task's oracle answer
  - stored agent answers: every run dir under <runs root> named after the task, with answer.txt, or with
    trace.jsonl only (Claude stream-json; the answer is the last `result` event's `result`)
Compared per answer: pass, metrics, details (dokimi's details.grader_version dropped), first error line.
oneDAL's process metrics (metrics.py, from events.jsonl) are not part of either grader and are not compared.
"""
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path

ONEDAL, GRADERS_DIR, RUNS = (Path(a).resolve() for a in sys.argv[1:4])
OUT = Path(sys.argv[sys.argv.index("--out") + 1]) if "--out" in sys.argv else None
EVAL = ONEDAL / "dev/agent_evals"
sys.path.insert(0, str(EVAL))
import contract  # noqa: E402  (oneDAL)
import graders  # noqa: E402  (oneDAL)

NAME = {"review": "review_seeds", "review_replay": "review_replay"}


def onedal_grade(t, text):
    g = graders.GRADERS[t["grader"]](None, None, t, {"answer": text})
    mets, det = contract.split_result(g)
    for k in t.get("gate", []):
        mets.setdefault(k, None)
    return {"pass": g.get("pass"), "metrics": mets, "details": det, "error": None}


def onedal(t, text):
    try:
        return onedal_grade(t, text)
    except Exception as e:  # what contract.grade() reports
        return {"pass": None, "metrics": {}, "details": {}, "error": f"{type(e).__name__}: {e}"}


def dokimi(t, text, kind="grade"):
    with tempfile.TemporaryDirectory() as d:
        rd = Path(d)
        (rd / "task.json").write_text(json.dumps(t))
        (rd / "answer.txt").write_text(text, encoding="utf-8")
        r = subprocess.run([sys.executable, "-I", "-B", str(GRADERS_DIR / "main.py"), kind, NAME[t["grader"]],
                            str(EVAL / "tasks" / t["id"]), str(rd)], capture_output=True, text=True)
        if r.returncode:
            raise RuntimeError(f"main.py {kind} exited {r.returncode}: {r.stderr[-500:]}")
        if kind == "oracle":
            return (rd / "answer.txt").read_text(encoding="utf-8")
        doc = json.loads((rd / "grade.json").read_text())
    doc["details"].pop("grader_version")
    return {"pass": doc["pass"], "metrics": doc["metrics"], "details": doc["details"],
            "error": doc["errors"][0] if doc["errors"] else None}


def review_tasks():
    out = []
    for p in sorted((EVAL / "tasks").glob("*/task.json")):
        t = json.loads(p.read_text())
        t.setdefault("id", p.parent.name)
        if t.get("grader") in NAME:
            out.append(t)
    return out


def fence(obj):
    return "```json\n" + json.dumps(obj, indent=1) + "\n```"


GARBAGE = [
    "", "no findings", "```json\n```", "```json\n{not json\n```", "```json\n{\"file\": \"a\"}\n```",
    "```json\n[1, \"x\", null]\n```", "```json\n[]\n```", "```json\nnull\n```",
    "```json\n[{\"file\": \"x\"}]\n``` then ```json\n[{\"file\": \"y\", \"line\": 1}]\n```",
    "```json\n[{\"file\": \"y\"}]\n``` trailing ```json\n{broken\n```",
    fence([{"file": None, "line": None, "severity": None}]),
    fence([{"line": "abc", "severity": "BLOCKER"}, {"line": [1], "severity": "Major"}, {"line": 3.7}]),
    "```JSON\n[{\"file\": \"a\"}]\n```",
]


def perturbed(t, oracle_text):
    """Variants of the oracle answer that exercise the matching rules."""
    base = json.loads(oracle_text.split("```json", 1)[1].rsplit("```", 1)[0])
    out = []
    for shift in (-7, -6, -5, -2, -1, 1, 2, 5, 6, 7):
        out.append([{**f, "line": (f.get("line") or 0) + shift} for f in base])
    out.append([{**f, "rule": "", "message": "", "evidence": ""} for f in base])  # line fallback only
    out.append([{**f, "rule": "", "message": "", "line": str(f.get("line"))} for f in base])
    out.append([{**f, "file": "*"} for f in base])
    out.append([{**f, "file": "./" + str(f.get("file"))} for f in base])
    out.append([{**f, "file": "/tmp/run/repo/" + str(f.get("file"))} for f in base])
    out.append([{**f, "file": str(f.get("file")).replace("/", "\\")} for f in base])
    out.append([{**f, "file": "/eval/run/workspace/" + str(f.get("file"))} for f in base])
    out.append([{**f, "file": "/eval/run/workspace/./" + str(f.get("file"))} for f in base])
    out.append([{**f, "file": "/old/run/repo/eval/run/workspace/" + str(f.get("file"))} for f in base])
    out.append([{**f, "file": "/eval/run/workspace/sub/repo/" + str(f.get("file"))} for f in base])
    out.append([{**f, "severity": "blocker"} for f in base])
    out.append(base + base)  # duplicates: one-to-one matching
    out.append(base[::-1])
    out.append(base[: len(base) // 2])
    out.append([{**f, "line": None} for f in base])
    return [fence(x) for x in out]


def trace_answer(path):
    answer = None
    for line in path.read_text(errors="replace").splitlines():
        try:
            ev = json.loads(line)
        except ValueError:
            continue
        if isinstance(ev, dict) and ev.get("type") == "result":
            answer = ev.get("result") or ""
    return answer


def stored(ids):
    """{task id: [(run dir, answer)]} for run dirs named after a review task. RUNS is a directory to walk,
    or a file listing answer.txt / trace.jsonl paths (relative to its "root:" first line or absolute)."""
    out = {i: [] for i in ids}
    if RUNS.is_file():
        lines = RUNS.read_text().splitlines()
        root = Path(lines[0].split(":", 1)[1]) if lines and lines[0].startswith("root:") else Path("/")
        dirs = sorted({str((root / l).parent) for l in lines[1:] if l.strip()})
        walk = ((d, [], os.listdir(d)) for d in dirs)
    else:
        walk = os.walk(RUNS)
    for dirpath, dirnames, filenames in walk:
        dirnames[:] = [d for d in dirnames if d not in ("repo", "workspace", "ob", "home", ".git", "claude-config",
                                                         "config", "grade-home", "repo-cache", "main-tree")]
        tid = os.path.basename(dirpath).split("__")[0]
        if tid not in out:
            continue
        if "answer.txt" in filenames:
            out[tid].append((dirpath, Path(dirpath, "answer.txt").read_text(errors="replace")))
        elif "trace.jsonl" in filenames:
            a = trace_answer(Path(dirpath, "trace.jsonl"))
            if a is not None:
                out[tid].append((dirpath, a))
    return out


def main():
    tasks = review_tasks()
    print("tasks:", ", ".join(f"{t['id']} ({t['grader']})" for t in tasks))
    oracles, rows, mismatches = {}, [], []
    for t in tasks:
        a = graders.ORACLES[t["grader"]](None, None, t)
        b = dokimi(t, "", "oracle")
        if a != b:
            mismatches.append({"task": t["id"], "source": "oracle text"})
        oracles[t["id"]] = a
    runs = stored([t["id"] for t in tasks])
    for t in tasks:
        cases = [("oracle", oracles[t["id"]])] + [("garbage", g) for g in GARBAGE]
        cases += [("perturbed", p) for p in perturbed(t, oracles[t["id"]])]
        cases += [("other-oracle", oracles[o["id"]]) for o in tasks if o["id"] != t["id"]]
        cases += [("stored", ans) for _, ans in runs[t["id"]]]
        for src, text in cases:
            a, b = onedal(t, text), dokimi(t, text)
            rows.append({"task": t["id"], "source": src})
            if a != b:
                mismatches.append({"task": t["id"], "source": src, "onedal": a, "dokimi": b, "answer": text[-300:]})
    by = {}
    for r in rows:
        by.setdefault(r["source"], 0)
        by[r["source"]] += 1
    stored_unique = len({(t, a) for t in runs for _, a in runs[t]})
    print("answers compared:", len(rows), by, "| stored run dirs:", sum(map(len, runs.values())),
          "unique stored (task, answer):", stored_unique)
    print("per task stored:", {k: len(v) for k, v in runs.items()})
    print("oracle texts identical:", not any(m["source"] == "oracle text" for m in mismatches))
    print("mismatches:", len(mismatches))
    for m in mismatches[:10]:
        print(json.dumps(m)[:1500])
    if OUT:
        OUT.write_text(json.dumps({"rows": len(rows), "by_source": by, "mismatches": mismatches}, indent=1))
    return 1 if mismatches else 0


if __name__ == "__main__":
    sys.exit(main())

Tests (tests/test_repo_builtin_graders.py, 20, all with small fixtures of their own):

tasks check on toy tasks dokimi:review_seeds / review_replay / clean control + a repo-graded task
  before (main grading.py): review tasks go to the toy repo's grade.py -> clean control BROKEN, no /eval/graders mount
  after: 5/5 OK (one replay task scored on recall_addressed); built-in specs: python3 -I -B, /eval/graders read-only, no {source_dir}; repo grader untouched
image without python3 (runner returns 127)
  before: FAIL   after: tasks check BROKEN and repo run grader_failure, reason names python3
unknown dokimi:review_seed
  before: FAIL (sent to the repo grader)   after: BROKEN "unknown built-in grader"

With grading.py reverted, 4 tests fail; the rest import the new package and fail without it. The review-fix tests (recall_addressed self-check task, nested repo/ paths) fail on the commit before each fix.

• outcome() is split into _run_reasons and _grade_reasons (CC 22 to 10, for CodeFactor). The reasons and their order are unchanged, and the existing grading tests cover it.
• Local: pytest -n auto --outcomes 5197+ passed, ruff, mypy, schema_check, config_check and bandit clean. Radon CC is at most 13.
• Docker end to end, repo tasks check with the real DockerRunner: python:3.12-slim 3/3 OK, oracle recall 1.0. ubuntu:20.04, which has no python3: 3/3 broken, with the python3 reason. The graders also run on Python 3.8.

Merge Danger

Door: two-way. This only adds a grader namespace. No existing grader name starts with dokimi:, and reverting removes it.

Blast Radius: additive

• dokimi: in task.json grader is now reserved. A repository grader with that prefix would be refused instead of reaching its grade entry point.
• The manifest schema still requires entry_points.grade, even when every task uses a built-in grader. Its schema and manifest.py belong to other steps and are unchanged here.
• Follow-up, not in this PR: dokimi's own .agent-eval review task (review-pr-conventions) could switch to dokimi:review_seeds. The §3 key table still says grader is "interpreted by the grade entry point" (§3 belongs to D3).

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.92405% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.47%. Comparing base (2a37bf1) to head (9155b75).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/graders/main.py	71.21%	13 Missing and 6 partials ⚠️
dokimi/dokimi/repos/grading.py	79.31%	7 Missing and 5 partials ⚠️
dokimi/dokimi/repos/manifest.py	78.94%	2 Missing and 2 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #120      +/-   ##
==========================================
+ Coverage   91.44%   91.47%   +0.02%     
==========================================
  Files         216      219       +3     
  Lines       26877    27166     +289     
  Branches     4411     4459      +48     
==========================================
+ Hits        24578    24849     +271     
- Misses       1634     1646      +12     
- Partials      665      671       +6     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo_run.py	92.44% <100.00%> (+0.34%)	⬆️
dokimi/dokimi/repos/compare.py	99.63% <100.00%> (ø)
dokimi/dokimi/repos/container.py	83.87% <100.00%> (+0.13%)	⬆️
dokimi/dokimi/repos/graders/__init__.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/graders/review.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/manifest.py	94.18% <78.94%> (-0.90%)	⬇️
dokimi/dokimi/repos/grading.py	89.94% <79.31%> (+2.28%)	⬆️
dokimi/dokimi/repos/graders/main.py	71.21% <71.21%> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Both graders incorrectly score valid findings using absolute container workspace paths as misses.

1 open issue

• Workspace path normalization misses /eval/run/workspace paths · New

What changed in this PR

Moves reusable review grading into dokimi so repository review tasks can be data-only, advancing #41.

Changes:

• Adds built-in seed and replay graders with versioned results.
• Dispatches grading and oracle runs through read-only framework code.
• Adds regression tests and documents the grader contract.

File	Description
dokimi/​tests/​test_repo_builtin_graders.py	Tests matching, dispatch, and failures.
dokimi/​dokimi/​repos/​grading.py	Dispatches built-in graders and refactors outcome reasons.
dokimi/​dokimi/​repos/​graders/​review.py	Implements review scoring and oracles.
dokimi/​dokimi/​repos/​graders/​main.py	Runs graders and writes result artifacts.
dokimi/​dokimi/​repos/​graders/​__init__.py	Defines built-in names and mount source.
dokimi/​dokimi/​repos/​container.py	Adds the grader mount path.
dokimi/​docs/​repo-evaluation-contract.md	Documents built-ins and oneDAL migration.
dokimi/​changelog.d/​20261018_221020_napetrov_builtin-review-graders.md	Records the new grading functionality.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Verified scoring and path-normalization defects can reject valid tasks or mis-score correct findings.

1 open issue

• Return addressed recall for replay tasks · New

1 previously missed issue

In code that hasn't changed since last review

Preserve nested repo directories in legacy path handling

dokimi/​dokimi/​repos/​graders/​review.py:79

The legacy branch also strips internal /repo/ components from repository-relative paths. A seed or gold path src/repo/calc.py becomes calc.py, while the equivalent finding /eval/run/workspace/src/repo/calc.py becomes src/repo/calc.py. Both graders consequently return recall 0 for the workspace-absolute finding, versus 1 for the relative finding. Preserve relative paths and distinguish the legacy workspace boundary from internal directories. Add regression coverage for both graders, including legacy absolute paths with nested repo directories.

1 resolved since last review

• Workspace path normalization misses /eval/run/workspace paths

_{Review effort: Balanced}

[napetrov]

On the review's "previously missed" point about nested repo directories: this one stays as it is on purpose. The /repo/ cut is oneDAL's legacy rule, kept so a regrade of stored oneDAL runs gives identical results. A relative src/repo/calc.py is cut to calc.py there too. A path into the dokimi workspace (/eval/run/workspace/...) is cut at the workspace before that rule applies, so it keeps src/repo/.... Contract §4.5 now states both cuts (b456969).

[napetrov]

Correction to my comment above: the nested-repo point was real. With the workspace cut as an elif, /eval/run/workspace/src/repo/calc.py became src/repo/calc.py, while the seed's own src/repo/calc.py became calc.py, so the finding missed. Fixed in 6a81986: both cuts now apply in turn to every path, seed and finding alike. A path without the workspace prefix still normalizes exactly as in oneDAL's graders, which keeps the regrade identical. test_a_nested_repo_directory_matches_from_any_path_form covers relative, workspace and legacy /repo/ forms for both graders.

[Copilot]

🔵 Needs a closer look

A verified path-normalization inconsistency causes equivalent findings to receive different recall scores in both graders.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Normalize workspace-absolute paths consistently for nested repo directories

dokimi/​dokimi/​repos/​graders/​review.py:79

For files under a real repo/ subdirectory, equivalent relative and workspace-absolute paths normalize differently: tools/repo/calc.py becomes calc.py, while /eval/run/workspace/tools/repo/calc.py becomes tools/repo/calc.py. Because same_file() also normalizes the reference path, both graders give the relative finding recall 1 but the absolute finding recall 0. Apply the legacy normalization after stripping the workspace prefix too, and cover this nested-directory case in both graders' regression tests. Update the normalization documentation to match.

1 resolved since last review

• Return addressed recall for replay tasks

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No unresolved blocking findings remain, and regression coverage addresses both previously reported defects.

No open issues 🎉

_{Review effort: Balanced}

[antonwolfy]

Review at 20ee90f. The dispatch is sound: the outcome() split keeps every branch and its order, the grader code is mounted read-only into the grading container only, and -I -B closes the PYTHONPATH route. Three things are worth fixing before merge: one crash on a hostile answer, one gap in the contract, one gap in the tests.

What I ran, on a Linux review host at 20ee90f:

• make ci-local: 19 passed, 0 failed, 4 skipped; 5199 tests passed. The run used Python 3.13, so I did not check the 80% changed-line gate against the CI matrix.
• Reproductions for the findings, with the LLM transport mocked.
• A mutation run on test_repo_builtin_graders.py: 15 mutants, 5 of which pass the full file (see the inline comment on the 127 test).
• main.py output for seeds and replay, with and without parse_error, validated against repo_grade.v1.json: all valid.

Findings outside the diff:

1. The contract still lists the old grading-container mounts. §4.2 (repo-evaluation-contract.md around lines 96-113: the mount list and the "Fixed paths" bullet) names only /eval/run, /eval/content and /eval/source. /eval/graders appears only in §4.5. §3 (lines 216 and 224) still says non-core task.json keys are "passed through unread" and that grader is "interpreted by the grade entry point", and the "Read by" column says repository. Both are false for dokimi: names. §4.5 also does not say that repo-eval.yaml must still declare entry_points.grade for an all-built-in repository: repo_manifest.v1.json requires it, so a review-only repository fails manifest validation before any built-in grader runs. Suggestion: add the mount to §4.2, reword the §3 intro and the grader row, and add one sentence to §4.5 on the required grade entry.

Checked, no issues

• outcome() split: compared branch by branch with the base; the not reasons guard on the missing-score check is preserved through failed=bool(reasons).
• Isolation: the agent container mounts only the workspace and its subdirectories; task.json is written by dokimi outside the container; task_dir comes from dokimi's own argv. Dropping read_only on the graders mount or the content mount fails the tests.
• Namespace: dokimi: with an unknown, empty, or trailing-space name is refused and never reaches the repository's grade.
• Packaging and rules: graders/ has an __init__.py, so the wheel ships the .py files; SPDX headers and the changelog fragment are in place; no schema change, so no version bump; the code uses no syntax newer than Python 3.8.

[napetrov]

Two follow-ups, both in b67a3ae.

1. Correction to my earlier reply to @vlad-perevezentsev (feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay #120 (comment)): that reply said dropping standalone mode was not agreed. It was: the 2026-10-14 decision is that oneDAL drops standalone mode fully. §5 of docs/repo-evaluation-contract.md and its table rows (prep(), ONEDAL_EVAL_SRC, contract.py) are restored to that wording, and oneDAL #3818 has to match it.
2. Built-in grader change not in the batch pin (feat(repos): built-in review graders dokimi:review_seeds and dokimi:review_replay #120 (comment)): option (a). repo run now hashes the built-in grader sources into _eval/batch.json (graders) and refuses to resume a batch whose graders changed, like an edit to .agent-eval/grade.py. repo compare adds graders to its stamp keys and warns on a mismatch. Tests: test_a_batch_refuses_resume_after_a_built_in_grader_changes, test_arms_from_batches_with_different_built_in_graders_warn (both fail on the old code). Stored batches have no graders pin and need a fresh --batch dir (changelog fragment added).

[xaleryb]

• Re-run at b67a3ae on Linux: full suite, ruff, mypy, bandit; repo run and compare with a working-tree grader edit; mutants for every earlier test-gap thread (all killed).
• Fixed: a grader edit now refuses resume of a batch with a dokimi: task, and compare warns.
• Open: a repository with no dokimi: task. Batch run, then any grader source edit, then resume: refused with "run from graders ..., use another --batch", and compare warns "different graders". Expected: resume as before, no warning, because no score depends on the graders.
• Open: oracle.json whose message quotes suggestion: the review_seeds oracle scores recall 0 and tasks check reports the task broken. Expected: 1.0.
• Pushed:
  • d8a2ce6: the seeds oracle escapes the way the replay oracle does. A file without gives the same oracle text as before.
  • 911187d: the graders pin applies only when a task names a dokimi: grader. An unreadable task.json still pins. Changelog fragment and contract §6 updated.
• Static checks fails on main too (errors.py header, fix: add the missing license header to two repos modules #144). Needs a re-run after that lands.

[antonwolfy]

Review at 911187d. The commits since d2d54cb are right in what they set out to do: the seeds oracle escapes a quoted fence, the batch pin now applies only when a task names a dokimi: grader, and both test gaps from my earlier threads are closed. Two test gaps remain in the new code, and one doc nit.

What I ran, on a Linux review host at 911187d:

• make ci-local: 26 passed, 0 failed, 4 skipped. The run used Python 3.13, so the other matrix legs were not run.
• A mutation run on the new code against test_repo_builtin_graders.py, test_repo_compare.py and the built-in cases of test_repo_run.py: 42 mutants. The same_file bool(path), the window default and scope, the seeds-oracle escape and the _uses_builtin_graders mutants are all killed. The ones below pass.

Checked, no issues

• Oracle escape: oracle_seeds and oracle_replay decode back to the same text for messages with 3, 4 and more backticks; dropping or narrowing the escape fails a test.
• Batch pin: any to all, always True, always False and an unreadable task.json returning False are all killed; graders is kept in the stamp and shown in repo compare.
• Rules: changelog fragments, SPDX headers, no schema change, no generated artifacts, ruff and mypy clean on the changed files.

[antonwolfy]

Nothing pins _too_deep or MAX_DEPTH. The only test nests a fence 100000 deep. On Python 3.13 json.loads already raises RecursionError there and last_json catches it, so the new code is never needed for that test to pass.

Mutation check: each of these still gives 44 passed on test_repo_builtin_graders.py: MAX_DEPTH = 10000, depth >= MAX_DEPTH, _too_deep returning False always, dropping the string/escape handling, and replacing depth -= 1 with pass. On 3.14 (which parses what 3.13 refuses) a bug here would go unnoticed.

The cap is also a visible change: a valid fence nested 101 to about 900 deep parses with json.loads on 3.13 but is now skipped, so an earlier fence is graded instead.

Suggestion: call _too_deep directly. "["*100 + "]"*100 is False, 101 levels is True, and last_json on a 150-deep fence after a valid one returns the valid one. Add a 200-bracket string with an escaped quote (["[[[\"[[["], expect False) and 300 sequential [] pairs (expect False). Say in §4.5 that a fence nested deeper than 100 is skipped like an unparseable one.

[napetrov]

Pinned in 966fa92: test_fence_nesting_cap_is_exactly_max_depth checks that depth == cap parses, cap+1 is unreadable, brackets inside strings are not counted, and the grader falls back to an earlier fence. d8a2ce6 and 911187d cover fence escaping and the conditional batch pin, not the cap.

[antonwolfy]

graders_digest is pinned only for an edit to review.py. The test writes one file, review.py, and changes its content.

Mutation check: glob("review.py") instead of glob("*.py"), so an edit to main.py (the entry point that runs in the container) no longer changes the digest, still passes. So does dropping path.name from the hash, where swapping the contents of two files keeps the digest. A batch would then resume over a changed grader, which is what the pin is for.

Suggestion: in the same test, write main.py too and assert that editing it changes the digest, and that swapping the bytes of main.py and review.py changes it. Non-blocking: the digest also covers the host-side __init__.py, so a comment edit there refuses a resume; hash only the files the container runs if that matters.

[napetrov]

Pinned in 966fa92: test_graders_digest_covers_main_and_pairs_names_with_contents fails if an edit to main.py, a content swap between files, or a rename leaves the digest unchanged. The old test only edited review.py.

[antonwolfy]

I pushed a commit per nit, up to 79484d2, which fix the nit from my review:

• 79484d2: dokimi/docs/repo-evaluation-contract.md:912: the pin condition now reads "names a dokimi: grader (or a task directory that cannot be read)".

make ci-local at 79484d2: 26 passed, 0 failed, 4 skipped.

Not approving yet. Still open: the two risks from my review.

[vchamarthi]

@exai-bot /review

[exai-bot]

@vchamarthi I did not start this: the limit of 20 requests a day per person is reached. Ask again after 2026-10-22 03:44 UTC.

[napetrov]

@antonwolfy thanks for the nit commit 79484d2 (pin condition wording in the contract); kept.

Open items from your review at 911187d:

• _too_deep / MAX_DEPTH not pinned: fixed in 966fa923. The test checks depth == cap parses, cap+1 is unreadable, brackets inside strings are not counted, and the grader falls back to an earlier fence. Test: test_fence_nesting_cap_is_exactly_max_depth.
• graders_digest pinned only for an edit to review.py: fixed in 966fa923. The test fails if an edit to main.py, a content swap between files, or a rename leaves the digest unchanged. Test: test_graders_digest_covers_main_and_pairs_names_with_contents.

Ready for another look.

[vchamarthi]

@exai-bot /re-review

[exai-bot]

Note

AI-generated review of 5e1fba82cbb1, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 12 of 80 turns, 1.0 of 40 minutes, 452k in / 6k out tokens.

This PR adds the dokimi:review_seeds and dokimi:review_replay built-in graders. run_entry mounts the grader code read-only and runs it with python3 -I -B instead of the repository's entry point. It also adds manifest checks, a graders digest in the batch pin, and a graders stamp key in compare. I read all the source changes and the first 20,000 of 34,276 characters of the builtin-graders test diff, and found no defects I could confirm. The outcome refactor keeps the original reason ordering and conditions. The grader code uses only Python 3.8-compatible syntax. --init is set in container.py, so the tini-based 127 detection is consistent. The author's 1414-answer equivalence run against oneDAL is strong evidence for the port. I did not read the docs.

No issues found.

For a human reviewer

• Pinning the batch to a digest of every .py under graders/ (including the host-side init.py) means any edit there refuses resume of existing batches. Decide whether that granularity is wanted.
• Contract §4.5 and the rewritten §5 are the public API for repositories that adopt dokimi: graders. Review the wording as product behaviour.

Not verified

I could not run the tests or the container. That includes the exit-127 / tini stderr signature ('exec python3 failed') on a real image without python3, and Python 3.8 execution of the graders. I did not check the equivalence claims against oneDAL. I did not read the last 14,276 characters of the builtin-graders test diff.

[xaleryb]

• Re-run at 5e1fba8, both alone and merged with current main, on Linux: full suite, ruff, mypy, bandit and mkdocs --strict all clean. Re-ran the original inputs of every earlier point with a real edit to the grader source: with no dokimi: task, the batch resumes and compare gives no warning; with one, resume is refused and compare warns; a seeds oracle that quotes a suggestion fence passes tasks check with score 1.0; the replay suggestion and deep-fence cases pass. Re-ran the digest mutants from the open thread: all killed.
• Open thread on _too_deep: dropping the escape branch, or replacing depth -= 1 with pass, still passes the file. Without the decrement, an answer with more than 100 findings becomes parse_error. §4.5 does not say that a valid fence deeper than 100 levels is skipped.
• Pushed:
  • 307403e: test that kills both mutants.
  • c70d512: §4.5 says that a fence nested past the cap counts as unparseable.
• pip-audit is red because of virtualenv CVE-2026-113722 in the shared lock, which is not part of this PR; ci(locks): bump virtualenv for CVE-2026-113722 #152 fixes it.

[xaleryb]

Approved at 9155b75 (main merged in for #152's virtualenv bump). Every earlier point re-run passes, no must-fix or mid item is open; full suite, ruff and mypy clean on Linux; CI green.