Repository evaluation: move the eval mechanics out of oneDAL dev/agent_evals into dokimi #41

Repository evaluation today has two implementations of the same mechanics. The framework side (dokimi repo validate | tasks check | run | report | scan) prepares workspaces, applies arms, runs agents and graders in containers, and gates results. The first repository to use it, uxlfoundation/oneDAL#3818 (dev/agent_evals, not merged), still carries its own copy: workspace prep, self-check, the repo_grade.v1 writer, process metrics, generic graders and a guidance claim checker, about 1.8k lines of Python, kept so the tasks run without the framework.

oneDAL reviewers asked why a mostly generic harness lives in their codebase. The answer is that it shouldn't: the tasks belong with the code they test, the mechanics belong here. This issue tracks moving the mechanics into dokimi so that a repository keeps only its evals.

Target split

oneDAL dev/agent_evals/              (after)
├── repo-eval.yaml                   manifest: guidance globs, image, defaults for built-in graders
├── Dockerfile, bin/bazel            the environment tasks build in
├── tasks/<id>/                      task.json, patches, hidden tests, review gold   (data only)
├── graders/build.py, make.py        the only graders that know oneDAL
└── README.md
.github/workflows/agent-evals.yml    ~30 lines calling dokimi's reusable workflow

dokimi                               (gains)
├── repo compare                     arm vs arm, paired by task
├── dokimi:* graders                 review seeds, review replay, hidden tests, mutation
├── task `host` requirements         unsupported instead of a false pass
├── process metrics                  from events.jsonl, plus manifest-declared counters
├── L0 parity                        the remaining oneDAL static_check rules
├── arms                             guidance from another revision, patch arms
└── reusable CI workflow             validate, scan, tasks check

Decided: oneDAL drops its standalone mode, and #3818 is trimmed to evals-only before it merges, so there is no later deletion PR.

Steps

Each step is one PR based on main. They are independent unless noted.

• D1. repo compare (feat(repo): repo compare, treatment arms against a baseline arm paired by task #42, merged): per-task paired deltas between a baseline arm and treatment arms over one or more batches, with a sign-flip permutation p (exact at reps=1, where it equals McNemar's exact test), a task bootstrap CI, Holm correction across comparisons, tasks up/down/tied, cost and turns, and any grade.json metric (--metric). Today repo report has no arm comparison; every analysis so far was an ad-hoc script.
• D2. Built-in graders, dokimi:review_seeds and dokimi:review_replay: task.json grader: "dokimi:<name>" is graded by framework code mounted read-only in the grading container (stdlib only); other grader names still go to the repository's grade entry point. Review tasks become pure data (manifest/gold JSON plus the answer). Ported from oneDAL graders/review.py and replay.py; dokimi's own review task is the second user.
• D3. Host requirements (feat(repo): task host requirements, unsupported instead of a false pass (#41 D3) #112, merged): task.json host: {min_cpus, cpu_vendors, cpu_flags}, checked against the container's view (affinity and cgroup quota). tasks check and run report the task as unsupported with the reason. Needed for a thread race that only reproduces with 3+ CPUs on Intel hosts.
• D4. Process metrics in the framework (feat(repo): process metrics in result.json, manifest process_metrics, grade optional for dokimi: graders #157): tool calls, guidance files read (from the manifest's globs), denied and failed commands, computed from events.jsonl into result.json. The manifest declares repository counters as regexes (process_metrics:, for example a bazel command count or a missing-compiler message), so oneDAL's metrics.py goes away.
• D5. dokimi:hidden_tests and dokimi:mutation (after D2): hidden files restored from a revision of {source_dir}, then test_cmd and regression_cmd templates, an optional format_cmd, root_cause, and diff size. Mutation: the agent's tests must pass and kill each mutant patch, with library sources restored from base by test_globs. Command defaults live in the manifest, not in every task.
• D6. L0 parity with oneDAL static_check.py (repo scan: L0 rules for parity with oneDAL static_check (D6) #115, merged): diff the claims both produce on oneDAL main and port the missing rules into the L0 adapters (Starlark macro keyword arguments, --config names from .bazelrc, and anything else the diff shows). The static entry point then becomes optional for oneDAL.
• D7. Arms beyond guidance on/off (feat(repo): guidance:rev:<rev> and patch:<file> arms (D7) #117, merged): guidance:rev:<sha> (the guidance of another revision on the task's base, the "does this PR's AGENTS.md help?" question, which needed generated task twins so far) and patch:<file> (tooling or layout changes).
• D8. Reusable CI workflow plus a pinned install (ci: reusable repo-eval workflow (validate, scan, tasks check) with dokimi pinned #113, merged): a workflow repositories call to run repo validate, repo scan and repo tasks check on the tasks a PR touches; a repository pins dokimi by tag or SHA, which the manifest records.
• oneDAL: trim #3818 (after D2–D6 and D8): delete common.py, config.py, contract.py, metrics.py, run.py, static_check.py and the generic graders; switch tasks to dokimi:* graders and host:; replace the workflow.

Parallel plan (2026-10-18)

Each wave-1 PR keeps to its own files and contract sections. Together they stay clear of the open PRs: #66 and #71 edit the contract head, §1 and the §8/§9 seam, and others edit architecture.md, README.md, AGENTS.md, docs/README.md, mkdocs.yml, ci.yml and the lines l0/adapters.py 51–58.

Acceptance for the oneDAL step

• Regrade every stored oneDAL run with the old and the new graders: verdicts and gate metrics agree on every run (the previous contract migration reached 308/308).
• dokimi repo tasks check is OK for all 27 tasks in the oneDAL image, with the host-dependent task unsupported rather than passing on hosts that cannot show its bug.
• repo scan on oneDAL main reports no fewer real failures than static_check.py.

Risks

• Grader semantics now ship with dokimi, so a regrade depends on its version: the manifest pins it, and grade.json records the grader version.
• Without standalone mode, a contributor needs docker and dokimi to self-check a task.
• Bazel details (--nocache_test_results, the output base shared by agent and grader) become manifest strings; only the regrade and tasks check catch a mistake there.
• Contract §5 ("the harness stays in oneDAL and keeps working standalone") is rewritten in D2.