diff --git a/README.md b/README.md index 05b341d2..9a195cb3 100644 --- a/README.md +++ b/README.md @@ -60,6 +60,8 @@ baseline with paired statistics. Arms are given as specs (`dokimi arms run | `skill-catalog:` | a whole skill catalog, to measure whether the right skill gets selected | | `profile:` | an agent persona / system prompt | | `guidance:none\|raw\|bridge` | a repository with its `AGENTS.md`/`CLAUDE.md` removed, kept, or bridged (repository mode, below) | +| `guidance:rev:` | the same repository with the guidance files of another revision, such as a PR's `AGENTS.md` (repository mode) | +| `patch:` | the repository with a unified diff applied on top of its own guidance: a new script, a Makefile target (repository mode) | The same arms drive question answering (LLM-as-judge with judge-validity checks), executable Docker coding tasks through Codex, Claude Code, Harbor or diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 00aec556..fc7d3abe 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -74,7 +74,7 @@ SPDX-License-Identifier: Apache-2.0 > and agent runs with their report: > > ```bash -> python cli.py repo tasks check --manifest [--task ] [--warm] +> python cli.py repo tasks check --manifest [--task ] [--warm] [--allow-unsupported] > python cli.py repo run --manifest --arm guidance:raw --model --reps N --batch NAME > --agent-network host > python cli.py repo report / @@ -311,12 +311,15 @@ with `git apply` on top of `guidance:raw` (after `setup_patches`). It is for tooling or layout treatments, such as a new script or a Makefile target, that no task carries. The file is read once per batch, so every cell applies the same bytes, and the arm is named `patch:@`, where `` is the file name without its directory, with every -character outside `[A-Za-z0-9.-]` replaced by `_` and cut to 40 characters, so -the name holds no `/` for `BATCH/ARM` selectors. The exact name is printed on -the `repo run` stderr line `arm = , sha256 ...`. A patch that does not apply to a task's base, or that writes into -`eval_dir`, fails that cell's preparation. A cell has one arm: `guidance:rev:` -and `patch:` do not combine. +sha256>`, where `` is the file name without its directory: each run of +characters outside `[A-Za-z0-9.-]` becomes one `_`, a leading or trailing `_` is +dropped, the result is cut to 40 characters and a `_` the cut leaves at the end +is dropped too (`patch` if nothing is left), so `my fix (v2).diff` gives +`my_fix_v2_.diff`. The name holds no `/` for `BATCH/ARM` selectors; the exact +name is printed on the `repo run` stderr line `arm = , sha256 ...`. +A patch that does not apply to a task's base, or that writes into `eval_dir`, +fails that cell's preparation. A cell has one arm: `guidance:rev:` and `patch:` +do not combine. Both write files, as step 1 does: no object of the source's git dir is copied into the workspace, so its object store gains only the blobs of the files the @@ -463,7 +466,9 @@ A task whose `host` requirements (ยง3) the probe container does not meet is `unsupported` with the reason, like a task whose `egress` needs the missing proxy, and `tasks check` exits 1 for it. With `--allow-unsupported` it is still printed with its reason but does not make the exit code 1; broken tasks, -tasks that need egress and tasks whose host probe failed still do. `repo run` does not run its cells: +tasks that need egress and tasks whose host probe failed still do. The probed +facts are recorded in `check.json` `host`, and for `repo run` in the batch's +`_eval/host.json`. `repo run` does not run its cells: each holds an `unsupported.json` with the reason, and `repo report` lists them apart, neither failed nor missing. If the probe itself fails, a task with `host` requirements is not run either, and `repo run` exits 1 when one of its @@ -961,6 +966,7 @@ line, else its last `error:` line, else its last line. ```bash # self-check: no LLM; exits 1 if any task is broken or unsupported +# (with --allow-unsupported, a task this host cannot run does not count) python cli.py repo tasks check ~/oneDAL@ --manifest dev/agent_evals/repo-eval.yaml \ --task fix_finiteness_mut --build-network host --cache bazel-repo-cache=/cache/bazel-repo -j 3