feat(repo): skill/skill-catalog arms, cache tokens, MCP config when repo ships one

dokimi/changelog.d/20261020_090000_napetrov_repo-skill-arms.md

@@ -0,0 +1,27 @@
+### Added
+
+- **`repo run` skill arms** — `--arm guidance:none+skill:<path>` installs one
+  skill under `.claude/skills/<name>/`, and `skill-catalog:<dir>` installs
+  every skill in a directory. `result.json` records `skills`
+  (`skills_used`, `skill_invoked`, `skill_selected_correct`).
+  `repo report --json` gives, per row, the mean of the token fields `result.json` already has
+  (`input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`).
+  `repo tasks check --arm` rejects a skill
+  path that does not exist, and a skill whose `SKILL.md` `name` differs from its
+  directory name. The arm is named by the directory (`skill:lint`), so
+  `repo compare` can select it. Resuming a batch with other skill content
+  under the same arm name is refused.
+- **`args_if_present` in harness profiles** — arguments appended only when a
+  file exists in the workspace and the tool mode is `all`.
+
+### Fixed
+
+- **claude-code starts a repo's MCP servers** — the stock profile passed
+  `--strict-mcp-config` without `--mcp-config`, so a repo that ships
+  `.mcp.json` ran with no MCP tools. It now passes `--mcp-config .mcp.json`
+  when that file is present and `--tools all` is used (`--tools none`/`read` do not
+  limit MCP tools, so they start no MCP server).
+  This shifts baselines: a repo that ships `.mcp.json` now starts its MCP
+  servers under `--tools all`, so its results differ from earlier runs of the
+  same repo and arm. Compare only batches run after this change.
+- **A malformed `.mcp.json` is caught up front** — `repo validate` warns (and still succeeds) about one that is not a JSON object (or whose `mcpServers` is not) at a task base, and `repo run --tools all` exits before any cell, instead of every cell failing as a task failure when the CLI rejects the file; a `patch:` arm that writes one is a cell error. `check_arms` also refuses, before any cell, a base that tracks `.claude` or `.claude/skills` as a symlink; a `guidance:none` arm installs a skill the base ships under the guidance globs; a `SKILL.md` `name` that YAML reads as an impossible date is not a crash.

dokimi/docs/benchmarking-and-comparison.md

@@ -274,6 +274,33 @@ python cli.py arms run \
 The `baseline` arm is the control. Every other arm is compared against it. Keep
 the answer model and judge fixed when comparing arms.
 
+### Skill arms in repository mode
+
+`repo run` arms take the same `+` composition. Besides `guidance:none`,
+`guidance:raw` and `guidance:bridge`, an arm can install Claude Code skills into
+the task workspace at `.claude/skills/<dir name>/` before the agent starts:
+
+- `skill:<path>`: one skill directory (holding `SKILL.md`);
+- `skill-catalog:<dir>`: every subdirectory of `<dir>` with a `SKILL.md`, so the
+  agent has to pick the right one.
+
+A relative path is relative to the manifest's `eval_dir`. An arm without a
+guidance part keeps the repository's guidance (`guidance:raw`).
+
+```bash
+dokimi repo run <repo>@<rev> --model <m1> --model <m2> --batch skills \
+  --arm guidance:raw,guidance:raw+skill:skills/lint,guidance:none+skill-catalog:skills
+dokimi repo tasks check <repo>@<rev> --arm skill-catalog:skills   # paths exist, catalog has expected_skill
+```
+
+When a cell installs skills, its `result.json` gets a `skills` object:
+`skills_used` (the skills the agent called with the `Skill` tool, or whose
+`SKILL.md` it read), `skill_invoked`, and `skill_selected_correct` (the task's
+optional `expected_skill` was the first skill it used). `repo report` counts
+both per row. Every cell also records `cache_read_tokens` and
+`cache_write_tokens` with the input and output tokens, and
+`repo report --json` gives their means.
+
 ### Reading the significance tables
 
 Arms are compared **paired per question** against the one shared `baseline`

dokimi/docs/repo-evaluation-contract.md

@@ -260,6 +260,7 @@ belongs to the repository's grader and is passed through unread, except `grader`
 | `timeout_s`, `max_turns`, `budget_usd` | no | framework | per-task caps, overriding run defaults; each a finite number > 0 |
 | `host` | no | framework | `{"min_cpus": <int>, "cpu_vendors": [<str>], "cpu_flags": [<str>]}`, every key optional: what the hidden test needs to show the bug (a thread race that needs 3+ CPUs, an ISA path). Vendor and flag names are `/proc/cpuinfo`'s (`GenuineIntel`, `avx512f`); both match in any case, and a vendor other than `GenuineIntel`, `AuthenticAMD`, `HygonGenuine`, `CentaurHauls`, `Shanghai` is refused (`repo validate` too). A probe that read neither vendor nor flags from `/proc/cpuinfo` is a probe failure, not a miss They are checked against a probe container started like the grading one: CPUs are its affinity mask capped by its cgroup CPU quota (`docker --cpus`). On a host that misses them the task is unsupported (§4.3), and the probed facts go to `meta.json` `host` of a cell that runs, or to its `unsupported.json` `host` when it does not |
 | `egress` | no | framework | extra hosts (DNS names, optional `:port`; `*` stands for one whole label and the last two labels are literal, so `*.pythonhosted.org` passes and `*` or `*.org` does not) this task's agent and entry points reach through the proxy (§2), for example `["pypi.org", "files.pythonhosted.org"]` |
+| `expected_skill` | no | framework | the name (directory name) of the skill the agent should use; a `skill-catalog:` arm must install it, or the arm is refused before anything runs. `result.json` `skills` records whether the agent used it and whether it used it first |
 | anything else | — | repository | for example `hidden_files`, `targets`, `regression_targets`, `manifest` |
 
 `setup_patch` (a single string) is accepted as a synonym for a one-element
@@ -322,6 +323,18 @@ Both write files, as step 1 does: no object of the source's git dir is copied
 into the workspace, so its object store gains only the blobs of the files the
 arm wrote, and step 8 drops the guidance the arm replaced.
 
+A skill arm (`skill:<path>`, `skill-catalog:<dir>`, joined to a guidance part
+with `+`) installs skill directories into `.claude/skills/<dir name>/` after the
+guidance part, so `guidance:none+skill:x` keeps `x`. The arm is named by the
+directory names (`guidance:none+skill:x`, no `/`), so `repo compare` can select it.
+The `name` in a skill's `SKILL.md` has to equal its directory name. A skill is refused before
+any cell runs if it holds a link that copying cannot take (out of the skill,
+dangling, looping, to a directory) or a file that is not regular. Preparation
+also refuses to write through a `.claude` or `.claude/skills` the repository
+ships as a link. A batch records the content of each skill arm it ran
+(`_eval/skills.json`), so resuming it with another path of the same
+directory name, or with edited skill content, is refused.
+
 `meta.json` records the base (`base_rev`, `base_sha`, `base_date`), the arm,
 the SHA of every applied patch, and `start_sha`: the workspace `HEAD` after
 step 7, the commit the agent starts from (§4). `base_sha` names the upstream
@@ -330,7 +343,9 @@ grader diffs the agent's changes from `start_sha`. With the arm come
 `arm_files`, the files it deleted, wrote or changed; for `guidance:rev:` an
 `arm_rev` object (`rev` as given, its `sha`, and `removed`: the base's guidance
 files with no counterpart at `<rev>`); for `patch:` an `arm_patch` object (the
-file's absolute `path` and full `sha256`).
+file's absolute `path` and full `sha256`). `args_if_present` lists the profile's
+`args_if_present` files the cell's command line carries (§6): `[]` when the
+workspace has none of them, and always outside the `all` tool mode.
 
 Step 3 removes history only. A fix that landed upstream before a model's
 training cutoff may still be known to the model, so reports split results by
@@ -382,6 +397,17 @@ Code's `modelUsage`), `main` when the harness reported the parent's only, and
 `null` when it reported none. `subagent_calls` counts the parent's and
 subagents' `Agent`/`Task` calls; `null` when the harness wrote no result record.
 
+A cell of an arm that installs skills also has a `skills` object in
+`result.json`: `measured`, `skills_installed` (the installed skill names),
+`skills_used` (the installed skills the agent called with the `Skill` tool or
+whose `SKILL.md` it read, in order of first use), `expected_skill` (the task's,
+or `null`), `skill_invoked` (the expected skill was used; any installed skill
+when the task names none) and `skill_selected_correct` (the expected skill was
+the first one used; `null` when the task names none). When the trace could not
+be read (the harness never started), `measured` is `false` and `skill_invoked`
+and `skill_selected_correct` are `null`: the usage is unknown, not "did not use
+the skill". `repo report` leaves such cells out of its skill counts.
+
 ### 4.2 `grade.json` (written by `grade` to `{run_dir}/grade.json`)
 
 ```json
@@ -614,6 +640,8 @@ guidance:                            # DECLARED; verified by the reach probe, ne
   resolution: ancestors-of-cwd + lazy-on-subtree-access
   imports: "@path"
 bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"}
+args_if_present:                     # argv added after the "{tools}" element when the workspace file exists and the tool mode is `all`
+  .mcp.json: ["--mcp-config", ".mcp.json"]
 ```
 
 - `invoke` and `version_cmd` are argv lists. Each element becomes exactly one
@@ -630,6 +658,17 @@ bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"}
   directory, with `{config_dir}` and `{support_dir}` (the framework's helper
   scripts) as the agent's container sees them. The built-in `claude-code`
   profile uses them for a hook that refuses backgrounded shell commands.
+- `args_if_present` checks that the file exists, not that the harness accepts
+  it. The built-in `claude-code` profile passes `.mcp.json` as `--mcp-config`
+  in the `all` tool mode, and the CLI exits 1 before its first turn on a file
+  that is not valid MCP configuration (`{` for one). So a `.mcp.json` that is
+  not a JSON object, or whose `mcpServers` is not an object, is caught up
+  front: `repo validate` warns about the one at each task base (at
+  the eval tree when the source has no git history) and still succeeds, `repo run --tools all`
+  exits 1 before the image build and any cell, and a cell whose arm wrote
+  such a file (a `patch:` arm) is a cell error, not a graded run. Fix or remove
+  the file in the task base; `--tools read` or `none` do not pass it, so
+  `repo run` does not check it there.
 
 Profiles for other harnesses (Codex, Gemini CLI, Copilot CLI, Cursor agent,
 Aider, OpenCode, in-house agents) follow the same shape. Their `guidance`

dokimi/dokimi/commands/repo_run.py

@@ -146,20 +146,26 @@ def _errors():
     from dokimi.repos import ManifestError, SourceError
     from dokimi.repos.container import ContainerError
     from dokimi.repos.harness_profile import ProfileError
+    from dokimi.repos.arms import ArmError
     from dokimi.repos.workspace import PrepError
 
-    return (SourceError, ManifestError, ContainerError, PrepError, ProfileError, ValueError, OSError)
+    return (ArmError, SourceError, ManifestError, ContainerError, PrepError, ProfileError, ValueError, OSError)
 
 
 def cmd_repo_tasks_check(args: argparse.Namespace) -> None:
     """Self-check (no LLM): null fails the gate, oracle passes it, the agent cannot see the tasks."""
+    from dokimi.repos.arms import check_arms
     from dokimi.repos.selfcheck import run_check
-    from dokimi.repos.workspace import load_tasks
+    from dokimi.repos.workspace import load_tasks, parse_arm
 
     out = Path(args.out).resolve()
     try:
         ctx, _ = _context(args, out)
         tasks = load_tasks(ctx.content_dir, ctx.manifest, args.task)
+        arms = [a for spec in args.arm or () for a in spec.split(",") if a]
+        check_arms([parse_arm(a) for a in arms], ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs)  # skill paths exist, catalogs hold each expected_skill
+        if arms:
+            print(f"{len(arms)} arm(s) ok: {', '.join(arms)}", file=sys.stderr)
         print(f"{len(tasks)} task(s), image {ctx.image}{' (warm: with network)' if args.warm else ''}",
               file=sys.stderr)
         results = run_check(ctx, tasks, out / "check", warm=args.warm, jobs=args.jobs)
@@ -186,6 +192,8 @@ def cmd_repo_run(args: argparse.Namespace) -> None:
 
 def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock: contextlib.ExitStack) -> None:
     from dokimi.repos import RepoSource, open_source
+    from dokimi.repos.arms import check_arms
+    from dokimi.repos.bases import check_mcp_configs
     from dokimi.repos.container import docker_network
     from dokimi.repos.harness_profile import load_profile
     from dokimi.repos.runner import (
@@ -205,6 +213,9 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
         lock.enter_context(batch_lock(batch_dir))  # before _context, which rewrites _eval/content
         ctx, _ = _context(args, batch_dir, pin=True)
         tasks = load_tasks(ctx.content_dir, ctx.manifest, args.task)
+        check_arms(arms, ctx.content_dir, tasks, ctx.git_dir, ctx.manifest.guidance_globs)  # skill paths exist, catalogs hold each expected_skill
+        check_mcp_configs(ctx.git_dir, tasks, profile, args.tools)  # a malformed .mcp.json fails here, not in every cell
+        skill_pin = _skill_pin(batch_dir, arms, ctx.content_dir)  # a resume may not swap an arm's skills
         validate_tasks(tasks)  # refuse malformed host requirements before containers start
         options = [AgentOptions(profile=profile, model=m, harness_bin=harness_bin, agent_network=args.agent_network,
                                 tools=args.tools, timeout_s=args.timeout, max_turns=args.max_turns,
@@ -222,6 +233,9 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
             raise ValueError("the agent's container is not isolated: " + "; ".join(problems))
         print(f"{profile.name} {version} in {ctx.image}; {len(tasks)} task(s) x {len(arms)} arm(s) x "
               f"{len(options)} model(s) x {args.reps} rep(s) -> {batch_dir}", file=sys.stderr)
+        if skill_pin is not None:
+            (batch_dir / "_eval" / SKILL_PIN).write_text(json.dumps(skill_pin, sort_keys=True) + "\n",
+                                                         encoding="utf-8")
         summary = run_batch(ctx, tasks, arms, options, args.reps, batch_dir, jobs=args.jobs, force=args.force,
                             max_batch_usd=args.max_batch_usd)
     except _errors() as exc:
@@ -236,6 +250,28 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
         sys.exit(1)
 
 
+SKILL_PIN = "skills.json"
+
+
+def _skill_pin(batch_dir: Path, arms: list[str], content_dir: Path) -> dict[str, list[list[str]]] | None:
+    """The batch's ``_eval/skills.json`` with this run's skill arms added (``None``: nothing to add): arm name ->
+    [skill name, tree sha256] of each skill. A skill arm is named by its directories, so a resume with another
+    path or edited content would add cells of other skills to the arm's row; that is refused."""
+    from dokimi.repos.arms import resolve_skills, tree_sha256
+    from dokimi.repos.workspace import parse_arm
+
+    now = {str(a): [[s.name, tree_sha256(s.dir)] for s in resolve_skills(a, content_dir)]
+           for a in map(parse_arm, arms) if a.skills}
+    path = batch_dir / "_eval" / SKILL_PIN
+    old = json.loads(path.read_text(encoding="utf-8")) if path.is_file() else {}
+    for arm, skills in now.items():
+        if old.get(arm, skills) != skills:
+            was, new = (", ".join(f"{name} {sha[:12]}" for name, sha in pin) for pin in (old[arm], skills))
+            raise ValueError(f"batch {batch_dir.name} ran arm {arm} with skills {was}, not {new}; "
+                             f"use another --batch")
+    return {**old, **now} if now.keys() - old.keys() else None
+
+
 def _pin_arms(arms: list[Arm], git_dir: Path, source: str) -> list[str]:
     """Each arm under the name its cells get, resolved once for the whole batch (a revision to its
     full SHA, so a branch that moves later names another arm); two spellings of one arm run once."""
@@ -247,8 +283,13 @@ def _pin_arms(arms: list[Arm], git_dir: Path, source: str) -> list[str]:
     worktree = None if remote else Path(parsed.location)  # HEAD of a linked worktree is not the main one's
     pinned: list[str] = []  # Arm objects: run_batch hands each to prepare_workspace
     first_path: dict[str, str | None] = {}
+    first_skills: dict[str, tuple[tuple[str, str], ...]] = {}
     for arm in arms:
         named = pin_arm(arm, git_dir, fetch=remote, worktree_dir=worktree)
+        if first_skills.setdefault(named, named.skills) != named.skills:  # skill arms are named by directory
+            raise ValueError(f"arms {named.spec!r} and another spelling of {named!r} name different skill paths "
+                             f"({named.skills} vs {first_skills[named]}) but would share one cell; "
+                             f"rename a directory or spell the path alike")
         if named != named.spec:
             detail = f"{named.patch_path}, sha256 {named.patch_sha256}" if named.patch is not None else named.spec
             print(f"arm {named} = {detail}", file=sys.stderr)
@@ -344,6 +385,9 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
     _common(check_p, source_help, cache_help, positive_int)
     check_p.add_argument("--warm", action="store_true",
                          help="Run null and oracle with network to fill the caches before offline runs")
+    check_p.add_argument("--arm", action="append", default=None,
+                         help="Also check this `repo run` arm: its skill paths exist and a skill-catalog "
+                         "holds each task's expected_skill; repeatable or comma-separated")
     check_p.add_argument("--allow-unsupported", action="store_true", dest="allow_unsupported",
                          help="Exit 0 when the only tasks that are not ok are unsupported because this host "
                          "misses their task.json `host` requirements (still printed with the reason). Broken "
@@ -360,7 +404,10 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None:
                        "the guidance files of revision REV of the source; the arm is named by REV's full SHA, "
                        "resolved once per batch) | patch:FILE (a unified diff on this host, applied with git "
                        "apply on top of guidance:raw; named patch:<file name>@<sha256 prefix>, path and "
-                       "sha256 in meta.json); repeatable or comma-separated")
+                       "sha256 in meta.json) | skill:<dir> | skill-catalog:<dir> (skills installed in the "
+                       "workspace; a relative dir is in eval_dir; named by the directory, not the path). A "
+                       "guidance part and skills join with + (guidance:none+skill:skills/x); "
+                       "repeatable or comma-separated")
     run_p.add_argument("--model", action="append", required=True,
                        help="Model string passed to the harness as it is; repeatable")
     run_p.add_argument("--reps", type=positive_int, default=1)