diff --git a/dokimi/changelog.d/20261019_050000_nikolay.a.petrov_codex_copilot_profiles.md b/dokimi/changelog.d/20261019_050000_nikolay.a.petrov_codex_copilot_profiles.md new file mode 100644 index 00000000..a60e1f60 --- /dev/null +++ b/dokimi/changelog.d/20261019_050000_nikolay.a.petrov_codex_copilot_profiles.md @@ -0,0 +1,7 @@ +### Added + +- **Codex CLI and Copilot CLI harness profiles for `repo run` and `repo reach`** — `--harness codex`, `codex-review`, `copilot` and `copilot-review` run oneDAL-style repository tasks under `codex exec`, Codex's native review mode (`codex exec review --commit HEAD`), `copilot -p` and `copilot -p` with a `/review`-prefixed prompt, so guidance arms can be compared per harness, not only under Claude Code. Their JSON traces convert to `events.jsonl` (`trace: codex-json`, `copilot-json`). A profile can now run the harness's own prompt (`native_prompt`) and convert its answer before grading (`answer_format: codex-review` appends a findings block parsed from Codex's review text); `repo reach` refuses a `native_prompt` profile. `bin_companions` mounts the helper files an executable needs beside it (Codex's `codex-code-mode-host`). + +### Fixed + +- **`repo reach` writes the profile's `config_files`** — only `repo run` wrote them, so a profile whose endpoint or hooks live in a config file (a Codex `config.toml` naming a provider) probed a different setup than it ran. diff --git a/dokimi/docs/architecture.md b/dokimi/docs/architecture.md index 76de3508..5db03112 100644 --- a/dokimi/docs/architecture.md +++ b/dokimi/docs/architecture.md @@ -241,8 +241,10 @@ next release. it can see. A file comes back `loaded`, `truncated` (one token — the file was cut), `fetched` (seen only after the agent read it), `not_loaded`, or `error` (no usable answer). Harness behaviour is data, not code: - `dokimi/repos/profiles/claude-code.yaml` is a `harness_profile.v1` - file and any other harness is another YAML. + `dokimi/repos/profiles/` holds `harness_profile.v1` files for Claude + Code, the Codex CLI (`codex`, and its native review mode `codex-review`) and + the Copilot CLI (`copilot`, and `copilot-review`, a `/review`-prefixed prompt variant of it); any + other harness is another YAML. - `dokimi/repos/budget.py` — `repo budget`: what the tree puts into an agent's context when it starts at the root, in characters, with no LLM call. Root CLAUDE.md files and their `@` imports, the root AGENTS.md files and their `@` imports, and @@ -269,7 +271,9 @@ next release. - `dokimi/repos/bases.py` — checks that need every task base: each `base` must resolve, and the guidance globs are evaluated at each base. - `dokimi/repos/events.py` — converts a harness trace to the - `events.jsonl` run trace. + `events.jsonl` run trace; `dokimi/repos/cli_traces.py` holds the Codex and + Copilot JSON converters, and `dokimi/repos/answers.py` the `answer_format` + conversions that turn a native review answer into a findings block. - `dokimi/repos/l0/` — `repo scan` (L0): checks the links, paths, identifiers, commands and build labels the guidance names against the same tree (`repo_readiness.v1`), with no LLM call. diff --git a/dokimi/docs/preparing-a-repository.md b/dokimi/docs/preparing-a-repository.md index 18af5ca6..0b4956c7 100644 --- a/dokimi/docs/preparing-a-repository.md +++ b/dokimi/docs/preparing-a-repository.md @@ -204,10 +204,11 @@ effect. Targets skipped because a `CLAUDE.md` already exists are listed as `bridge_skipped` in `meta.json`; `repo report` does not show them. Loading changes between versions. Run `dokimi repo reach` for the harness and -version you use instead of relying on this table. Only the `claude-code` profile is -built in; for Codex or Copilot, pass a `harness_profile.v1` file with `--harness` +version you use instead of relying on this table. The `claude-code`, `codex`, +`codex-review`, `copilot` and `copilot-review` profiles are built in; for another +harness, pass a `harness_profile.v1` file with `--harness` ([contract](repo-evaluation-contract.md)). Tool-use counts need a trace the profile -can convert, which today means `claude-stream-json`; a `text` trace gives no +can convert, which today means `claude-stream-json`, `codex-json` or `copilot-json`; a `text` trace gives no events, turns or cost, so for such a harness `repo reach` can report a file as loaded but never as fetched by the agent. For Claude Code it requires the `claude` CLI, credentials and a mandatory `--model` flag; `repo --help` notes it costs one @@ -266,5 +267,5 @@ Treat each item above as a hypothesis for your repository. - C/C++: `compile_commands.json` covers every build variant; indexes are prebuilt if you offer navigation tools. - Recurring review rules, curated, are in the review section of `AGENTS.md`. - A `CLAUDE.md` containing `@AGENTS.md` sits next to each `AGENTS.md` (§7). -- `dokimi repo reach` (only `claude-code` is built in; others need a `harness_profile.v1` file) confirms each harness you use loads the files you rely on. +- `dokimi repo reach` (`claude-code`, `codex`, `copilot` and `copilot-review` are built in; `codex-review` runs its own prompt and is refused; others need a `harness_profile.v1` file) confirms each harness you use loads the files you rely on. - Each change was compared against the baseline on current models. diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 99d57849..0389ef12 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -31,7 +31,7 @@ SPDX-License-Identifier: Apache-2.0 > Slice S3 adds harness profiles (§6) and the reach probe (§7): > > ```bash -> python cli.py repo reach --manifest --model [--harness claude-code] +> python cli.py repo reach --manifest --model [--harness claude-code|codex|copilot|copilot-review|] > [--bin ] [--scenarios root,cwd,read] [--only ] [--dry-run] > ``` > @@ -608,7 +608,7 @@ env_allow: [PATH, HOME, AWS_REGION, AWS_BEARER_TOKEN_BEDROCK, ANTHROPIC_API_KEY, env_set: {CLAUDE_CODE_DISABLE_BACKGROUND_TASKS: "1", DISABLE_AUTOUPDATER: "1"} egress: ["bedrock-runtime.*.amazonaws.com", "api.anthropic.com"] # the only hosts the agent container reaches (§2) endpoint_env: ANTHROPIC_BASE_URL # used by payload capture, if set -trace: claude-stream-json # converter to events.jsonl; or: text +trace: claude-stream-json # converter to events.jsonl; or: codex-json, copilot-json, text guidance: # DECLARED; verified by the reach probe, never trusted names: [CLAUDE.md, CLAUDE.local.md, AGENTS.md] resolution: ancestors-of-cwd + lazy-on-subtree-access @@ -631,8 +631,56 @@ bridge: {for: AGENTS.md, write: CLAUDE.md, content: "@AGENTS.md"} scripts) as the agent's container sees them. The built-in `claude-code` profile uses them for a hook that refuses backgrounded shell commands. -Profiles for other harnesses (Codex, Gemini CLI, Copilot CLI, Cursor agent, -Aider, OpenCode, in-house agents) follow the same shape. Their `guidance` +- `trace` names the converter of the harness's stdout to `events.jsonl`: + `claude-stream-json`, `codex-json` (`codex exec --json`), `copilot-json` + (`copilot -p --output-format json`) or `text` (stdout is the answer, no + events). Codex has no read tool, so its `read` events come from shell + commands that only print files (`cat`, `head`, `tail`, `sed -n`, `nl`). + Neither Codex nor Copilot reports a cost: `cost_usd` is `null` and a batch + budget counts each cell's cap. +- `bin_companions` names files the executable starts from its own directory. + `repo run` mounts the executable alone at `/eval/harness/`, so each + companion is mounted read-only beside it, from beside `--bin`; a missing one + stops the run before any agent starts. Codex 0.160 needs + `codex-code-mode-host`: every tool call goes through it. +- `repo reach` writes `config_files` into each scenario's config directory as + `repo run` does, with `{config_dir}` and `{support_dir}` as host paths. +- `native_prompt: true` marks a harness that runs its own built-in prompt, such + as a native review mode. `invoke` then passes no `{prompt}` (and there is no + `prompt_suffix`), the agent never sees the task prompt, and `repo reach` + refuses the profile. Use such a profile only on tasks whose grader reads what + it answers, which `answer_format` provides. +- `answer_format` converts the final answer before grading. `codex-review` + keeps Codex's review text and appends a ```` ```json ```` block of findings + `{file, line, end_line, severity, priority, rule, message}` parsed from its + `- [P1] title — path:12-14` lines: `file` relative to the workspace, + `severity` `blocker` (P0), `major` (P1) or `minor` (P2, P3), `rule` the title + and `message` the body. + +Built-in profiles besides `claude-code`: + +| profile | runs | trace | notes | +|---|---|---|---| +| `codex` | `codex exec --json … -- {prompt}` | `codex-json` | `CODEX_HOME` is the per-run config dir; no turn or spend flag, the timeout bounds a run; `none`/`read` get Codex's read-only sandbox, which needs unprivileged user namespaces | +| `codex-review` | `codex exec review --commit HEAD --json` | `codex-json` | `native_prompt`, `answer_format: codex-review`; review tasks only (the change under review is HEAD) | +| `copilot` | `copilot --prompt={prompt} --output-format json --allow-all-tools …` | `copilot-json` | `COPILOT_HOME` per run; a GitHub token as `COPILOT_GITHUB_TOKEN` (`GH_TOKEN` and `GITHUB_TOKEN` are not forwarded: Copilot leaves them readable by the agent's commands), or BYOK through the `COPILOT_PROVIDER_*` variables; `none` names a tool that does not exist, because an empty `--available-tools` leaves every tool on | +| `copilot-review` | as `copilot`, with the prompt `/review {prompt}` | `copilot-json` | a prompt variant of `copilot`, not Copilot's review agent: under `-p`, `/review` reaches the model as plain text, with the same system prompt and tools; the task prompt still asks for its answer format | + +npm installs `codex` and `copilot` as Node launchers. A run mounts one file as +the harness, so `--bin` names the native binary the launcher starts +(`@openai/codex-/vendor//bin/codex`, +`@github/copilot-/copilot`). To route a CLI to another endpoint, +copy its profile: for Copilot, set the `COPILOT_PROVIDER_*` variables (and +`COPILOT_OFFLINE=true` to skip GitHub sign-in); for Codex, add a `config.toml` +under `config_files` that defines a `model_providers` entry with the endpoint's +`base_url`, `wire_api = "responses"` and `env_key = "CODEX_API_KEY"` (the variable that +holds its key; without it Codex sends no `Authorization` header) and selects it with `model_provider`. +Add the endpoint's host to `egress` in both cases. The hosted pull-request +reviewers (Copilot code review, `@codex review`, CodeRabbit) are not CLIs and +have no profile. + +Profiles for other harnesses (Gemini CLI, Cursor agent, Aider, OpenCode, +in-house agents) follow the same shape. Their `guidance` blocks start as the vendor's documented behaviour and are corrected by the reach probe. Loading changes between versions of one harness, so reach results are keyed by version. @@ -702,7 +750,12 @@ per scenario). - `class` is one of: - `loaded`: both tokens reported; - `truncated`: one token reported; - - `fetched`: reported only after the agent read the file with a tool; + - `fetched`: reported only after the agent read the file with a tool. For the + Codex and Copilot profiles a shell read counts only as a bare `cat`, `head`, + `tail`, `nl` or `sed -n`; for Codex, a command that is one such read of one + file and exits non-zero is not a read (the exit code of `cat a missing` does + not say which file failed, so both stay reads). A read through a pipe, `grep` + or `rg` is not seen, so such a file can be classed `loaded`; - `not_loaded`; - `error`: the scenario produced no usable answer: no answer, a non-zero exit, or an error result. @@ -962,7 +1015,8 @@ python cli.py repo compare repo-runs/pilot --baseline guidance:none --metric roo names a `dokimi:` grader (or a task directory that cannot be read) also stays on the built-in graders' sources. - The agent's container mounts the workspace, the per-run `home`, `config` and scratch directories (each `image.env` value under `{run_dir}/`), the caches, - the harness executable (`--bin`, read-only at `/eval/harness`) and the + the harness executable (`--bin`, read-only at `/eval/harness`, with its + `bin_companions` beside it, §6) and the framework's helper scripts (read-only at `/eval/support`). Before a batch, the harness must report its version inside the image, and a probe container with the same mounts must see neither `/eval/content` nor `/eval/source`, no diff --git a/dokimi/dokimi/repos/answers.py b/dokimi/dokimi/repos/answers.py new file mode 100644 index 00000000..60c68af4 --- /dev/null +++ b/dokimi/dokimi/repos/answers.py @@ -0,0 +1,82 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Final-answer conversions a harness profile names in ``answer_format`` (contract §6). + +A native review mode answers in its own format, not the ```` ```json ```` block +a review task asks for. The conversion keeps the harness's text and appends +that block, so a grader reading the last ```` ```json ```` block of the answer +reads the native findings; nothing is dropped from what the agent said. + +Findings are ``{file, line, end_line, severity, priority, rule, message}``: +``file`` relative to the workspace, ``line`` the first line of the range, +``severity`` ``blocker`` (P0), ``major`` (P1) or ``minor`` (P2, P3) and ``None`` for a +priority this module does not know, ``rule`` the finding's title and ``message`` its body. +""" + +from __future__ import annotations + +import json +import logging +import re +from collections.abc import Callable +from typing import Any + +from .transcript import relative_path + +log = logging.getLogger(__name__) + +ANSWER_FORMATS = ("codex-review",) +_SEVERITY = {"P0": "blocker", "P1": "major", "P2": "minor", "P3": "minor"} +# the title may hold " — ": the path part is what follows the last one and holds none +_CODEX_FINDING = re.compile(r"^- \[(P\d)\] (.+) — ([^\s—][^—]*?):(\d+)(?:-(\d+))?[ \t]*$", re.M) +_FINDING_MARK = re.compile(r"^\s*[-*]?\s*\[P\d\]", re.M | re.I) + + +def codex_review_findings(text: str, workdir: str | None = None) -> list[dict[str, Any]]: + """Findings of Codex's native review text: ``- [P1] title — /abs/path:12-14`` then an indented body.""" + found = list(_CODEX_FINDING.finditer(text)) + out = [] + for i, m in enumerate(found): + end = found[i + 1].start() if i + 1 < len(found) else len(text) + body_lines = [] + for line in text[m.end():end].splitlines()[1:]: # [0] is the rest of the header line + if not line.strip(): # a blank line may separate paragraphs of the body + continue + if not line[0].isspace(): # the body is indented; a flush line is prose after the findings + break + body_lines.append(line.strip()) + body = " ".join(body_lines) + first = int(m.group(4)) + if m.group(1) not in _SEVERITY: + log.warning("unknown priority %s in a review finding; its severity is left unset", m.group(1)) + out.append({"file": relative_path(m.group(3), workdir), "line": first, + "end_line": int(m.group(5)) if m.group(5) else first, + "severity": _SEVERITY.get(m.group(1)), "priority": m.group(1), + "rule": m.group(2).strip(), "message": body}) + return out + + +_CONVERTERS: dict[str, Callable[[str, str | None], list[dict[str, Any]]]] = { + "codex-review": codex_review_findings, +} + + +def convert_answer(fmt: str | None, answer: str, workdir: str | None = None, *, failed: bool = False) -> str: + """``answer`` with the findings block of ``fmt`` appended; unchanged when ``fmt`` is ``None``. + + A ``failed`` run or an empty answer stays as it is: a missing review, not a clean one. So does + an answer with more finding markers than findings parsed: no block is appended, so a grader + does not read a review that reported a problem as a clean one. + """ + if fmt is None or failed or not answer.strip(): + return answer + if fmt not in _CONVERTERS: + raise ValueError(f"unknown answer format {fmt!r} (known: {', '.join(ANSWER_FORMATS)})") + findings = _CONVERTERS[fmt](answer, workdir) + marked = len(_FINDING_MARK.findall(answer)) + if marked > len(findings): # format drift must not read as a clean review + log.warning("%s: %d finding marker(s) in the answer but %d parsed; no findings block", fmt, marked, + len(findings)) + return answer + return f"{answer}\n\n```json\n{json.dumps(findings, indent=1)}\n```\n" diff --git a/dokimi/dokimi/repos/cli_traces.py b/dokimi/dokimi/repos/cli_traces.py new file mode 100644 index 00000000..52ada036 --- /dev/null +++ b/dokimi/dokimi/repos/cli_traces.py @@ -0,0 +1,561 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Codex CLI and Copilot CLI JSON traces -> :class:`~dokimi.repos.events.Transcript` (contract §4.1). + +``codex-json`` is the stdout of ``codex exec --json`` (codex-cli 0.160): one +``item.*`` event per agent message, shell command, file change, MCP call or web +search, and ``turn.completed`` with token usage. Codex has no read tool, so a +``read`` event is derived from a shell command that only prints files +(``cat``, ``head``, ``tail``, ``sed -n``, ``nl``) and names them. + +``copilot-json`` is the stdout of ``copilot -p --output-format json`` (Copilot +CLI 1.0.91): ``tool.execution_start`` / ``tool.execution_complete`` pairs, +``assistant.message`` and a final ``result``. It reports no tokens and no cost. + +Neither CLI reports a cost, so ``cost_usd`` stays ``None`` and a batch budget +counts the cell's cap. +""" + +from __future__ import annotations + +import json +import posixpath +import re +import shlex +from datetime import datetime +from typing import Any + +from .transcript import Transcript, _parse_time, clip_output, relative_path + +_BASH_WRAP = re.compile(r"^(?:/\S*/)?(?:ba|z)?sh -lc (.+)$", re.S) +_PRINTERS = {"cat", "head", "tail", "nl", "sed"} +_COPILOT_SHELL = {"bash", "powershell"} +_COPILOT_READ = {"view"} +_COPILOT_EDIT = {"edit", "create", "str_replace_editor", "write"} +_PATCH_FILE = re.compile(r"^\*\*\* (?:Add|Update|Delete) File: (.+)$", re.M) +_CONTINUATION = re.compile(r"\\\n") +_HEREDOC = re.compile(r"<<-?[ \t]*(['\"]?)(\w+)\1([^\n]*)\n.*?^[ \t]*\2[ \t]*$", re.S | re.M) +# a redirect that writes nowhere a read could be confused with: 2>&1, 2>/dev/null, &>/dev/null +_HARMLESS_REDIRECT = re.compile(r"(?:\d*>&\d+|&>>?\s*/dev/null|\d*>>?\s*/dev/null)(?=\s|$|[;&|])") +_KEYWORDS = {"if", "then", "else", "elif", "do", "while", "until", "!"} +_DIR_STACK = {"pushd", "popd"} +_PIPE = re.compile(r"(? str: + """The script of ``/usr/bin/bash -lc '