Repository navigation
feat(repo): Codex and Copilot CLI harness profiles, native review mode #116
+1,900
−113
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| ### Added | ||
|
|
||
| - **Codex CLI and Copilot CLI harness profiles for `repo run` and `repo reach`** — `--harness codex`, `codex-review`, `copilot` and `copilot-review` run oneDAL-style repository tasks under `codex exec`, Codex's native review mode (`codex exec review --commit HEAD`), `copilot -p` and `copilot -p` with a `/review`-prefixed prompt, so guidance arms can be compared per harness, not only under Claude Code. Their JSON traces convert to `events.jsonl` (`trace: codex-json`, `copilot-json`). A profile can now run the harness's own prompt (`native_prompt`) and convert its answer before grading (`answer_format: codex-review` appends a findings block parsed from Codex's review text); `repo reach` refuses a `native_prompt` profile. `bin_companions` mounts the helper files an executable needs beside it (Codex's `codex-code-mode-host`). | ||
|
|
||
| ### Fixed | ||
|
|
||
| - **`repo reach` writes the profile's `config_files`** — only `repo run` wrote them, so a profile whose endpoint or hooks live in a config file (a Codex `config.toml` naming a provider) probed a different setup than it ran. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """Final-answer conversions a harness profile names in ``answer_format`` (contract §6). | ||
|
|
||
| A native review mode answers in its own format, not the ```` ```json ```` block | ||
| a review task asks for. The conversion keeps the harness's text and appends | ||
| that block, so a grader reading the last ```` ```json ```` block of the answer | ||
| reads the native findings; nothing is dropped from what the agent said. | ||
|
|
||
| Findings are ``{file, line, end_line, severity, priority, rule, message}``: | ||
| ``file`` relative to the workspace, ``line`` the first line of the range, | ||
| ``severity`` ``blocker`` (P0), ``major`` (P1) or ``minor`` (P2, P3) and ``None`` for a | ||
| priority this module does not know, ``rule`` the finding's title and ``message`` its body. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import logging | ||
| import re | ||
| from collections.abc import Callable | ||
| from typing import Any | ||
|
|
||
| from .transcript import relative_path | ||
|
|
||
| log = logging.getLogger(__name__) | ||
|
|
||
| ANSWER_FORMATS = ("codex-review",) | ||
| _SEVERITY = {"P0": "blocker", "P1": "major", "P2": "minor", "P3": "minor"} | ||
|
antonwolfy marked this conversation as resolved.
|
||
| # the title may hold " — ": the path part is what follows the last one and holds none | ||
| _CODEX_FINDING = re.compile(r"^- \[(P\d)\] (.+) — ([^\s—][^—]*?):(\d+)(?:-(\d+))?[ \t]*$", re.M) | ||
| _FINDING_MARK = re.compile(r"^\s*[-*]?\s*\[P\d\]", re.M | re.I) | ||
|
|
||
|
|
||
| def codex_review_findings(text: str, workdir: str | None = None) -> list[dict[str, Any]]: | ||
| """Findings of Codex's native review text: ``- [P1] title — /abs/path:12-14`` then an indented body.""" | ||
| found = list(_CODEX_FINDING.finditer(text)) | ||
| out = [] | ||
| for i, m in enumerate(found): | ||
| end = found[i + 1].start() if i + 1 < len(found) else len(text) | ||
| body_lines = [] | ||
| for line in text[m.end():end].splitlines()[1:]: # [0] is the rest of the header line | ||
| if not line.strip(): # a blank line may separate paragraphs of the body | ||
| continue | ||
| if not line[0].isspace(): # the body is indented; a flush line is prose after the findings | ||
| break | ||
| body_lines.append(line.strip()) | ||
| body = " ".join(body_lines) | ||
| first = int(m.group(4)) | ||
| if m.group(1) not in _SEVERITY: | ||
| log.warning("unknown priority %s in a review finding; its severity is left unset", m.group(1)) | ||
| out.append({"file": relative_path(m.group(3), workdir), "line": first, | ||
| "end_line": int(m.group(5)) if m.group(5) else first, | ||
| "severity": _SEVERITY.get(m.group(1)), "priority": m.group(1), | ||
| "rule": m.group(2).strip(), "message": body}) | ||
| return out | ||
|
|
||
|
|
||
| _CONVERTERS: dict[str, Callable[[str, str | None], list[dict[str, Any]]]] = { | ||
| "codex-review": codex_review_findings, | ||
| } | ||
|
|
||
|
|
||
| def convert_answer(fmt: str | None, answer: str, workdir: str | None = None, *, failed: bool = False) -> str: | ||
| """``answer`` with the findings block of ``fmt`` appended; unchanged when ``fmt`` is ``None``. | ||
|
|
||
| A ``failed`` run or an empty answer stays as it is: a missing review, not a clean one. So does | ||
| an answer with more finding markers than findings parsed: no block is appended, so a grader | ||
| does not read a review that reported a problem as a clean one. | ||
| """ | ||
| if fmt is None or failed or not answer.strip(): | ||
| return answer | ||
| if fmt not in _CONVERTERS: | ||
| raise ValueError(f"unknown answer format {fmt!r} (known: {', '.join(ANSWER_FORMATS)})") | ||
| findings = _CONVERTERS[fmt](answer, workdir) | ||
| marked = len(_FINDING_MARK.findall(answer)) | ||
| if marked > len(findings): # format drift must not read as a clean review | ||
| log.warning("%s: %d finding marker(s) in the answer but %d parsed; no findings block", fmt, marked, | ||
| len(findings)) | ||
| return answer | ||
| return f"{answer}\n\n```json\n{json.dumps(findings, indent=1)}\n```\n" | ||
|
napetrov marked this conversation as resolved.
Comment on lines
+78
to
+82
There was a problem hiding this comment. When the answer has more [Pn] markers than parsed findings, this only logs a warning and still appends the block. An answer with "- [P1] Leak - a.c:2" (hyphen, not em dash) ends with
Contributor
Author
There was a problem hiding this comment. Fixed in b55ccad: an answer with more finding markers than parsed findings gets no findings block and is returned as is (with the warning), as for a failed run. The caplog test now asserts the unchanged answer. |
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit (plausible, maintainability): Answer-format list is repeated in the schema and in code
ANSWER_FORMATS,_CONVERTERSand theanswer_formatenum inharness_profile.v1.jsoneach list the formats. Adding a format to one but not the others fails late, after the agent has run, as aValueErrorthatrun_cellrecords as a host-error failure. A small test that the three lists agree would catch this.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added in bc66cd0: test_answer_format_lists_agree asserts ANSWER_FORMATS, _CONVERTERS and the schema enum match, so drift fails in CI instead of after an agent run.