Skip to content

Commit 7e107fd

Browse files
tzhouamclaude
andcommitted
Stop publishing [resolved] lines; file residuals as their own comments
A [resolved] findings line was re-filed as a P2 comment whenever its prose matched a residual keyword. The bare "but " marker, and plain "residual:", also matched lines that decline their point, so vllm-omni#6968 published "... (author follow-up) — not re-raised." as a finding. The coverage editor received the same lines as raw material and could do the same. A [resolved] line is now only the review's record. The reviewer prompt asks for a residual that needs action to be filed as its own comment with disposition publish, a file:line and the [resolved] line as evidence, which is the structured selection #154 introduced. The prose promotion is removed and the coverage editor no longer sees [resolved] lines. Tests use the verbatim #6968 line (never published) and the measured pr4977 residual filed as a publish comment (published). Fixes #155 Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 48e8b4a commit 7e107fd

5 files changed

Lines changed: 76 additions & 102 deletions

File tree

‎doc/architecture/SPEC/engine/steps/review.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# engine/steps/review/ —— 规范
22

3-
<!-- verified-against: 2026-09-26 -->
3+
<!-- verified-against: 2026-09-29 -->
44

55
`patch gate + PR review + quality(9 个源文件) · step 库(评审) · refactor-status: ok`
66

@@ -49,6 +49,11 @@ agent、评审结果的有界精炼、评测调优过的 prompt 数据和确定
4949
「Checked, no defect found」注记:保留证据,去掉诉求,不带优先级。
5050
在此之前取舍只活在模型的 `summary` 散文里,而发布读的是列表,于是
5151
summary 说要丢弃的请求照样行内发出(#141)。
52+
- **`[resolved]` 是记录,不是诉求**:findings 里的 `[resolved]` 行从不变成评论。
53+
残留问题若需要贡献者处理,由评审者自己另发一条 `publish` 评论(file:line、
54+
要做的改动、`evidence` 引用该 `[resolved]` 行);覆盖补充(coverage editor)
55+
的输入也剔除 `[resolved]` 行。此前按关键词从散文里“提拔”残留,会把明确
56+
写着不再追究的行发成 P2(#155)。
5257
- **`review_verdict` 是发布的 state 字段,不是散文**:由
5358
`_review_verdict(review_comments, pr_state)` 计算(**与渲染器同一个
5459
helper,绝不第二份校准规则**),随 `review_text`/`review_summary`/

‎src/infermatrix_copilot/engine/steps/review/prompts.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,12 @@
4747
stating the residual you checked (what the fix does NOT cover): a fix that landed \
4848
mid-PR usually narrows rather than closes, and reviewing its residue is the work. On \
4949
amended/post-review heads these confirmations are most of what a reader checks the \
50-
review against; silence about a resolved concern reads as never having looked. \
50+
review against; silence about a resolved concern reads as never having looked. The \
51+
`[resolved]` line is the RECORD, never the ask, and is never published: when the \
52+
residual needs the contributor to act, ALSO file it as its own comment \
53+
(`disposition: publish`, file:line of the uncovered path, the change or test to \
54+
add, the `[resolved]` line quoted in `evidence`); a residual left only in a \
55+
findings line is not raised. \
5156
CAUTION on absence claims: verify "X was removed/is absent" with `diff_stat`, but \
5257
never render a bare "X is absent from the diff" assertion — on amended PRs the \
5358
reader's thread may reference an earlier revision, and an absolute absence claim \

‎src/infermatrix_copilot/engine/steps/review/refinement.py‎

Lines changed: 7 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -13,69 +13,7 @@
1313

1414
from ....llm import parse_json_reply
1515
from ...step import StepContext
16-
from .utils import _SEVERITY_ORDER, _same_finding
17-
18-
19-
_RESIDUAL_MARKERS = ("residual", "not covered", "does not cover", "still ",
20-
"remains ", "left unfixed", "but ")
21-
22-
23-
def _promote_resolved_residuals(ctx: StepContext, output: dict) -> dict:
24-
"""Turn every `[resolved]` findings line that states a RESIDUAL into a
25-
review comment.
26-
27-
Measured across two holdouts: the ground truth on merged/amended heads is
28-
~70% "a reviewer raised X, the fix landed — what does it still not
29-
cover?", and the passes DO produce that reasoning (the `[resolved]`
30-
contract in prompts.py asks for exactly it). But `[resolved]` is a
31-
FINDINGS line, and findings render into the unscored 'Validated' block —
32-
so the arm's best answers to the dominant question class were routed away
33-
from the only channel a reader (or judge) scores. This promotion is
34-
grounded by construction: it re-files a line the run already wrote and
35-
verified, so it cannot invent a claim, and it sits outside the
36-
coverage-promotion cap because it is not a discretionary addition."""
37-
findings = [str(f) for f in (output.get("findings") or [])]
38-
comments = list(output.get("review_comments") or [])
39-
covered = {(str(c.get("file") or ""), str(c.get("comment") or "")[:60])
40-
for c in comments}
41-
added: list[dict] = []
42-
for line in findings:
43-
low = line.lstrip().lower()
44-
if not low.startswith("[resolved]"):
45-
continue
46-
body = line.split("]", 1)[-1].strip()
47-
if not any(m in body.lower() for m in _RESIDUAL_MARKERS):
48-
continue # a bare confirmation is not a finding
49-
m = re.search(r"([\w./\-]+\.\w+):(\d+)", body)
50-
file_, line_no = (m.group(1), int(m.group(2))) if m else ("", None)
51-
if (file_, body[:60]) in covered:
52-
continue
53-
# ...and against EACH OTHER. `covered` only holds the pre-existing
54-
# comments, so N resolved lines about one residual promoted N times:
55-
# pr4977 shipped four near-identical "the PR description still claims
56-
# trust_remote_code" comments (the cap of 4, saturated) out of five
57-
# `[resolved]` lines stating that one residual, and the judge docked
58-
# it for "4 nearly-identical inline comments restating the same
59-
# PR-description staleness point, hurting signal density". Promotion
60-
# is where that duplication is cheapest to stop — before these
61-
# compete for the comment budget against distinct findings.
62-
if any(_same_finding(body, prev["comment"]) for prev in added):
63-
continue
64-
added.append({"file": file_, "line": line_no, "severity": "minor",
65-
"comment": body,
66-
"evidence": line.strip(),
67-
"corroborated_by": ["resolved-residual"]})
68-
if not added:
69-
ctx.trace.record("review_resolved_promoted", step="agent.review_diff",
70-
added=0, resolved_lines=sum(
71-
1 for f in findings
72-
if f.lstrip().lower().startswith("[resolved]")))
73-
return output
74-
out = dict(output)
75-
out["review_comments"] = comments + added[:4]
76-
ctx.trace.record("review_resolved_promoted", step="agent.review_diff",
77-
added=len(added[:4]))
78-
return out
16+
from .utils import _SEVERITY_ORDER
7917

8018

8119
async def _promote_uncovered(ctx: StepContext, output: dict,
@@ -91,7 +29,12 @@ async def _promote_uncovered(ctx: StepContext, output: dict,
9129
comment. Promotion is grounded by construction — the call may only cite
9230
material from the lines it is shown, never new claims — so it raises
9331
recall without the speculation cost of widening the lens budget."""
94-
findings = [str(x) for x in (output.get("findings") or [])][:60]
32+
# `[resolved]` lines are the review's record of a concern it closed, not
33+
# raw material: a residual that needs action is filed by the reviewer as
34+
# its own `publish` comment (prompts.py), and mining the record here
35+
# re-published lines that explicitly decline their point (#155).
36+
findings = [str(x) for x in (output.get("findings") or [])
37+
if not str(x).lstrip().lower().startswith("[resolved]")][:60]
9538
blockers = [str(x) for x in (output.get("blockers") or [])][:10]
9639
assumptions = [str(x) for x in (output.get("assumptions") or [])][:10]
9740
comments = output.get("review_comments") or []
@@ -516,7 +459,6 @@ async def refine_review(
516459
disposition and comment budget.
517460
"""
518461
if depth != "light":
519-
output = _promote_resolved_residuals(ctx, output)
520462
output = await _promote_uncovered(ctx, output, spec)
521463
output = await _second_round(ctx, output, common, diff)
522464
return await _verify_comments(ctx, output, common)

‎test/test_agent_ensemble.py‎

Lines changed: 38 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1159,28 +1159,44 @@ def test_outcome_blocked_distinguishes_dead_seat_from_quiet_seat():
11591159
"_tools_used": ["grep"]})
11601160

11611161

1162-
def test_resolved_residual_becomes_a_comment(settings, trace, tmp_path):
1163-
"""A `[resolved]` line that states a RESIDUAL is promoted into a scored
1164-
comment. Ground truth on merged/amended heads is ~70% "the fix landed —
1165-
what does it still not cover?", and the passes produce exactly that
1166-
reasoning; it was rendering into the unscored Validated block."""
1167-
from infermatrix_copilot.engine.steps.review.refinement import (
1168-
_promote_resolved_residuals,
1169-
)
1170-
out = _promote_resolved_residuals(
1171-
_ctx(settings, trace, tmp_path),
1172-
{"review_comments": [],
1173-
"findings": [
1174-
"[resolved] prior concern 'guard runs after the collective': "
1175-
"fixed at pipe.py:120 `validate(x)`. Residual: the batch path "
1176-
"at pipe.py:340 still calls it inside the rank-0 branch.",
1177-
"[resolved] prior concern 'missing pin': fixed at req.txt:3.",
1178-
"[sweep] read 40 files, nothing else of note"]})
1179-
kept = out["review_comments"]
1180-
assert len(kept) == 1, "only the residual-bearing line promotes"
1181-
assert kept[0]["file"] == "pipe.py" and kept[0]["line"] == 120
1182-
assert "Residual" in kept[0]["comment"]
1183-
assert next(trace.events("review_resolved_promoted"))["added"] == 1
1162+
# vllm-project/vllm-omni#6968, verbatim: the `[resolved]` line that the old
1163+
# prose promotion published as a P2 even though it says it is not re-raised.
1164+
_DECLINED_RESOLVED = (
1165+
"[resolved] TI2I silent guidance-mode flip on co-batch: request-unique key "
1166+
"(`_boogu_batch_compatibility_key` → (\"boogu_image\", \"ti2i\", request_id)) "
1167+
"+ RuntimeError fail-closed at pipeline_boogu_image.py:667; residual: TI2I "
1168+
"stays batch=1 until RequestBatchSamplingParamsKey adds "
1169+
"`guidance_scale_2_provided` (author follow-up) — not re-raised.")
1170+
1171+
1172+
def test_resolved_line_is_never_mined_into_a_comment(settings, trace, tmp_path):
1173+
"""#155: a `[resolved]` findings line is the review's record, not an ask.
1174+
The prose promotion that turned residual-sounding lines into comments is
1175+
gone, and the coverage editor no longer receives `[resolved]` lines as raw
1176+
material, so the #6968 line cannot become a published finding."""
1177+
from types import SimpleNamespace
1178+
1179+
from infermatrix_copilot.engine.steps.review import refinement
1180+
1181+
assert not hasattr(refinement, "_promote_resolved_residuals")
1182+
seen: list[str] = []
1183+
1184+
class _LLM:
1185+
available = True
1186+
1187+
def create(self, *, messages, **_kw):
1188+
seen.append(messages[0]["content"])
1189+
return SimpleNamespace(text='{"additions": []}')
1190+
1191+
ctx = _ctx(settings, trace, tmp_path, llm=_LLM())
1192+
out = asyncio.run(refinement._promote_uncovered(
1193+
ctx, {"review_comments": [],
1194+
"findings": [_DECLINED_RESOLVED,
1195+
"[validated] pipeline_boogu_image.py:690 image_gs "
1196+
"forced to 1.0 without a reference"]}, {}))
1197+
assert out["review_comments"] == []
1198+
assert seen and "TI2I silent guidance-mode flip" not in seen[0]
1199+
assert "image_gs forced to 1.0" in seen[0]
11841200

11851201

11861202
def test_empty_final_lens_is_retried_not_discarded(settings, trace, tmp_path):

‎test/test_review_render.py‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -156,19 +156,25 @@ def test_findings_with_no_file_do_not_invent_a_broken_reference():
156156
assert "`PR description`" in md and "`general`" in md
157157

158158

159-
def test_promoted_residuals_are_deduped_against_each_other():
160-
"""`covered` only held the pre-existing comments, so N resolved lines
161-
about one residual promoted N times - pr4977 shipped four near-identical
162-
PR-description-staleness comments, saturating the cap of 4."""
163-
import asyncio
164-
from types import SimpleNamespace
165-
from infermatrix_copilot.engine.steps.review.refinement import (
166-
_promote_resolved_residuals)
167-
ctx = SimpleNamespace(trace=SimpleNamespace(record=lambda *a, **k: None))
168-
out = _promote_resolved_residuals(ctx, {"findings": list(_TRC),
169-
"review_comments": []})
170-
assert len(out["review_comments"]) == 1
171-
assert asyncio is not None # keep the import honest for the linter
159+
def test_residual_filed_as_a_publish_comment_is_published():
160+
"""#155: a genuine residual still ships, as the reviewer's own comment.
161+
The pr4977 residual (`_TRC`: the PR description still advertises the
162+
dropped trust_remote_code kwarg) arrives as ONE `publish` comment quoting
163+
its `[resolved]` line; the `[resolved]` lines themselves stay records."""
164+
from infermatrix_copilot.engine.steps.review.utils import (
165+
finalize_review_dispositions)
166+
residual = {"file": "PR description", "line": None, "severity": "minor",
167+
"disposition": "publish",
168+
"comment": "The PR description still advertises the "
169+
"trust_remote_code kwarg that 9947f414 dropped for "
170+
"kernels 0.13.x compat; update it.",
171+
"evidence": _TRC[3]}
172+
record = {"file": "flash_attn_hub.py", "line": 34, "severity": "minor",
173+
"disposition": "resolved", "comment": _TRC[1], "evidence": _TRC[1]}
174+
published, withheld = finalize_review_dispositions(
175+
{"review_comments": [residual, record], "findings": list(_TRC)})
176+
assert [c["comment"] for c in published] == [residual["comment"]]
177+
assert [c["comment"] for c in withheld] == [_TRC[1]]
172178

173179

174180
def test_skill_curation_queue_stays_out_of_the_deliverable():

0 commit comments

Comments
 (0)