Repository navigation
test(architecture): fix two record-decision eval graders - #16629
Conversation
lists-citation, in rd-audit-lists-gone-record and
rd-reversal-asks-before-retiring, ended its pattern in \b, so a range
or list starting at the citing line (cookie_store.py:1-2) passed. A
lookahead now rejects a following digit, range or list.
asks-before-retiring missed correct asks phrased with a past participle
("Mark ADR 0002 as superseded by ADR 0003?", "it would be set to
superseded ... Should I update it?"), a present-tense verb, or "needs
your yes" with no question mark. The new branches keep rejecting a
finished retirement followed by an unrelated question.
Each grader has samples that reproduce the old miss.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review (incremental, blocking only): no security issues foundI found no security issues in this push. It changes two eval files, and I read every hunk of both in the incremental diff:
What it touches. Nothing security-relevant:
So skip gate 2 applies. Regex backtracking. The new branch's repeated group Prior findings. The only earlier post is the last round's summary, which reported no findings, so there is nothing to repeat. This review was blocking only: it would have reported CRITICAL and IMPORTANT findings, and dropped any SUGGESTION. Spend. None. Nothing here can cause a charge. critical-findings: 0 |
|
claude-review reviewed Code review (incremental, blocking only): PR #16629This review covers only the push since
This lane already reviewed an earlier head, so this review is blocking only. Neither file matches a trusted path, and no manifest changed. My earlier finding (case.yaml:52) is fixed. I checked the new What changed in the pattern:
How the samples behave under it:
The changelog's claim that the grader rejects "an unrelated question" now holds. The new samples are in the Verification. The owner reports that Spend. There's no new ongoing spend. The only use is the author's eval reruns, on prepaid usage. No blocking findings. Verifier dropped 0 of 0 candidates. |
…ntence In asks-before-retiring, the -s verb forms (deletes, supersedes, ...) sat in the branch whose [^?]*\? tail crosses sentences, so "I wrote ADR 0003, which supersedes ADR 0002. Anything else?" passed. They now need a question mark in the same sentence, as the participle branch does. Two must-fail samples cover the review finding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
Refs: #16552
Summary
Two regex graders in the architecture plugin's record-decision eval cases gave wrong verdicts:
lists-citation(inrd-audit-lists-gone-recordandrd-reversal-asks-before-retiring) was tooloose. The pattern ended in
\b, so a line range or list starting at the citing line(
cookie_store.py:1-2) passed. Only line 1 cites the record. A drop proposal on1-2would alsoedit the docstring, which SKILL.md Step 3 forbids ("edit each proposed citation line as proposed
and nothing else").
asks-before-retiringwas too strict.supersede\bdoes not match "superseded", so a correct askfailed, for example "Should I change it to
Status: superseded by 0003…?". So did an ask thatsays "needs your yes" with no question mark, which the skill's own with-arm wrote.
session: AI Hero
Fix
lists-citation:\bbecomes a lookahead that rejects a following digit, range or list(
:1-2,:1–2,:1,2).:1:,:1,followed by text, and:1 (…)still pass.asks-before-retiring: every retirement verb form is spelled out. A third-person form("supersedes") counts only when the same sentence ends in
?. A past participle counts when thesame sentence ends in
?, or when a later permission question follows ("Should I…","go ahead"). "Needs your yes" counts next to a base-form retirement verb. "Deleted 0002 and wrote
0003. Anything else?" and "Marked 0002 superseded by 0003. Anything else?" still fail. In every
run of both arms, the three
old-record-not-*tool graders confirm that no reply retired 0002..changes/architecture/.Verification
/evals:validateonplugins/architecture/evals: the new samples run against the old patternsgive 16 FAIL findings: 8 samples, each misjudged in both Python and JavaScript
RegExp. After thechange, and with the two eval-run phrasings added as samples: 0 FAIL. Two more must-fail samples
come from the review finding ("…which supersedes ADR 0002. Anything else?"). Every regex grader
in both cases was also checked against its samples under bun 1.4.4: no mismatches.
claude plugin eval(Claude Code 2.1.325,--runs 3,--scaffold, the Docker link swap on WSL2),validity gate
VALIDfor both cases:rd-audit-lists-gone-record: VALID (warnings: skill fired, because the prompt slash-invokes theskill so no Skill call exists, and ceiling). Both arms passed every grader in all three runs.
This is a null measurement: the case cannot show a plugin gain. Spot-read: both arms cite
cookie_store.py:1exactly.rd-reversal-asks-before-retiring, rerun at 52253ab after the review fix that keeps thethird-person forms inside one sentence: VALID (warning: skill fired). The gain is
not established: the noise report says n is too small to call, with one comparable case. The
with-arm passed 8/8 in all three runs. The without-arm passed 7/8 in all three, failing only
lists-citationbecause it runs no citation sweep. Spot-read: the without-arm replies that passasks-before-retiringask "Should I change 0002 toStatus: superseded by …?", and thewith-arm asks "Before I retire ADR 0002, I need your yes".
cookie_store.py:1-2reply comes from the earlier run report. The traces of that runare gone, so it is unverified at its source. The validator's must-fail samples reproduce the shape.
Related
lists-citationgraders and the citation sweep they test.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2