Skip to content

test(architecture): fix two record-decision eval graders - #16629

Merged
kyle-sexton merged 3 commits into
mainfrom
fix/architecture-eval-grader-tightening
Nov 16, 2026
Merged

kyle-sexton merged 3 commits into
mainfrom
fix/architecture-eval-grader-tightening

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Refs: #16552

Summary

Two regex graders in the architecture plugin's record-decision eval cases gave wrong verdicts:

  • lists-citation (in rd-audit-lists-gone-record and rd-reversal-asks-before-retiring) was too
    loose. The pattern ended in \b, so a line range or list starting at the citing line
    (cookie_store.py:1-2) passed. Only line 1 cites the record. A drop proposal on 1-2 would also
    edit the docstring, which SKILL.md Step 3 forbids ("edit each proposed citation line as proposed
    and nothing else").
  • asks-before-retiring was too strict. supersede\b does not match "superseded", so a correct ask
    failed, for example "Should I change it to Status: superseded by 0003…?". So did an ask that
    says "needs your yes" with no question mark, which the skill's own with-arm wrote.

session: AI Hero

Fix

  • lists-citation: \b becomes a lookahead that rejects a following digit, range or list
    (:1-2, :1–2, :1,2). :1:, :1, followed by text, and :1 (…) still pass.
  • asks-before-retiring: every retirement verb form is spelled out. A third-person form
    ("supersedes") counts only when the same sentence ends in ?. A past participle counts when the
    same sentence ends in ?, or when a later permission question follows ("Should I…",
    "go ahead"). "Needs your yes" counts next to a base-form retirement verb. "Deleted 0002 and wrote
    0003. Anything else?" and "Marked 0002 superseded by 0003. Anything else?" still fail. In every
    run of both arms, the three old-record-not-* tool graders confirm that no reply retired 0002.
  • Samples added to each grader: the old pattern fails each new one. The changelog fragment is under
    .changes/architecture/.

Verification

  • /evals:validate on plugins/architecture/evals: the new samples run against the old patterns
    give 16 FAIL findings: 8 samples, each misjudged in both Python and JavaScript RegExp. After the
    change, and with the two eval-run phrasings added as samples: 0 FAIL. Two more must-fail samples
    come from the review finding ("…which supersedes ADR 0002. Anything else?"). Every regex grader
    in both cases was also checked against its samples under bun 1.4.4: no mismatches.
  • claude plugin eval (Claude Code 2.1.325, --runs 3, --scaffold, the Docker link swap on WSL2),
    validity gate VALID for both cases:
    • rd-audit-lists-gone-record: VALID (warnings: skill fired, because the prompt slash-invokes the
      skill so no Skill call exists, and ceiling). Both arms passed every grader in all three runs.
      This is a null measurement: the case cannot show a plugin gain. Spot-read: both arms cite
      cookie_store.py:1 exactly.
    • rd-reversal-asks-before-retiring, rerun at 52253ab after the review fix that keeps the
      third-person forms inside one sentence: VALID (warning: skill fired). The gain is
      not established: the noise report says n is too small to call, with one comparable case. The
      with-arm passed 8/8 in all three runs. The without-arm passed 7/8 in all three, failing only
      lists-citation because it runs no citation sweep. Spot-read: the without-arm replies that pass
      asks-before-retiring ask "Should I change 0002 to Status: superseded by …?", and the
      with-arm asks "Before I retire ADR 0002, I need your yes".
  • The quoted cookie_store.py:1-2 reply comes from the earlier run report. The traces of that run
    are gone, so it is unverified at its source. The validator's must-fail samples reproduce the shape.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2

lists-citation, in rd-audit-lists-gone-record and
rd-reversal-asks-before-retiring, ended its pattern in \b, so a range
or list starting at the citing line (cookie_store.py:1-2) passed. A
lookahead now rejects a following digit, range or list.

asks-before-retiring missed correct asks phrased with a past participle
("Mark ADR 0002 as superseded by ADR 0003?", "it would be set to
superseded ... Should I update it?"), a present-tense verb, or "needs
your yes" with no question mark. The new branches keep rejecting a
finished retirement followed by an unrelated question.

Each grader has samples that reproduce the old miss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 20:06
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T20:08:07.613854Z 68a273d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 2b79f3f65b3b785f8465f9fae615fbcc8eec1fec: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, blocking only): no security issues found

I found no security issues in this push. It changes two eval files, and I read every hunk of both in the incremental diff:

  • plugins/architecture/evals/rd-reversal-asks-before-retiring/case.yaml: the asks-before-retiring regex grader now handles third-person verbs (deletes, removes, retires, supersedes, archives) separately. These verbs now need a question mark in the same sentence, so an unrelated question in a later sentence no longer counts as asking first.
  • plugins/architecture/evals/rd-reversal-asks-before-retiring/samples/asks-before-retiring.json: two new sample answers that the grader must not accept.

What it touches. Nothing security-relevant:

  • no trusted path
  • no trust control (trusted-trigger, lane token, kill switch, broker or merge hold)
  • no workflow, plugin manifest, hook or userConfig key
  • no standing instruction deleted, narrowed or softened

So skip gate 2 applies.

Regex backtracking. The new branch's repeated group (?:[^?.\n]|\.(?=\S))* has two alternatives that can never match the same character, so it cannot backtrack catastrophically. The pattern also only runs against eval model output (target: last_message), which no attacker controls. Not a finding.

Prior findings. The only earlier post is the last round's summary, which reported no findings, so there is nothing to repeat. This review was blocking only: it would have reported CRITICAL and IMPORTANT findings, and dropped any SUGGESTION.

Spend. None. Nothing here can cause a charge.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 68a273dbf1c563bdfa2796bd6f038ad69f6408c0; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 2b79f3f65b3b785f8465f9fae615fbcc8eec1fec: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, blocking only): PR #16629

This review covers only the push since 68a273db. It changes two files, and I read every hunk of both:

  • plugins/architecture/evals/rd-reversal-asks-before-retiring/case.yaml
  • plugins/architecture/evals/rd-reversal-asks-before-retiring/samples/asks-before-retiring.json

This lane already reviewed an earlier head, so this review is blocking only. Neither file matches a trusted path, and no manifest changed.

My earlier finding (case.yaml:52) is fixed. I checked the new asks-before-retiring pattern by hand.

What changed in the pattern:

  • The third-person forms (deletes|removes|retires|supersedes|archives) now have their own branch, (?:[^?.\n]|\.(?=\S))*\?. That branch counts a question mark only when it ends the same sentence, the same rule the participle branch uses.
  • The base and -ing forms keep the tail that can reach a question mark in a later sentence.

How the samples behave under it:

  • New fail samples. Both now fail:

    • "I wrote ADR 0003, which supersedes ADR 0002. Anything else?"
    • "This removes the GraphQL adapter. Anything else?"

    In each, supersede\b and remove\b cannot match inside the -s word, and the new branch stops at . .

  • Existing pass samples. They still pass. For example, in This deletes docs/adr/0002-….md; go ahead? each . is followed by a non-space, so the match reaches ?.

  • Existing fail samples. They still fail.

The changelog's claim that the grader rejects "an unrelated question" now holds. The new samples are in the fail list.

Verification. The owner reports that /evals:validate gave 0 FAIL and that the case rerun was VALID. I have no shell here, so both claims are unverified. I checked the regex by hand only.

Spend. There's no new ongoing spend. The only use is the author's eval reruns, on prepaid usage.

No blocking findings.

Verifier dropped 0 of 0 candidates.

kyle-sexton and others added 2 commits November 16, 2026 15:16
…ntence

In asks-before-retiring, the -s verb forms (deletes, supersedes, ...) sat
in the branch whose [^?]*\? tail crosses sentences, so "I wrote ADR 0003,
which supersedes ADR 0002. Anything else?" passed. They now need a
question mark in the same sentence, as the participle branch does. Two
must-fail samples cover the review finding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 7765034 Nov 16, 2026
45 checks passed
@kyle-sexton
kyle-sexton deleted the fix/architecture-eval-grader-tightening branch November 16, 2026 21:23
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant