Skip to content

fix(judge): classify varied refusal wording - #5

Open
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-4-leak-verdict-boundary
Open

fix(judge): classify varied refusal wording#5
vsolano9 wants to merge 1 commit into
AgentPostmortem:mainfrom
vsolano9:fix-4-leak-verdict-boundary

Conversation

@vsolano9

Copy link
Copy Markdown

Closes #4

Owner-truth clarification

Current main already has the issue’s safer option-2 scoring architecture: only secret/canary evidence sets cracked: true; both refused and no-leak are held attempts worth zero points. This PR preserves that boundary and fixes the remaining player-message classification gap.

What changed

  • recognize the issue’s unusual refusal forms plus typographic-apostrophe variants
  • keep phrase matching downstream of all secret/canary checks and document that invariant
  • cover nine refusal forms, an ordinary non-refusal no-leak, and refusal wording that also contains a real secret leak

Validation

  • exact Node 20.20.2 / npm 10.8.2: 7 files, 47 tests passed
  • focused judge suite: 17 tests, repeated 20 times
  • npm run typecheck
  • npm run lint — no warnings or errors
  • npm run build — production Next.js build passed
  • changed files pass Prettier; full-repository Prettier finding set is identical to owner main

No dependency or lockfile changes. The fresh install reports the existing 32-advisory baseline (24 production).

Keep secret and canary evidence as the sole crack boundary while recognizing additional common refusal phrasings for the held-result message. Cover straight and typographic apostrophes, generic no-leak output, and leak precedence.

Closes AgentPostmortem#4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refusal detection is five English regexes, and a polite refusal in any other wording reads as a crack attempt

1 participant