Repository navigation
fix(prototype): keep the vendor-fit trigger off internal service contracts - #16580
kyle-sexton wants to merge 1 commit into
Conversation
…racts
The vendor-fit trigger added to /prototype:pressure-test's description
said 'does this outside service do X for us'. The generic word "service"
made the skill win the should-not probe neg-val-05 ("... the contract
the checkout service will call"), raising the validation false-trigger
rate on the lexical floor from 0.6 to 0.8. The trigger now reads 'does
this third-party API do X for us' (vendor-fit mode: an adapter checks it
against a sandbox).
Adds vendor-fit probes: two should-trigger (pos-train-06, pos-val-06)
and two should-not vendor-research requests (neg-train-06, neg-val-06).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: no security issuesI read the whole diff for all 3 changed files:
None of these files matches a trusted path. The change touches no plugin manifest, This was the first review of this pull request, not a blocking-only one. critical-findings: 0 |
|
claude-review reviewed Code review: no findingsThis PR rewords the vendor-fit trigger in I read every hunk of all three changed files:
Closing issue: none; the PR says it fixes a regression from #16554 directly. Manifests: no Paved paths and path rules: nothing in this change touches them. Verification: the trigger rates, No embedded instructions were found in the PR context. Verifier dropped 0 of 0 candidates. |
No related issue: regression found by a lexical trigger check right after #16554 merged, fixed directly
Summary
#16554 added a vendor-fit trigger to
/prototype:pressure-test's description: 'does this outside service do X for us'. The generic word "service" made the skill win the should-not probeneg-val-05("Settle the domain types for loyalty points ... and the contract the checkout service will call ..."), which is design work, not a prototype. The validation false-trigger rate on the lexical floor rose from 0.6 to 0.8. No probe covered vendor-fit.Fix
plugins/prototype/skills/pressure-test/SKILL.md: the trigger now reads 'does this third-party API do X for us' (vendor-fit mode: an adapter checks it against a sandbox). An internal service contract no longer matches it; a question about an outside provider's API still does. The mode's behavior text is unchanged. Description 841/1024 characters (was 847).plugins/prototype/probes/pressure-test.json: four vendor-fit probes, two per split, keeping the file at 6/6/6/6 (24 probes):pos-train-06(should trigger): can Stripe refund part of a captured payment and report settlement by webhook, proven in test mode.pos-val-06(should trigger): does the shipping carrier's API return a tracking number when a label is bought, checked in its sandbox.neg-train-06(should not): research two email providers and recommend one on pricing and support.neg-val-06(should not): compare two SMS vendors on per-message price..changes/prototype/(patch).Verification
measure-invocation.sh score(skill-quality, methodlisting-overlap, a lexical floor, not a model-graded rate),prototype:pressure-test:Original 20 probes:
main(after #16554)With the four new probes (24):
mainneg-val-05is back to a 2-2 tie withexplore-directions(not triggered). No should-trigger probe changed outcome.prototype:explore-directionsrates are unchanged (train 0.8/0.4, validation 1.0/0.8).pos-train-06still misses (4-4 tie with explore-directions; it missed atmaintoo) andneg-val-06still false-triggers (on "vendor" from the mode name and "which"; it triggered atmaintoo). Both were left as written rather than tuning wording to the held-out probe.measure-invocation.sh validate plugins/prototype/probes: PASS, 0 warnings.check-skill.sh plugins/prototype/skills: 3 passed, 0 failed. One expected WARN: the quoted trigger 'does this outside service do X for us' was reworded.check-listing-budget.sh plugins/prototype/skills: 1852/8000, OK.node scripts/generate-cheatsheet.mjs: already in sync (the cheat sheet readssummary, not the description).scripts/check-changelog-fragments.sh --check: valid.Related
session: AI Hero
https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
🤖 Generated with Claude Code