Skip to content

fix(prototype): keep the vendor-fit trigger off internal service contracts - #16580

Queued
kyle-sexton wants to merge 1 commit into
mainfrom
fix/prototype-pressure-test-vendor-fit-trigger
Queued

kyle-sexton wants to merge 1 commit into
mainfrom
fix/prototype-pressure-test-vendor-fit-trigger

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: regression found by a lexical trigger check right after #16554 merged, fixed directly

Summary

#16554 added a vendor-fit trigger to /prototype:pressure-test's description: 'does this outside service do X for us'. The generic word "service" made the skill win the should-not probe neg-val-05 ("Settle the domain types for loyalty points ... and the contract the checkout service will call ..."), which is design work, not a prototype. The validation false-trigger rate on the lexical floor rose from 0.6 to 0.8. No probe covered vendor-fit.

Fix

  • plugins/prototype/skills/pressure-test/SKILL.md: the trigger now reads 'does this third-party API do X for us' (vendor-fit mode: an adapter checks it against a sandbox). An internal service contract no longer matches it; a question about an outside provider's API still does. The mode's behavior text is unchanged. Description 841/1024 characters (was 847).
  • plugins/prototype/probes/pressure-test.json: four vendor-fit probes, two per split, keeping the file at 6/6/6/6 (24 probes):
    • pos-train-06 (should trigger): can Stripe refund part of a captured payment and report settlement by webhook, proven in test mode.
    • pos-val-06 (should trigger): does the shipping carrier's API return a tracking number when a label is bought, checked in its sandbox.
    • neg-train-06 (should not): research two email providers and recommend one on pricing and support.
    • neg-val-06 (should not): compare two SMS vendors on per-message price.
  • Changelog fragment .changes/prototype/ (patch).

Verification

measure-invocation.sh score (skill-quality, method listing-overlap, a lexical floor, not a model-graded rate), prototype:pressure-test:

Original 20 probes:

Description Train trigger Train false-trigger Validation trigger Validation false-trigger
Before #16554 0.6 0.6 1.0 0.6
main (after #16554) 0.6 0.6 1.0 0.8
This PR 0.6 0.6 1.0 0.6

With the four new probes (24):

Description Train trigger Train false-trigger Validation trigger Validation false-trigger
main 0.5 0.5 1.0 0.83 (5/6)
This PR 0.5 0.5 1.0 0.67 (4/6)
  • neg-val-05 is back to a 2-2 tie with explore-directions (not triggered). No should-trigger probe changed outcome. prototype:explore-directions rates are unchanged (train 0.8/0.4, validation 1.0/0.8).
  • On the floor, pos-train-06 still misses (4-4 tie with explore-directions; it missed at main too) and neg-val-06 still false-triggers (on "vendor" from the mode name and "which"; it triggered at main too). Both were left as written rather than tuning wording to the held-out probe.
  • measure-invocation.sh validate plugins/prototype/probes: PASS, 0 warnings.
  • check-skill.sh plugins/prototype/skills: 3 passed, 0 failed. One expected WARN: the quoted trigger 'does this outside service do X for us' was reworded.
  • check-listing-budget.sh plugins/prototype/skills: 1852/8000, OK.
  • node scripts/generate-cheatsheet.mjs: already in sync (the cheat sheet reads summary, not the description).
  • scripts/check-changelog-fragments.sh --check: valid.

Related

session: AI Hero

https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2

🤖 Generated with Claude Code

…racts

The vendor-fit trigger added to /prototype:pressure-test's description
said 'does this outside service do X for us'. The generic word "service"
made the skill win the should-not probe neg-val-05 ("... the contract
the checkout service will call"), raising the validation false-trigger
rate on the lexical floor from 0.6 to 0.8. The trigger now reads 'does
this third-party API do X for us' (vendor-fit mode: an adapter checks it
against a sandbox).

Adds vendor-fit probes: two should-trigger (pos-train-06, pos-val-06)
and two should-not vendor-research requests (neg-train-06, neg-val-06).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASNT5Knyq99gphnV6B2
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 17:31
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T17:32:23.675947Z fd218f0 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed fd218f09e937d1a5a24b8f8aa22c3528b3fcf5d0: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues

I read the whole diff for all 3 changed files:

  • .changes/prototype/fix-prototype-pressure-test-vendor-fit-trigger-9c463248.md: a new changelog fragment (patch bump) with prose only.
  • plugins/prototype/probes/pressure-test.json: four new invocation probes. Each is a short request string with an expect_trigger boolean. They are eval data and nothing runs them as commands.
  • plugins/prototype/skills/pressure-test/SKILL.md: the frontmatter description changes its vendor-fit trigger from "does this outside service do X for us" to "does this third-party API do X for us". argument-hint, user-invocable, disable-model-invocation and the skill body are unchanged. No tool grant is added.

None of these files matches a trusted path. The change touches no plugin manifest, userConfig key, hook, workflow, token request, merge hold or trust control. The description rewording only changes when the model picks this skill. It removes no guardrail, so the instruction-surface lens has nothing to check. The PR context file contains no embedded directives aimed at the lane.

This was the first review of this pull request, not a blocking-only one.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed fd218f09e937d1a5a24b8f8aa22c3528b3fcf5d0: read 3 of 3 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

This PR rewords the vendor-fit trigger in /prototype:pressure-test's description. The old text was 'does this outside service do X for us' and the new text is 'does this third-party API do X for us'. The aim is to stop the generic word "service" from pulling in internal design work, such as the neg-val-05 probe about a checkout service contract. The PR also adds four vendor-fit invocation probes and a patch changelog fragment.

I read every hunk of all three changed files:

  • Changelog fragment. bump: patch fits a description-only wording fix.
  • plugins/prototype/probes/pressure-test.json. The four new probes are well-formed JSON, and their IDs don't clash with existing ones. The splits are balanced (train and validation, positive and negative), so the file stays at 6/6/6/6. The negatives are vendor research and price comparison, not hands-on sandbox checks, which is the right boundary for vendor-fit.
  • plugins/prototype/skills/pressure-test/SKILL.md. Only the description's trigger phrase changed. The skill body still says "outside service" (the section heading at line 201, its anchor, and the table row at line 78). That is behaviour text, not trigger text, and the PR says on purpose that it left it unchanged. No links break because the anchor is unchanged.

Closing issue: none; the PR says it fixes a regression from #16554 directly.

Manifests: no plugin.json or marketplace.json changed, so there are no manifest lines.

Paved paths and path rules: nothing in this change touches them.

Verification: the trigger rates, check-skill.sh, the listing-budget check and the fragment check in the PR body are the author's claims. I did not run any of them.

No embedded instructions were found in the PR context.

Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant