Skip to content

fix(testing): ask for expected-value sources per planned test and run-e2e for API changes - #16577

Merged
kyle-sexton merged 2 commits into
mainfrom
fix/16366-plan-e2e-expected-value
Nov 16, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
fix/16366-plan-e2e-expected-value

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #16366

Summary

In the pl-branch-test-plan eval, /testing:plan replies sometimes planned canned responses for a third-party client without naming testing:test-value. Others skipped /testing:run-e2e for an API route change because no served app existed. The skill already stated both rules, but in the wrong place: the expected-value sentence came after the output list, and the run-e2e suggestion said only "UI/API verification scenarios". This change moves both rules to where the reply writes them. The registered eval below shows the case passes after the change and that the change does no harm. It does not show that the change caused a gain, because the before side also passed every grader in this run.

Fix

  • plugins/testing/skills/plan/SKILL.md, step 2: every planned test entry names where its expected value comes from and cites testing:test-value (if enabled), canned responses for a third-party client included. The unit and integration template rows carry expected value: {source}. The old sentence after the output list is removed.
  • The E2E section heading and the /testing:run-e2e suggestion say an API endpoint change counts even when no served app exists, and point at /testing:run-e2e non-ui, whose playbook covers an API app in process.
  • Changelog fragment for testing (patch).

Verification

Checks run on the measured head 60f8b5313cfd before main was merged in:

  • scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 3 warnings (no Gotchas surface, precompute heuristic, body grew by 2 lines).
  • validate-cases.py plugins/testing/evals: no FAIL; pl-branch-test-plan warns only that Bash needs --allow-tools, as before.
  • markdownlint on the changed files: 0 issues. scripts/check-changelog-fragments.sh --check: valid. scripts/check-cost-figures.sh --check: clean. scripts/check-affected.sh --base origin/main --no-fetch: 16 passed, 2 skipped. The testing plugin's shell suites: pass (one suite skipped for a missing optional node harness).
  • After the merge of main, the pre-push check-affected hook passed on 4e3259591bef.

Eval record, from a pre-registered record written and frozen before any launch:

  • Case pl-branch-test-plan alone, claude plugin eval on Claude Code 2.1.325, model Sonnet, judge claude-opus-5-5, 8 runs a side. BEFORE is the measured head's parent 4d786c973491; AFTER is 60f8b5313cfd. The two plugin copies differ only in skills/plan/SKILL.md.
  • No run excluded on either side; every run's trace shows the skill's gather step ran and loaded that side's copy.
  • Per grader, BEFORE and AFTER: contract-test-for-billing 8/8 and 8/8; fuzz-and-contract-rows 8/8 and 8/8; fuzz-the-hook-parser 8/8 and 8/8; meets-legacy-plan-1 8/8 and 8/8; property-tests-for-parser 8/8 and 8/8; read-the-parser 8/8 and 8/8; types-not-tested 8/8 and 8/8. Case score 1.00: 8 of 8 on both sides.
  • Pass rules: P1 both sides hold 8 launched runs, all counted; P2 no batch check recorded a stop; B1 AFTER case 1.00 in at least 7 of 8; B2 AFTER fuzz-and-contract-rows 8 of 8; B3 AFTER meets-legacy-plan-1 8 of 8. All met. Verdict: MET.
  • Attribution, as the result script prints it: attribution: BEFORE also meets B1-B3; this pair does not show the gain came from the change. This run did not reproduce the misses the issue cites (7 of 8 and 6 of 8 in the earlier kept result). The pair shows the case passes after the change and the change does no harm on it; it does not show the change caused an improvement.
  • Smoke (2 AFTER runs, not counted in the compare): CLEAN, every grader 2 of 2, zero graders: none.
  • Amendment, made before any launch: the compare launcher refuses to start when any of the case's seven graders passed in 0 of the smoke's runs. Pass rules, bars, run counts, exclusions and eval flags did not change.
  • Cost at list price, judge included: smoke 0.556, BEFORE 1.847, AFTER 2.159, total 4.562 USD. The largest single run cost 0.330 USD, under the registered per-run bound of 0.688.

Merge of main: after the eval, main (23 commits) was merged into the branch. git diff 60f8b5313cfd 4e3259591bef -- plugins/testing/skills/plan plugins/testing/evals/pl-branch-test-plan is empty, so the measured paths are byte-identical. Elsewhere in plugins/testing, main changed skills/run-e2e/context/worst-case.md (wording "unbounded" to "no limit found", an initials row), skills/write/context/write.md (refactor-off note), skills/audit/scripts/cant-fail-scan.test.sh, plugin.json and CHANGELOG.md; neither the plan skill's folder nor the case references any of them.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c

kyle-sexton and others added 2 commits November 16, 2026 05:58
…-e2e for API changes

The plan skill now asks each planned test entry to name where its
expected value comes from, citing testing:test-value, beside the entry
in the template instead of after the output list. It also suggests
/testing:run-e2e for an API endpoint change when no served app exists,
pointing at the non-UI route.

Refs: #16366

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 17:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T17:28:56.636768Z 4e32595 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 4e3259591befde9a0279675866bcb41ef00bb153: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read the whole change in both files: plugins/testing/skills/plan/SKILL.md and the new changelog fragment .changes/testing/fix-16366-plan-e2e-expected-value-45498e62.md. Neither file matches a trusted-path glob.

What the change does. It edits only the prose and output template of the /testing:plan skill:

  • Every planned test entry now names where its expected value comes from, citing testing:test-value (if enabled). The unit and integration template rows gain an expected value: {source} field.
  • The E2E heading and the /testing:run-e2e suggestion now say that an API endpoint change counts even when no served app exists.
  • The new changelog fragment is a patch bump.

Instruction-surface check. The diff removes one standing sentence ("For each planned test, name where its expected value will come from: testing:test-value (if enabled).") and restates it, slightly broadened, in step 2 at SKILL.md:80-82. The rule still exists. It only moved, and it protects nothing security-related. None of the changed lines touch a guardrail, an allowlist, a permission, tool grants or a hook.

Other surfaces. No manifest, userConfig, workflow, trust control, token, merge hold or code path changed. No new path lets untrusted text reach a lane.

PR context. It holds no embedded directives aimed at the lane. The closing issue's body was left out upstream because its author isn't on the allowed list.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 4e3259591befde9a0279675866bcb41ef00bb153: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #16577, no findings

This change edits the /testing:plan skill body (plugins/testing/skills/plan/SKILL.md) and adds a testing changelog fragment marked bump: patch. It does two things:

  • Expected-value sentence moved. The sentence that asks for each planned test's expected-value source was after the ## Output list. It is now in step 2, where the plan's entries are written. It also says canned responses for a third-party client are included. The Unit and Integration template rows now carry expected value: {source}, per testing:test-value.
  • API changes without a served app now count for E2E. The E2E section heading and the /testing:run-e2e suggestion now say an API endpoint change counts even when no served app exists. They point at /testing:run-e2e non-ui. That argument exists (plugins/testing/skills/run-e2e/SKILL.md:64), and its playbook has an "API app (in-process)" row (plugins/testing/skills/run-e2e/context/non-ui.md:20), so the pointer leads somewhere real.

Closing issue #16366: Done when

  • plan/SKILL.md says /testing:run-e2e is suggested for an API endpoint change even when no served app exists: met (SKILL.md#L146-L147, heading at L113). The meets-legacy-plan-1 grader of the pl-branch-test-plan eval case covers it.
  • plan/SKILL.md asks for each planned test's expected-value source where the per-test entries are written: met (SKILL.md#L80-L82, template rows at L91 and L95). The fuzz-and-contract-rows grader covers it.
  • /evals:validate passes on pl-branch-test-plan: the author says it passes. I did not run it (unverified).
  • Before/after eval, 8 runs a side on Sonnet: the author recorded it in the PR body. AFTER scored 1.00 in 8 of 8, and both target graders passed 8 of 8. I did not run it (unverified). The author also notes that the BEFORE side scored 8 of 8 too, so this run shows no harm but does not show the change caused a gain.

Path-scoped rules

Two rules under .claude/rules/ match the changed skill file: skill-bodies-state-current-rules.md and cost-claims.md. Neither is broken:

  • The change restates no upstream behavior, limit or version.
  • ## Next is unchanged and still comes before the last H2.
  • The skill body states no cost figure.

Other checks

  • No plugin manifest changed, so the manifest checks do not apply. The fragment's patch bump fits a wording fix to a skill body.
  • Neither changed file matches a TRUSTED PATHS glob.
  • The PR body and issue text contain no instruction aimed at this lane.
  • This is the first review of this pull request, so there are no prior findings to repeat and the review was not blocking only.
  • Spend: the change spends no prepaid credit or subscription usage beyond the skill's normal runs.

Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit b4bf48d Nov 16, 2026
77 checks passed
@kyle-sexton
kyle-sexton deleted the fix/16366-plan-e2e-expected-value branch November 16, 2026 17:40
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(testing): plan replies name testing:test-value for a third-party client and suggest /testing:run-e2e for an API change

1 participant