Repository navigation
fix(testing): ask for expected-value sources per planned test and run-e2e for API changes - #16577
Conversation
…-e2e for API changes The plan skill now asks each planned test entry to name where its expected value comes from, citing testing:test-value, beside the entry in the template instead of after the output list. It also suggests /testing:run-e2e for an API endpoint change when no served app exists, pointing at the non-UI route. Refs: #16366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: no security issues foundI read the whole change in both files: What the change does. It edits only the prose and output template of the
Instruction-surface check. The diff removes one standing sentence ("For each planned test, name where its expected value will come from: Other surfaces. No manifest, PR context. It holds no embedded directives aimed at the lane. The closing issue's body was left out upstream because its author isn't on the allowed list. critical-findings: 0 |
|
claude-review reviewed Code review: PR #16577, no findingsThis change edits the
Closing issue #16366: Done when
Path-scoped rulesTwo rules under
Other checks
Verifier dropped 0 of 0 candidates. |
Closes #16366
Summary
In the
pl-branch-test-planeval,/testing:planreplies sometimes planned canned responses for a third-party client without namingtesting:test-value. Others skipped/testing:run-e2efor an API route change because no served app existed. The skill already stated both rules, but in the wrong place: the expected-value sentence came after the output list, and the run-e2e suggestion said only "UI/API verification scenarios". This change moves both rules to where the reply writes them. The registered eval below shows the case passes after the change and that the change does no harm. It does not show that the change caused a gain, because the before side also passed every grader in this run.Fix
plugins/testing/skills/plan/SKILL.md, step 2: every planned test entry names where its expected value comes from and citestesting:test-value(if enabled), canned responses for a third-party client included. The unit and integration template rows carryexpected value: {source}. The old sentence after the output list is removed./testing:run-e2esuggestion say an API endpoint change counts even when no served app exists, and point at/testing:run-e2e non-ui, whose playbook covers an API app in process.testing(patch).Verification
Checks run on the measured head
60f8b5313cfdbefore main was merged in:scripts/check-changed-skills.sh origin/main: PASS, 0 errors, 3 warnings (no Gotchas surface, precompute heuristic, body grew by 2 lines).validate-cases.py plugins/testing/evals: no FAIL;pl-branch-test-planwarns only that Bash needs--allow-tools, as before.scripts/check-changelog-fragments.sh --check: valid.scripts/check-cost-figures.sh --check: clean.scripts/check-affected.sh --base origin/main --no-fetch: 16 passed, 2 skipped. The testing plugin's shell suites: pass (one suite skipped for a missing optional node harness).check-affectedhook passed on4e3259591bef.Eval record, from a pre-registered record written and frozen before any launch:
pl-branch-test-planalone,claude plugin evalon Claude Code 2.1.325, model Sonnet, judgeclaude-opus-5-5, 8 runs a side. BEFORE is the measured head's parent4d786c973491; AFTER is60f8b5313cfd. The two plugin copies differ only inskills/plan/SKILL.md.contract-test-for-billing8/8 and 8/8;fuzz-and-contract-rows8/8 and 8/8;fuzz-the-hook-parser8/8 and 8/8;meets-legacy-plan-18/8 and 8/8;property-tests-for-parser8/8 and 8/8;read-the-parser8/8 and 8/8;types-not-tested8/8 and 8/8. Case score 1.00: 8 of 8 on both sides.fuzz-and-contract-rows8 of 8; B3 AFTERmeets-legacy-plan-18 of 8. All met. Verdict: MET.attribution: BEFORE also meets B1-B3; this pair does not show the gain came from the change. This run did not reproduce the misses the issue cites (7 of 8 and 6 of 8 in the earlier kept result). The pair shows the case passes after the change and the change does no harm on it; it does not show the change caused an improvement.zero graders: none.Merge of main: after the eval, main (23 commits) was merged into the branch.
git diff 60f8b5313cfd 4e3259591bef -- plugins/testing/skills/plan plugins/testing/evals/pl-branch-test-planis empty, so the measured paths are byte-identical. Elsewhere inplugins/testing, main changedskills/run-e2e/context/worst-case.md(wording "unbounded" to "no limit found", an initials row),skills/write/context/write.md(refactor-off note),skills/audit/scripts/cant-fail-scan.test.sh,plugin.jsonandCHANGELOG.md; neither the plan skill's folder nor the case references any of them.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c