You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Which test-stage chart techniques apply to what consumers of a plugin marketplace build? #15667
Which techniques on the test-stage chart apply to what consumers of a plugin marketplace build, and which of those does the marketplace already help with?
From the talk
Chapter 8, slide 31: a 2018 chart listing verification techniques by stage, from pre-production to production.
Today and the gap
25 of its 42 techniques are covered at least partly, nearly all pre-production; the gaps are fuzzing, formal verification, contract tests and dynamic exploration. No skill uses the chart as a coverage lens for a consumer repository.
What resolves it
Per technique: does it apply to what consumers build, which skill covers it, and is a coverage lens in /testing:plan or the verification plugin worth adding. Evidence from the chart's primary source and current practice. Basis: menu priority P2.
Done would add
A technique inventory used as a lens by an existing skill, if the research supports one. Dogfood: the lens run against this repository.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: this is a research question. For each technique on the 2018 test-stage chart, decide whether it applies to what marketplace consumers build, and which existing skill covers it. Then decide whether a coverage lens in /testing:plan or the verification plugin is worth adding, and if so, run it against this repository.
The issue is specific enough to start. The scope is the 42 techniques, the stated gaps (fuzzing, formal verification, contract tests, dynamic exploration) and the priority basis (P2). Nothing is missing from the report. I found no duplicate among the open issues and no merged pull request that already answers this.
Resolved: no technique-coverage lens. Of the chart's 42 techniques, 26 can live in a consumer repository and 16 need a running production service; the marketplace covers 25 at least partly (17 fully). The repository-side gaps are few and specific, so they fit as rows in /testing:plan's existing table, if the owner wants them, not as a new lens. Basis: the research below, graded by a fresh verifier; the lens verdict itself is judgment.
From the talk
Slide 31 (1:13:58), chapter 8 "Why now": a chart "from this tweet by Cindy Sridharan (@copyconstruct), redrawn". Her quoted tweet: "A mix of the techniques shown here, in short ... turns out this is what I necessarily require to trust agent generated code. It's not optional anymore, imo."
Slide 30 (1:13:34), same author: "70% of my daily token spend goes into verifying patches, and only ~20-30% into creating them."
The talk uses the chart as a menu of verification techniques, not as a checklist to complete.
The chart's source
Cindy Sridharan, "Testing in Production, the safe way", printed date Mar 25, 2018. The three-phase chart (deploy, release, post-release) is the post's second image and has 39 entries. The post's first image is the two-column predecessor from her December 2017 post.
Slide 31 has 42 entries: it adds formal verification, performance tests and soak tests. Not established: whether her 2026 tweet's image or the talk's redraw added them (the tweet image was not viewed; the slide list was read from our digest's transcription).
The post limits its production phases to services the engineers deploy themselves. That accounts for 15 of the 16 rows marked as needing a running service; the sixteenth, penetration tests, is marked so by judgment.
Per technique
Applies in a consumer repository and covered (17): unit, functional, component, property-based, static analysis, mutation, coverage, benchmark, lint, acceptance, regression, smoke, usability, UI/UX, performance, integration, profiling.
Applies in a consumer repository, partly covered or not covered (9):
no skill plans or writes one (searched: the testing, verification, app-control, bugs:scan and review:security-review skill files and the autonomy routines reference)
Contract tests
/testing:test-value mentions them as a source of expected values
no skill plans or writes one (same search scope)
Threat modeling
/review:security-review reviews code
no model per feature
Tap compare
offline old-versus-new run in /verification:confirm and /testing:write
live replay needs traffic
Config tests
/testing:plan config row
consumer application config beyond that
Tracing
local traces in /debugging:analyze-profile and playwright
Needs a running production service, outside what a skill run in a repository can do (16): penetration tests, load tests, shadowing, soak tests, canarying, monitoring (release and post-release), traffic shaping, feature flagging, exception tracking, teeing, logs/events, chaos testing, A/B tests, real user monitoring, oncall experience.
The count of 25 matches the earlier digest count with two rows moved: fuzz from none to partial, logs/events from partial to none (/harness-ops:observability reads Claude Code's own telemetry, not a consumer application's logs).
Why no lens
Sixteen rows could only ever be reported as absent, with no action per change.
/testing:plan is change-driven: it maps each changed file to the test types it needs (plugins/testing/skills/plan/SKILL.md:54-66). A repository-wide inventory of the 42 answers a different question that nobody acts on per change.
The gaps a repository can close are two: fuzz and contract tests.
Open for the owner
Whether to add two rows to /testing:plan's classification table, each routed to /testing:write and each naming the capability, never a tool (the ecosystem-specifics Brief's rule): a fuzz target for a parser or a handler of untrusted input, and a consumer-driven contract test for a service client or an endpoint another service consumes. The evidence for current practice on both is thin (read as search excerpts only, not accepted as findings).
Verification
Six claims accepted, each resting on one publisher's own artifact (her post and image; this repository's files). A fresh verifier passed independent corroboration, confidence and the accepted count, and re-checked the image, the post's date and the cited repository lines. It failed one row twice: several lines in the research files restate a claim without its qualifier or its id. The claims themselves were confirmed; this comment carries the qualifiers. The full research files go on the map's dossier branch.
Owner decision (2026-11-12, "Let's go with recommendations"): yes to the open question above. /testing:plan gains two rows, each naming the capability and never a tool: a fuzz target for a parser or a handler of untrusted input, and a consumer-driven contract test for a service client or an endpoint another service consumes. It becomes one small work item when the map is decomposed.
Activity
github-actions commentedon Nov 12, 2026
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: this is a research question. For each technique on the 2018 test-stage chart, decide whether it applies to what marketplace consumers build, and which existing skill covers it. Then decide whether a coverage lens in
/testing:planor the verification plugin is worth adding, and if so, run it against this repository.The issue is specific enough to start. The scope is the 42 techniques, the stated gaps (fuzzing, formal verification, contract tests, dynamic exploration) and the priority basis (P2). Nothing is missing from the report. I found no duplicate among the open issues and no merged pull request that already answers this.
Automated triage. A maintainer reviews it.
kyle-sexton commentedon Nov 12, 2026
kyle-sexton commentedon Nov 12, 2026
Resolved: no technique-coverage lens. Of the chart's 42 techniques, 26 can live in a consumer repository and 16 need a running production service; the marketplace covers 25 at least partly (17 fully). The repository-side gaps are few and specific, so they fit as rows in
/testing:plan's existing table, if the owner wants them, not as a new lens. Basis: the research below, graded by a fresh verifier; the lens verdict itself is judgment.From the talk
The chart's source
Per technique
Applies in a consumer repository and covered (17): unit, functional, component, property-based, static analysis, mutation, coverage, benchmark, lint, acceptance, regression, smoke, usability, UI/UX, performance, integration, profiling.
Applies in a consumer repository, partly covered or not covered (9):
/testing:auditaudits Go fuzz tests it findsbugs:scanandreview:security-reviewskill files and the autonomy routines reference)/testing:test-valuementions them as a source of expected values/review:security-reviewreviews code/verification:confirmand/testing:write/testing:planconfig row/debugging:analyze-profileand playwright/app-control:driveon a local build/github:audit,/codebase-health:auditNeeds a running production service, outside what a skill run in a repository can do (16): penetration tests, load tests, shadowing, soak tests, canarying, monitoring (release and post-release), traffic shaping, feature flagging, exception tracking, teeing, logs/events, chaos testing, A/B tests, real user monitoring, oncall experience.
The count of 25 matches the earlier digest count with two rows moved: fuzz from none to partial, logs/events from partial to none (
/harness-ops:observabilityreads Claude Code's own telemetry, not a consumer application's logs).Why no lens
/testing:planis change-driven: it maps each changed file to the test types it needs (plugins/testing/skills/plan/SKILL.md:54-66). A repository-wide inventory of the 42 answers a different question that nobody acts on per change.Open for the owner
Whether to add two rows to
/testing:plan's classification table, each routed to/testing:writeand each naming the capability, never a tool (the ecosystem-specifics Brief's rule): a fuzz target for a parser or a handler of untrusted input, and a consumer-driven contract test for a service client or an endpoint another service consumes. The evidence for current practice on both is thin (read as search excerpts only, not accepted as findings).Verification
Six claims accepted, each resting on one publisher's own artifact (her post and image; this repository's files). A fresh verifier passed independent corroboration, confidence and the accepted count, and re-checked the image, the post's date and the cited repository lines. It failed one row twice: several lines in the research files restate a claim without its qualifier or its id. The claims themselves were confirmed; this comment carries the qualifiers. The full research files go on the map's dossier branch.
kyle-sexton commentedon Nov 12, 2026
Done (2026-11-12): Resolved; see the resolution comment: no technique-coverage lens, with one owner question left on two optional /testing:plan rows.
kyle-sexton commentedon Nov 12, 2026
Owner decision (2026-11-12, "Let's go with recommendations"): yes to the open question above.
/testing:plangains two rows, each naming the capability and never a tool: a fuzz target for a parser or a handler of untrusted input, and a consumer-driven contract test for a service client or an endpoint another service consumes. It becomes one small work item when the map is decomposed.