Skip to content

Which test-stage chart techniques apply to what consumers of a plugin marketplace build? #15667

Closed

Description

@kyle-sexton

Question

Which techniques on the test-stage chart apply to what consumers of a plugin marketplace build, and which of those does the marketplace already help with?

From the talk

Chapter 8, slide 31: a 2018 chart listing verification techniques by stage, from pre-production to production.

Today and the gap

25 of its 42 techniques are covered at least partly, nearly all pre-production; the gaps are fuzzing, formal verification, contract tests and dynamic exploration. No skill uses the chart as a coverage lens for a consumer repository.

What resolves it

Per technique: does it apply to what consumers build, which skill covers it, and is a coverage lens in /testing:plan or the verification plugin worth adding. Evidence from the chart's primary source and current practice. Basis: menu priority P2.

Done would add

A technique inventory used as a lens by an existing skill, if the research supports one. Dogfood: the lens run against this repository.

Activity

  1. added
    wayfind: researchWayfind decision item: external-evidence question; autonomous-capable.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  4. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: this is a research question. For each technique on the 2018 test-stage chart, decide whether it applies to what marketplace consumers build, and which existing skill covers it. Then decide whether a coverage lens in /testing:plan or the verification plugin is worth adding, and if so, run it against this repository.

    The issue is specific enough to start. The scope is the 42 techniques, the stated gaps (fuzzing, formal verification, contract tests, dynamic exploration) and the priority basis (P2). Nothing is missing from the report. I found no duplicate among the open issues and no merged pull request that already answers this.

    Automated triage. A maintainer reviews it.

  5. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  6. self-assigned this
    on Nov 12, 2026
  7. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor
    No description provided.
  8. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Resolved: no technique-coverage lens. Of the chart's 42 techniques, 26 can live in a consumer repository and 16 need a running production service; the marketplace covers 25 at least partly (17 fully). The repository-side gaps are few and specific, so they fit as rows in /testing:plan's existing table, if the owner wants them, not as a new lens. Basis: the research below, graded by a fresh verifier; the lens verdict itself is judgment.

    From the talk

    • Slide 31 (1:13:58), chapter 8 "Why now": a chart "from this tweet by Cindy Sridharan (@copyconstruct), redrawn". Her quoted tweet: "A mix of the techniques shown here, in short ... turns out this is what I necessarily require to trust agent generated code. It's not optional anymore, imo."
    • Slide 30 (1:13:34), same author: "70% of my daily token spend goes into verifying patches, and only ~20-30% into creating them."
    • The talk uses the chart as a menu of verification techniques, not as a checklist to complete.

    The chart's source

    • Cindy Sridharan, "Testing in Production, the safe way", printed date Mar 25, 2018. The three-phase chart (deploy, release, post-release) is the post's second image and has 39 entries. The post's first image is the two-column predecessor from her December 2017 post.
    • Slide 31 has 42 entries: it adds formal verification, performance tests and soak tests. Not established: whether her 2026 tweet's image or the talk's redraw added them (the tweet image was not viewed; the slide list was read from our digest's transcription).
    • The post limits its production phases to services the engineers deploy themselves. That accounts for 15 of the 16 rows marked as needing a running service; the sixteenth, penetration tests, is marked so by judgment.

    Per technique

    Applies in a consumer repository and covered (17): unit, functional, component, property-based, static analysis, mutation, coverage, benchmark, lint, acceptance, regression, smoke, usability, UI/UX, performance, integration, profiling.

    Applies in a consumer repository, partly covered or not covered (9):

    Technique Today Gap
    Formal verification none parked as Model the loop-lane claim protocol in TLA+ as one bounded proof of concept
    Fuzz tests /testing:audit audits Go fuzz tests it finds no skill plans or writes one (searched: the testing, verification, app-control, bugs:scan and review:security-review skill files and the autonomy routines reference)
    Contract tests /testing:test-value mentions them as a source of expected values no skill plans or writes one (same search scope)
    Threat modeling /review:security-review reviews code no model per feature
    Tap compare offline old-versus-new run in /verification:confirm and /testing:write live replay needs traffic
    Config tests /testing:plan config row consumer application config beyond that
    Tracing local traces in /debugging:analyze-profile and playwright distributed tracing needs a running system
    Dynamic exploration /app-control:drive on a local build covered further by Add a live-app exploring skill to the testing plugin, proven on a fixture app
    Auditing /github:audit, /codebase-health:audit audit trails of production actions

    Needs a running production service, outside what a skill run in a repository can do (16): penetration tests, load tests, shadowing, soak tests, canarying, monitoring (release and post-release), traffic shaping, feature flagging, exception tracking, teeing, logs/events, chaos testing, A/B tests, real user monitoring, oncall experience.

    The count of 25 matches the earlier digest count with two rows moved: fuzz from none to partial, logs/events from partial to none (/harness-ops:observability reads Claude Code's own telemetry, not a consumer application's logs).

    Why no lens

    1. Sixteen rows could only ever be reported as absent, with no action per change.
    2. /testing:plan is change-driven: it maps each changed file to the test types it needs (plugins/testing/skills/plan/SKILL.md:54-66). A repository-wide inventory of the 42 answers a different question that nobody acts on per change.
    3. The gaps a repository can close are two: fuzz and contract tests.

    Open for the owner

    Whether to add two rows to /testing:plan's classification table, each routed to /testing:write and each naming the capability, never a tool (the ecosystem-specifics Brief's rule): a fuzz target for a parser or a handler of untrusted input, and a consumer-driven contract test for a service client or an endpoint another service consumes. The evidence for current practice on both is thin (read as search excerpts only, not accepted as findings).

    Verification

    Six claims accepted, each resting on one publisher's own artifact (her post and image; this repository's files). A fresh verifier passed independent corroboration, confidence and the accepted count, and re-checked the image, the post's date and the cited repository lines. It failed one row twice: several lines in the research files restate a claim without its qualifier or its id. The claims themselves were confirmed; this comment carries the qualifiers. The full research files go on the map's dossier branch.

  9. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Done (2026-11-12): Resolved; see the resolution comment: no technique-coverage lens, with one owner question left on two optional /testing:plan rows.

  10. kyle-sexton commented on Nov 12, 2026

    @kyle-sexton
    MemberAuthor

    Owner decision (2026-11-12, "Let's go with recommendations"): yes to the open question above. /testing:plan gains two rows, each naming the capability and never a tool: a fuzz target for a parser or a handler of untrusted input, and a consumer-driven contract test for a service client or an endpoint another service consumes. It becomes one small work item when the map is decomposed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

wayfind: researchWayfind decision item: external-evidence question; autonomous-capable.

Type

Fields

Work status

ready

Priority

medium

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions