Repository navigation
Description
Decision (owner, 2026-11-12, revised the same day; conditions 2026-11-13; answers 2026-11-15)
Source: the talk "All Code Is Slop (Except Mine)" (Lauren Tan, AI Engineer Code Summit, 2026-11-11), slides 42-43 and the transcript at 1:17:48: an agent runs the whole build "like a real user", and the report marks each finding new or already present ("NEW: 3 bugs ... PRE-EXISTING: 47 bugs", "Each reply keeps its finding id").
No new skill. The pass is a composite of skills that exist, plus three missing parts:
- A whole-map scope on the existing live verification. Today a live verification run covers only the features a change touches (
/testing:run-e2e). The pass runs the same judging over every entry in the feature map, with no change to diff against. - A routine row in the
autonomyplugin, with no schedule. - The comparison with the previous pass in that pass's report: each finding marked new, pre-existing or gone, keyed by feature-map entry, with an id that stays the same across runs. A first run reports everything as new.
Findings go through /bugs:write; fixes go through the existing loop lanes. /app-control:drive still only drives and records.
The owner's conditions of 2026-11-13 (comment "Interview status, 2026-11-13: deferred until #15913 lands") replace "on each release": the pass starts by hand only, with no schedule until one measured pass has been reviewed; the owner controls when and where it triggers, and it is easy to turn on and off and to run on demand; any later cadence is weekly, every two weeks or monthly, never per release; it proves its value with measurements against tokens, time and plan usage; driven sessions run on Opus 5.5, as a setting.
Answers (interview, 2026-11-15; restatement confirmed by the owner in session, "go w/ recommendations")
- Trigger. On demand only: a workflow whose only trigger is manual dispatch by a listed human, turned on and off with
gh workflow enableanddisable, stopped by the lanes kill switch. The comparison is with the previous pass, not with a release. - Credential and first pass (owner). A pass spends the pr-verify API key inside the pull-request capture's sandbox and key proxy, under the existing credit gate and per-job token cap, because only that sandbox keeps a driven skill's shell. The first measured pass is capped at about 40 skills across at least 8 plugins; a whole-map pass is decided later with its numbers.
- Home.
/testing:run-e2egains amapscope: every feature of the control skill, no base ref, entries driven one after another, its control-skill refresh step skipped./verification:confirmis unchanged. - Unit. Settings
map_unit(featuredefault, orpart) andmap_entry_points(alldefault, orfirst); this repository setspartandfirst, so one entry is one skill through its print-mode slash command. - Defects the map states nothing about. Reported, labeled
unstated, from a closed list of kinds (error, crash, refusal, no output, garbled output). This repository's stated results only say a skill's output appears, so this is where most findings come from. - Finding id. A short hash of feature, part, failure kind and where (the drive step number,
proof, orrun); the entry point is not in it, so one failure seen through several entry points is one finding. Matched across passes by exact id. A blocked or not-driven entry isnot checked, never a finding, and never makes a finding GONE. - Baseline. The skill reads
baseline=<findings.json>and writes its ownfindings.json; storage is the caller's. This repository keeps it as a workflow artifact read by the next dispatch. Nothing is committed. - Report. NEW, then PRE-EXISTING, then GONE, each by severity (
critical,high,medium,low), a summary line per group, both pass builds in the header, measured tokens and wall time per entry. Settingmap_report_order. - Filing. Settings
file_findings(nonedefault, ornew) andfile_findings_floor(highdefault). The first pass files nothing. Whether this repository setsnewis deferred by the owner to the review of the second measured pass. - Duplicates. Deterministic only: one finding per id; no model merge.
- Routine. One
autonomycatalog row,whole-map-live-verification, statusjoin: proven recurring manual pattern, so no schedule and no leaf until a consumer records a manual run.
Done when
- The testing plugin's findings compare script takes two
findings.jsonfiles and labels each finding id NEW, PRE-EXISTING, GONE or not checked, with no model call. - The testing plugin's findings compare script never labels an id GONE when its entry was not driven in the newer pass, and labels every finding NEW when no baseline is given.
- The testing plugin's findings compare script has a test file covering each label and the not-checked rule, run by CI.
- The testing plugin's
findings.jsonschema holds per finding the id, feature, part, kind, where, entry points, severity,statedorunstatedlabel and evidence paths, and per pass the build and each entry's outcome, wall time and tokens. -
/testing:run-e2e mapdrives every feature of the control skill without calling drive'saffectedaction and without a base ref, and acceptsfeatures=to narrow it andmax_entries=to cap it, reporting the restnot checked (cap). -
/testing:run-e2ehas settingsmap_unit(featuredefault,part) andmap_entry_points(alldefault,first) in its plugin options and repository conventions, the repository value winning, and underpartdrives each<feature>/<part>drive resolves. -
/testing:run-e2e maprecords a failed stated result or proof step as a finding, records observed breakage as anunstatedfinding of kind error, crash, refusal, no-output or garbled-output, and records a blocked entry asnot checkedwith its reason. -
/testing:run-e2e mapgives each finding an id hashed from feature, part, kind and where, without the entry point, listing every entry point the finding showed at. -
/testing:run-e2e mapskips its control-skill refresh step, edits no tracked file, and writesfindings.jsonande2e-summary.jsonat the top of its run directory. -
/testing:run-e2e mapreports each entry's measured wall time and tokens, or null with the reason, and the pass totals, with no estimate and no price. -
/testing:run-e2e map baseline=<findings.json>runs the compare script and opens its report with both pass builds, then NEW, PRE-EXISTING and GONE findings by severity with one summary line per group, ordered by themap_report_ordersetting. -
/testing:run-e2ehas settingsfile_findings(nonedefault,new) andfile_findings_floor(highdefault); undernewit states the count, then hands each NEW finding at or above the floor to/bugs:writewith its id in the report, and never closes a GONE one. -
/testing:run-e2eeval cases on a two-version fixture app assert that the second pass, given the first pass'sfindings.json, reports exactly the planted NEW, PRE-EXISTING and GONE findings. - The
autonomyroutines catalog has awhole-map-live-verificationrow (AGT,R + WI,repo,C1,join: proven recurring manual pattern) and a class parameter saying the pass is/testing:run-e2e map, starts by hand, takes any cadence the consumer picks, and gets no leaf until a consumer records a manual run. - This repository's testing conventions set
map_unit: part,map_entry_points: firstandfile_findings: none.
Moved to #16564 (host work)
The manual-dispatch workflow, its inputs, its artifact upload and baseline download, and the driven model it uses. Its artifacts are kept for the repository's 90-day setting, which adds no charge (owner, 2026-11-16).
Owner step
The owner dispatches two measured passes on two different pass builds, the first capped at about 40 skills across at least 8 plugins, and reviews each pass's counts, measured tokens and wall time, and the second pass's NEW, PRE-EXISTING and GONE lists.
At that review the owner decides whether this repository sets file_findings: new, whether to run a whole-map pass, and any cadence.
Not part of this item
- Any schedule or cadence before the owner's review of the measured passes.
- Flood-style fuzzing (random or malformed input against a target) and UI property testing (slide 28): both belong to the property guidance in
/testing:write. - A worker split for the
mapscope, and a model-judged merge of duplicates across parts. - Changes to
/verification:confirm,/app-control:driveor/app-control:map refresh. - Widening plugin features'
sources:for the per-pull-request capture.
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Priority
Activity
github-actions commentedon Nov 12, 2026
Size: Large
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Understood: add a
mapscope to/testing:run-e2ethat drives every feature-map entry, plus a deterministic findings compare script (NEW / PRE-EXISTING / GONE / not checked), afindings.jsonschema, settings, evals on a two-version fixture, and anautonomycatalog row. The pass runs on demand only.A decision is recorded on this issue, so the earlier questions are settled:
#15649 (comment)
Cost decided: artifact retention at the existing 90-day setting; no charge added, covered by the owner decision of 2026-11-16.
This decision settles the open questions. The workflow part is tracked in #16564. The size is Large because the done-when list is long and may need splitting. I found no duplicate or merged fix.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 12, 2026
The latest triage verdict holds nothing: the issue is at
status: ready.melodic-automation-lanes commentedon Nov 12, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: A new testing:explore skill that generates actions, drives a fixture app through /app-control:drive, judges results against stated invariants, and files findings via /bugs:write, with an autonomy routine as the release trigger.
Related work
Open questions
RECOMMENDED: A small web app under plugins/testing/tests/fixtures/, beside the skill it proves, with a few seeded bugs so the run has known findings. Basis: The issue lists this as open; plugins/testing has no fixtures directory today (only LICENSE, CHANGELOG.md, README.md, prerequisites.json).
RECOMMENDED: A per-app file of named assertions, each checkable against the evidence /app-control:drive records; lock it in /planning:interview. Basis: plugins/app-control/skills/drive/SKILL.md:20 says drive carries no verdict and the caller judges; the issue says the interview locks the invariant format.
RECOMMENDED: Add a separate explore-on-release routine row, since it is an agent session that judges, unlike the deterministic fuzzer row; settle trigger and autonomy level in the interview. Basis: plugins/autonomy/reference/routines.md:237
RECOMMENDED: Yes, track it as its own issue; the body calls it a separate decision item. Basis: issue body, Open section
RECOMMENDED: No record of it; run it, then post the answers as a Decision recorded comment. Basis: issue body, Before build section; no maintainer comments in context
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.[-]Add a live-app exploring skill to the testing plugin, proven on a fixture app[/-][+]Run a whole-map live verification pass on each release and report what is new since the last release[/+]kyle-sexton commentedon Nov 14, 2026
Interview status, 2026-11-13: deferred until #15913 lands
The interview on this item started and is paused. Nothing here changes the Decision in the body.
Owner conditions for when it resumes
A hand trial, and what it does and does not show
Five of this repository's skills were driven once each through the control skill's print-mode capture, locally, on Claude Code 2.1.325 and Opus 5.5.
/coupling:reduce dry-run/user-experience:structure/retro-audio:sfx/adhd:clarify/actionlint:setup checkA proposed skip was checked and overturned
Declining this item was proposed after the trial. An independent check overturned the skip:
sources:line, which lists onlyplugins/<plugin>/skills/*/SKILL.md. A change to a skill's scripts or reference files, a shared library, or the Claude Code version drives nothing./app-control:map refreshalready drives every feature on demand and routes product regressions to/bugs:write; nothing schedules it.Questions still to settle when it resumes
The unit of one entry, reporting defects with no stated expectation, which skill carries the scope and the comparison, the finding id and matching rule, where the last run's findings live, what gets filed, report order, duplicate merging, build identity, the fixture, and which credential a pass spends.
10 remaining items
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane yet: sub-issue #16567 is open; sub-issue #16568 is open; sub-issue #16569 is open; sub-issue #16571 is open.
kyle-sexton commentedon Nov 16, 2026
Decision recorded.
Owner, 2026-11-16, in session ("Go w/ recommendations"), posted by an agent session:
gh api repos/melodic-software/claude-code-plugins/actions/permissions/artifact-and-log-retention:days: 90,maximum_allowed_days: 90). No setting changes. A pass dispatched more than 90 days after the previous one has no baseline and reports every finding NEW.melodic-automation-lanes commentedon Nov 16, 2026
Split at intake (owner Q93). This issue is Size Large, so its done-when is split into 6 sub-issues, each holding its task-list lines as written.
Automated split by the lanes App. Admission waits on this issue while a sub-issue is open; work-unblock closes it as completed once every sub-issue is closed as completed by a merge or a listed human and its done-when is covered.