Skip to content

Run a whole-map live verification pass on demand and report what is new since the last pass #15649

Open
2 of 6 issues completed

Description

@kyle-sexton

Decision (owner, 2026-11-12, revised the same day; conditions 2026-11-13; answers 2026-11-15)

Source: the talk "All Code Is Slop (Except Mine)" (Lauren Tan, AI Engineer Code Summit, 2026-11-11), slides 42-43 and the transcript at 1:17:48: an agent runs the whole build "like a real user", and the report marks each finding new or already present ("NEW: 3 bugs ... PRE-EXISTING: 47 bugs", "Each reply keeps its finding id").

No new skill. The pass is a composite of skills that exist, plus three missing parts:

  • A whole-map scope on the existing live verification. Today a live verification run covers only the features a change touches (/testing:run-e2e). The pass runs the same judging over every entry in the feature map, with no change to diff against.
  • A routine row in the autonomy plugin, with no schedule.
  • The comparison with the previous pass in that pass's report: each finding marked new, pre-existing or gone, keyed by feature-map entry, with an id that stays the same across runs. A first run reports everything as new.

Findings go through /bugs:write; fixes go through the existing loop lanes. /app-control:drive still only drives and records.

The owner's conditions of 2026-11-13 (comment "Interview status, 2026-11-13: deferred until #15913 lands") replace "on each release": the pass starts by hand only, with no schedule until one measured pass has been reviewed; the owner controls when and where it triggers, and it is easy to turn on and off and to run on demand; any later cadence is weekly, every two weeks or monthly, never per release; it proves its value with measurements against tokens, time and plan usage; driven sessions run on Opus 5.5, as a setting.

Answers (interview, 2026-11-15; restatement confirmed by the owner in session, "go w/ recommendations")

  • Trigger. On demand only: a workflow whose only trigger is manual dispatch by a listed human, turned on and off with gh workflow enable and disable, stopped by the lanes kill switch. The comparison is with the previous pass, not with a release.
  • Credential and first pass (owner). A pass spends the pr-verify API key inside the pull-request capture's sandbox and key proxy, under the existing credit gate and per-job token cap, because only that sandbox keeps a driven skill's shell. The first measured pass is capped at about 40 skills across at least 8 plugins; a whole-map pass is decided later with its numbers.
  • Home. /testing:run-e2e gains a map scope: every feature of the control skill, no base ref, entries driven one after another, its control-skill refresh step skipped. /verification:confirm is unchanged.
  • Unit. Settings map_unit (feature default, or part) and map_entry_points (all default, or first); this repository sets part and first, so one entry is one skill through its print-mode slash command.
  • Defects the map states nothing about. Reported, labeled unstated, from a closed list of kinds (error, crash, refusal, no output, garbled output). This repository's stated results only say a skill's output appears, so this is where most findings come from.
  • Finding id. A short hash of feature, part, failure kind and where (the drive step number, proof, or run); the entry point is not in it, so one failure seen through several entry points is one finding. Matched across passes by exact id. A blocked or not-driven entry is not checked, never a finding, and never makes a finding GONE.
  • Baseline. The skill reads baseline=<findings.json> and writes its own findings.json; storage is the caller's. This repository keeps it as a workflow artifact read by the next dispatch. Nothing is committed.
  • Report. NEW, then PRE-EXISTING, then GONE, each by severity (critical, high, medium, low), a summary line per group, both pass builds in the header, measured tokens and wall time per entry. Setting map_report_order.
  • Filing. Settings file_findings (none default, or new) and file_findings_floor (high default). The first pass files nothing. Whether this repository sets new is deferred by the owner to the review of the second measured pass.
  • Duplicates. Deterministic only: one finding per id; no model merge.
  • Routine. One autonomy catalog row, whole-map-live-verification, status join: proven recurring manual pattern, so no schedule and no leaf until a consumer records a manual run.

Done when

  • The testing plugin's findings compare script takes two findings.json files and labels each finding id NEW, PRE-EXISTING, GONE or not checked, with no model call.
  • The testing plugin's findings compare script never labels an id GONE when its entry was not driven in the newer pass, and labels every finding NEW when no baseline is given.
  • The testing plugin's findings compare script has a test file covering each label and the not-checked rule, run by CI.
  • The testing plugin's findings.json schema holds per finding the id, feature, part, kind, where, entry points, severity, stated or unstated label and evidence paths, and per pass the build and each entry's outcome, wall time and tokens.
  • /testing:run-e2e map drives every feature of the control skill without calling drive's affected action and without a base ref, and accepts features= to narrow it and max_entries= to cap it, reporting the rest not checked (cap).
  • /testing:run-e2e has settings map_unit (feature default, part) and map_entry_points (all default, first) in its plugin options and repository conventions, the repository value winning, and under part drives each <feature>/<part> drive resolves.
  • /testing:run-e2e map records a failed stated result or proof step as a finding, records observed breakage as an unstated finding of kind error, crash, refusal, no-output or garbled-output, and records a blocked entry as not checked with its reason.
  • /testing:run-e2e map gives each finding an id hashed from feature, part, kind and where, without the entry point, listing every entry point the finding showed at.
  • /testing:run-e2e map skips its control-skill refresh step, edits no tracked file, and writes findings.json and e2e-summary.json at the top of its run directory.
  • /testing:run-e2e map reports each entry's measured wall time and tokens, or null with the reason, and the pass totals, with no estimate and no price.
  • /testing:run-e2e map baseline=<findings.json> runs the compare script and opens its report with both pass builds, then NEW, PRE-EXISTING and GONE findings by severity with one summary line per group, ordered by the map_report_order setting.
  • /testing:run-e2e has settings file_findings (none default, new) and file_findings_floor (high default); under new it states the count, then hands each NEW finding at or above the floor to /bugs:write with its id in the report, and never closes a GONE one.
  • /testing:run-e2e eval cases on a two-version fixture app assert that the second pass, given the first pass's findings.json, reports exactly the planted NEW, PRE-EXISTING and GONE findings.
  • The autonomy routines catalog has a whole-map-live-verification row (AGT, R + WI, repo, C1, join: proven recurring manual pattern) and a class parameter saying the pass is /testing:run-e2e map, starts by hand, takes any cadence the consumer picks, and gets no leaf until a consumer records a manual run.
  • This repository's testing conventions set map_unit: part, map_entry_points: first and file_findings: none.

Moved to #16564 (host work)

The manual-dispatch workflow, its inputs, its artifact upload and baseline download, and the driven model it uses. Its artifacts are kept for the repository's 90-day setting, which adds no charge (owner, 2026-11-16).

Owner step

The owner dispatches two measured passes on two different pass builds, the first capped at about 40 skills across at least 8 plugins, and reviews each pass's counts, measured tokens and wall time, and the second pass's NEW, PRE-EXISTING and GONE lists.
At that review the owner decides whether this repository sets file_findings: new, whether to run a whole-map pass, and any cadence.

Not part of this item

  • Any schedule or cadence before the owner's review of the measured passes.
  • Flood-style fuzzing (random or malformed input against a target) and UI property testing (slide 28): both belong to the property guidance in /testing:write.
  • A worker split for the map scope, and a model-judged merge of duplicates across parts.
  • Changes to /verification:confirm, /app-control:drive or /app-control:map refresh.
  • Widening plugin features' sources: for the per-pull-request capture.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Large
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: add a map scope to /testing:run-e2e that drives every feature-map entry, plus a deterministic findings compare script (NEW / PRE-EXISTING / GONE / not checked), a findings.json schema, settings, evals on a two-version fixture, and an autonomy catalog row. The pass runs on demand only.

    A decision is recorded on this issue, so the earlier questions are settled:

    Decision recorded.

    Owner, 2026-11-16, in session ("Go w/ recommendations"), posted by an agent session:

    • No charge is added. The findings artifact is kept for this repository's existing setting, 90 days, which is also the most a public repository allows (gh api repos/melodic-software/claude-code-plugins/actions/permissions/artifact-and-log-retention: days: 90, maximum_allowed_days: 90). No setting changes. A pass dispatched more than 90 days after the previous one has no baseline and reports every finding NEW.
    • The manual-dispatch workflow (its dispatch, inputs, artifact upload and baseline download, and driven model) moved to Add the manual-dispatch workflow for the whole-map live verification pass #16564 for the host queue. This issue keeps the skill, compare script, settings, eval and catalog items, all of which a lane run can do.

    #15649 (comment)

    Cost decided: artifact retention at the existing 90-day setting; no charge added, covered by the owner decision of 2026-11-16.

    This decision settles the open questions. The workflow part is tracked in #16564. The size is Large because the done-when list is long and may need splitting. I found no duplicate or merged fix.

    Automated triage. A maintainer reviews it.

  4. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: the issue is at status: ready.

  5. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: A new testing:explore skill that generates actions, drives a fixture app through /app-control:drive, judges results against stated invariants, and files findings via /bugs:write, with an autonomy routine as the release trigger.

    Related work

    Open questions

    1. What stack does the fixture app use, and where does it live?
      RECOMMENDED: A small web app under plugins/testing/tests/fixtures/, beside the skill it proves, with a few seeded bugs so the run has known findings. Basis: The issue lists this as open; plugins/testing has no fixtures directory today (only LICENSE, CHANGELOG.md, README.md, prerequisites.json).
    2. What format do invariants take?
      RECOMMENDED: A per-app file of named assertions, each checkable against the evidence /app-control:drive records; lock it in /planning:interview. Basis: plugins/app-control/skills/drive/SKILL.md:20 says drive carries no verdict and the caller judges; the issue says the interview locks the invariant format.
    3. What are the autonomy routine's trigger and contract, given gui-crash-fuzzing is catalogued not-a-routine?
      RECOMMENDED: Add a separate explore-on-release routine row, since it is an agent session that judges, unlike the deterministic fuzzer row; settle trigger and autonomy level in the interview. Basis: plugins/autonomy/reference/routines.md:237
    4. Is the per-release baseline (stable finding ids, new vs pre-existing) out of scope here?
      RECOMMENDED: Yes, track it as its own issue; the body calls it a separate decision item. Basis: issue body, Open section
    5. Has /planning:interview been run to lock the contract, which the issue requires before build?
      RECOMMENDED: No record of it; run it, then post the answers as a Decision recorded comment. Basis: issue body, Before build section; no maintainer comments in context

    Sources read

    • plugins/app-control/skills/drive/SKILL.md:20: drive records evidence, carries no verdict, the caller judges
    • plugins/autonomy/reference/routines.md:237: gui-crash-fuzzing is DET, not-a-routine, L3 required for GUI actuation
    • plugins/autonomy/reference/routines.md:326: the L3 floor for gui-crash-fuzzing comes from the mapping rule
    • plugins/testing: contains only LICENSE, CHANGELOG.md, README.md, prerequisites.json in this checkout; no skills or fixtures found by my glob

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  6. changed the title [-]Add a live-app exploring skill to the testing plugin, proven on a fixture app[/-] [+]Run a whole-map live verification pass on each release and report what is new since the last release[/+] on Nov 13, 2026
  7. kyle-sexton commented on Nov 14, 2026

    @kyle-sexton
    MemberAuthor

    Interview status, 2026-11-13: deferred until #15913 lands

    The interview on this item started and is paused. Nothing here changes the Decision in the body.

    Owner conditions for when it resumes

    • The pass starts by hand only. No schedule until one measured pass has been reviewed.
    • The owner controls when and where it triggers, and it is easy to turn on and off and to run on demand.
    • Any later cadence is weekly, every two weeks or monthly, never per release: releases here merge about 31 times a day.
    • It has to prove its value with measurements and baselines, judged against tokens, time and plan usage.
    • Driven sessions run on Opus 5.5, as a setting.

    A hand trial, and what it does and does not show

    Five of this repository's skills were driven once each through the control skill's print-mode capture, locally, on Claude Code 2.1.325 and Opus 5.5.

    Skill Wall time Result
    /coupling:reduce dry-run 10 min, stopped by the driver's limit Worked; its git-history step could not run and its file write was declined
    /user-experience:structure 95 s Worked fully
    /retro-audio:sfx 24 s Could not render: no shell
    /adhd:clarify 17 s Degraded: its resolver step needs a shell
    /actionlint:setup check 1 s Failed at load: its first shell line was denied
    • A local drive launches with the restricted flag, which removes the shell, so four of five skills could not do their work. A local hand run therefore says little about skill behaviour.
    • Run size varied about 40 times between the lightest and the heaviest skill.
    • The trial tested this repository's skills as the app. It did not test the design decided above (a whole-map scope for any app, proven on a two-version fixture), so it is not evidence for or against that design.

    A proposed skip was checked and overturned

    Declining this item was proposed after the trial. An independent check overturned the skip:

    • No scheduled workflow runs the eval suites, so they do not catch a skill that breaks without being edited.
    • The per-pull-request capture selects features by each plugin's sources: line, which lists only plugins/<plugin>/skills/*/SKILL.md. A change to a skill's scripts or reference files, a shared library, or the Claude Code version drives nothing.
    • /app-control:map refresh already drives every feature on demand and routes product regressions to /bugs:write; nothing schedules it.

    Questions still to settle when it resumes

    The unit of one entry, reporting defects with no stated expectation, which skill carries the scope and the comparison, the finding id and matching rule, where the last run's findings live, what gets filed, report order, duplicate merging, build identity, the fixture, and which credential a pass spends.

  8. 10 remaining items

  9. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: sub-issue #16567 is open; sub-issue #16568 is open; sub-issue #16569 is open; sub-issue #16571 is open.

  10. kyle-sexton commented on Nov 16, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Owner, 2026-11-16, in session ("Go w/ recommendations"), posted by an agent session:

    • No charge is added. The findings artifact is kept for this repository's existing setting, 90 days, which is also the most a public repository allows (gh api repos/melodic-software/claude-code-plugins/actions/permissions/artifact-and-log-retention: days: 90, maximum_allowed_days: 90). No setting changes. A pass dispatched more than 90 days after the previous one has no baseline and reports every finding NEW.
    • The manual-dispatch workflow (its dispatch, inputs, artifact upload and baseline download, and driven model) moved to Add the manual-dispatch workflow for the whole-map live verification pass #16564 for the host queue. This issue keeps the skill, compare script, settings, eval and catalog items, all of which a lane run can do.
  11. melodic-automation-lanes commented on Nov 16, 2026

    @melodic-automation-lanes
    Contributor

    Split at intake (owner Q93). This issue is Size Large, so its done-when is split into 6 sub-issues, each holding its task-list lines as written.

    Automated split by the lanes App. Admission waits on this issue while a sub-issue is open; work-unblock closes it as completed once every sub-issue is closed as completed by a merge or a listed human and its done-when is covered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    ready

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions