Skip to content

Should the release exploring run keep a per-release baseline and report only new findings? #15655

Closed

Description

@kyle-sexton

Question

Should the release exploring run keep a per-release baseline: diff against the last release, split new from pre-existing findings, and keep stable finding ids?

From the talk

Chapter 10, slides 42-43: an outer-loop run explores each release build like a user and compares it with the last release.

Today and the gap

Absent. A GUI crash-fuzzing run is catalogued as not-a-routine (plugins/autonomy/reference/routines.md:237). The trigger (an autonomy routine) and findings route (/bugs:write) were decided with the live-app exploring skill, Add a live-app exploring skill to the testing plugin, proven on a fixture app; the baseline was not.

Recommendation

Proof of concept (menu priority P1): the exploring skill keeps a baseline per release and reports only new findings as new, with stable ids. Basis: without it, every run refiles known findings.

Alternatives

No baseline, deduplicate at /bugs:write filing time; or a release mode on /bugs:scan holding the baseline.

Done would add

A baseline record and diff step in the exploring skill. Dogfood: two runs against this repository's fixture app across two fixture releases, the second reporting only what changed.

Activity

  1. added
    wayfind: interviewWayfind decision item: contract/requirements call the user must make; human in the loop.
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: #15649

    I understood this as a request that the release exploring run keep a per-release baseline, diff each run against the last release, report only new findings as new, and give findings stable ids. The issue recommends putting this in the exploring skill. It names two alternatives: deduplicating at /bugs:write filing time, or a release mode on /bugs:scan. The done-when is two dogfood runs against the fixture app across two fixture releases, where the second run reports only what changed.

    The exploring skill is tracked in #15649 and is still open, so this work depends on it. I found no merged pull request that already covers this.

    Automated triage. A maintainer reviews it.

  4. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    The triage verdict left this issue at needs-owner-decision:

    • held by needs-human
    • no lanes App verdict comment shows whether an earlier verdict was flagged

    Reason: intent

  5. kyle-sexton commented on Nov 13, 2026

    @kyle-sexton
    MemberAuthor

    Resolved: yes, and it is not a separate skill or item. The comparison with the last release is part of the release pass's own report.

    From the talk. Slides 42-43 and the transcript at 1:17:48: on each release an agent runs the whole build "like a real user". The slide-43 report compares every finding with the previous release: "NEW: 3 bugs ... PRE-EXISTING: 47 bugs", and "Each reply keeps its finding id from the full report".

    Decision (owner, 2026-11-12: "Let's go with recommendations so far").

    • The release pass is a whole-map scope on the existing live verification, started by a release routine. No new exploring skill.
    • That pass's report marks each finding new, pre-existing or gone against the last release's report, keyed by feature-map entry, with an id that stays the same across runs. A first run reports everything as new.
    • Finding bugs in a build and matching them against the last release change together, so they live in one place.

    Where it is built. Run a whole-map live verification pass on each release and report what is new since the last release, whose interview locks the finding id, the matching rule and where the last report is kept.

    Declined alternatives. A delta companion skill (a second skill coupled to the first one's finding format); a release mode on /bugs:scan (it hunts by reading code and runs no scanned code unattended); removing duplicates only when /bugs:write files a finding (loses the pre-existing count).

    Basis: the talk passages above; her guide (pstack/docs/guide/06-verify-and-ship.md:73 in cursor/plugins), which describes the pass as a swarm of verifiers split by feature-map entry; plugins/verification/skills/confirm/SKILL.md:176-181 and plugins/app-control/skills/map/SKILL.md:206-210, which already hold the fan-out and the every-feature pass.

  6. kyle-sexton commented on Nov 13, 2026

    @kyle-sexton
    MemberAuthor

    Resolved; see the resolution comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.wayfind: interviewWayfind decision item: contract/requirements call the user must make; human in the loop.

    Type

    Fields

    Work status

    needs-human

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions