Skip to content

Should verification artifacts be shown in the pull request where a reviewer sees them? #15658

Closed

Description

@kyle-sexton

Question

Should verification artifacts (a capture summary, screenshots, video) be linked in the pull request where a human reviewer sees them?

From the talk

Chapter 7, slide 24: review verification artifacts, not diffs.

Today and the gap

Our review lanes' judge reads artifacts, but they are not shown in the pull request, and video is on request only (/playwright:demo-video).

Recommendation

Investigate (menu priority P2), extending /review:explain-change and /playwright:demo-video. Basis: the gap above.

Alternatives

Keep artifacts for the agent judge only; attach video by default.

Done would add

An artifacts section in the pull request body or explainer. Dogfood: how it applies to plugin pull requests here, which change prose and scripts, is open on the map's fog list.

Activity

  1. added
    wayfind: interviewWayfind decision item: contract/requirements call the user must make; human in the loop.
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 12, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Knowable
    Flags: intent
    Blocked by: none

    I understand this as a question on whether verification artifacts (capture summary, screenshots, video) should appear in the pull request for human reviewers, by extending /review:explain-change and /playwright:demo-video. The issue lists two alternatives, and it says how this applies to plugin PRs is still open. No duplicate or merged fix found.

    1. Should artifacts be linked in the PR body or explainer by default?
      RECOMMENDED: Yes, as a short artifacts section. Basis: reviewers cannot see what the judge reads today.
    2. Should video be attached by default or stay on request?
      RECOMMENDED: Stay on request, with screenshots and the summary by default. Basis: video is costly to produce and rarely applies to prose and script changes.
    3. What should plugin PRs here, which change prose and scripts, show?
      RECOMMENDED: The capture summary only, with the section omitted when there are no artifacts. Basis: screenshots and video do not apply to such changes.

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated triage. A maintainer reviews it.

  4. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 12, 2026
  5. melodic-automation-lanes commented on Nov 12, 2026

    @melodic-automation-lanes
    Contributor

    The triage verdict left this issue at needs-owner-decision:

    • flagged: intent
    • held by needs-owner-decision
    • held by needs-human
    • no lanes App verdict comment shows whether an earlier verdict was flagged

    Reason: intent

  6. kyle-sexton commented on Nov 13, 2026

    @kyle-sexton
    MemberAuthor

    Resolved: already built, with one join added. No new skill and no second place that shows evidence.

    From the talk. Slide 24 and the transcript at 1:11:14: "our code review process has now really shrunk into looking at verification artifacts instead. So things like videos, screenshots, diagrams, formal verification."

    What already exists.

    • /source-control:pull-request evidence posts one evidence comment on the pull request, or updates it in place for a new head: the text summary, publishable captures in collapsed sections, and the CI capture artifact link (plugins/source-control/skills/pull-request/reference/evidence.md:5-9).
    • It posts only after the verifier passed for the pull request's current head: "the evidence comment follows a verifier pass, never stands in for one" (evidence.md:49-51). So an artifact from a run that failed is never posted, and nothing is produced twice for one head.
    • Unattended, the merge lane posts the same comment after a passing verdict and before merge (evidence.md:11-16).
    • /playwright:demo-video hands its video to the same step (plugins/playwright/skills/demo-video/SKILL.md:110).

    The gap, and the decision (owner, 2026-11-12: "Yes, let's go with recommendations"). In an attended local session nothing runs the evidence step unless someone asks. The ready step will run it once the head is confirmed and there is evidence to post: Post the evidence comment from the ready step once the head is confirmed. That one comment is the home for every kind of artifact: captures, video, and later a formal-verification chart.

    Declined. Extending /review:explain-change to show artifacts, and attaching video by default: both would add a second place that shows evidence.

    Still open on the map's fog list: what evidence means for this repository's own pull requests, which change prose and scripts and have no app.

  7. kyle-sexton commented on Nov 13, 2026

    @kyle-sexton
    MemberAuthor

    Resolved; see the resolution comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.needs-owner-decisionAwaiting a human or maintainer judgment call.needs-triageNot yet classified. Floor until a type and one priority tier are set.wayfind: interviewWayfind decision item: contract/requirements call the user must make; human in the loop.

    Type

    Fields

    Work status

    needs-human

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions