Skip to content

feat(source-control): show small-change evidence inline and add an optional record block - #16616

Merged
kyle-sexton merged 7 commits into
mainfrom
feat/pr-evidence-size-rule
Nov 16, 2026
Merged

kyle-sexton merged 7 commits into
mainfrom
feat/pr-evidence-size-rule

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

No related issue: an owner-decided change to the pull-request skill's evidence comment layout.

Summary

The /source-control:pull-request evidence comment now leads with the run's text result and sizes its media to the change: a small change shows its Before/After captures inline as a table, and a larger change keeps one collapsed section per acceptance criterion. A repository can switch on a record block (who ran the verification, when, and the issues found with their resolution) for teams that keep such a record. The verifier-verdict and leak-scan posting gates are unchanged.

Fix

  • skills/pull-request/reference/evidence.md, templates/evidence-comment.md: one template, two layouts. A change is small when it has at most two sections (acceptance criteria, else entry points) and at most evidence_inline_max_captures captures, Before and After counted together. Small renders a table with one column per interaction, Before and After rows, a caption under each image, and one line naming the capturing tool and what each row's state means. Videos go in their own paragraph under the table, since a video plays only alone in a paragraph. Images stay markdown references, never <img> tags, so gh --attach rewrites them.
  • Default threshold 4: one Before and one After for each of two interactions, which is the two-criterion bound. 0 keeps every change in collapsed sections.
  • New inputs: --before <file> (the base run's evidence file, held to its own publishable list and the same sidecar check), --verified-by, --issues.
  • evidence_record_fields (default false) adds the record block. A missing value renders not recorded and never stops the post. The fields follow the record elements in FDA's Computer Software Assurance guidance, section (6), as an analogy for regulated teams, not a legal requirement.
  • scripts/read-evidence-layout.sh (+ tests) reads both keys from docs/conventions/source-control.yaml at the pull request's base, honored only when the base is the default branch, like review_venue. No userConfig option: both are repository policy. Schema and reference/config-resolution.md document the keys; SKILL.md's action row and create.md's Visual evidence paragraph point at the new layout.
  • evals/fixtures/gh-forge/gh: pr comment --help lists the flags, --attach among them.
  • Three eval cases: small change gives a table and no record by default; three criteria give collapsed sections; the record switch gives the record block. Each also checks that text comes before any media.

Verification

  • bash plugins/source-control/skills/pull-request/scripts/read-evidence-layout.test.sh: all pass.
  • bash scripts/check-convention-yaml.sh, bash scripts/check-convention-yaml.test.sh, read-pre-ready.test.sh: pass.
  • markdownlint-cli2 on the changed markdown: clean. shellcheck on the new scripts: clean.
  • validate-cases.py plugins/source-control/evals: the new cases load and every deterministic grader passes and rejects its samples as written.
  • claude plugin eval (Claude Code 2.1.325, with-plugin arm only, 2 runs per case) on the three new cases: 6/6 runs scored 1.00, every grader passing. The first run caught a fixture gap: the eval forge's gh did not answer pr comment --help, so the model took step 7's no-attach fallback and rendered no table. The forge now lists the pr comment flags.

Left out: the CI pr-merge lane's evidence comment (babysit_evidence_comment.py) attaches no file and already leads with text; it does not render the table or the record block.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

kyle-sexton and others added 3 commits November 16, 2026 14:23
…tional record block

The pull-request evidence comment now leads with the run's text result,
lays out a small change (at most two acceptance criteria and at most
evidence_inline_max_captures captures, default 4) as a Before/After
table, and keeps collapsed sections for a larger change. A team can turn
on evidence_record_fields to add who verified, when, and the issues
found with their resolution. Both keys are read at the pull request's
base by read-evidence-layout.sh. The verdict and leak-scan gates are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A change with no capture keeps collapsed sections, the capture line names
the PR's base branch for the Before row, the leak scan reads the
--before file's hub too, each publishable list resolves against its own
file's directory, a video reference takes the capture path check, and
every table cell escapes pipes and line breaks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:41:30.073339Z 1e95a71 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1e95a711aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…nt at the video rule

With --before, each capture in a collapsed section carries a
Before (base) or After (head) label line and alt prefix, since a base
and a head capture can share a file name. The video paragraph keeps its
instruction and the pointer to GitHub's embedding section without
restating the upstream behavior. The large-change eval now uses shared
file names and grades the run labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 1e95a711aac59baae7d7877db95703a5176fcaf0; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed acb793eeda7260f1724fa824f7cfda6aaa6c337f: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, acb793e since 30ffb8b; blocking only)

This review covers one file changed since 30ffb8b3: plugins/source-control/skills/pull-request/reference/evidence.md. I read its whole change in the incremental diff. This lane already reviewed an earlier head, so the review was blocking only: it reports CRITICAL and IMPORTANT findings only.

No security issues in this push.

What changed. At evidence.md L12-16, a hyperlink to the FDA guidance page was swapped for the guidance's title in plain text. The edit is prose only:

  • No instruction was removed, narrowed or softened.
  • The verifier-verdict gate, the --attach → publishable gate and the leak scan are untouched.
  • No new URL or outside source was added.

Instruction surface. I did not consult the protected-class register because the edit removes no rule.

Outside this push's scope. The eval-only gh-forge-path/.claude-plugin/plugin.json has not changed since 30ffb8b3, and it is not in the incremental scope. The manifest flags this lane posted on it earlier (hooks, name, version) are not raised again here. Their threads are resolved with owner replies, and their lines have not changed since resolution.

Embedded text. I found no embedded instructions in the PR context or the settled threads.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed acb793eeda7260f1724fa824f7cfda6aaa6c337f: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Incremental code review: acb793e (since 30ffb8b), blocking findings only

This lane already reviewed earlier heads, so this pass reports blocking findings only. I read the whole incremental change: one hunk in 1 of 1 listed files, plugins/source-control/skills/pull-request/reference/evidence.md.

What changed

  • The FDA reference in the opening "Text and logs are the record" paragraph no longer links to fda.gov. It now names the guidance by its full title, "Computer Software Assurance for Production and Quality Management System Software", next to the same section (6) "Establishing the Appropriate Record" (evidence.md#L15-L16).
  • The meaning stays the same: the guidance is still framed as an analogy, not a requirement the step meets.

Findings

None blocking.

Other notes

  • Not counted as a finding. This lane is only reporting blocking findings now, so this one is not counted. The {{#record}} bullet still says "the FDA guidance's section (6) linked above" (evidence.md#L106). Nothing above it is a link any more, so "linked above" could become "cited above" in a later edit.
  • Settled threads. None touch the changed lines. The three eval-only manifest threads are marked lines changed since resolution: no, so they stay settled. No thread contains embedded instructions.
  • Prior findings. The earlier finding about captures no column claims was resolved on an earlier head. This change does not touch it.
  • Manifests. No plugin.json or marketplace.json changed in this increment.
  • Spend. No change in this increment affects lane spend.

Verifier dropped 0 of 0 candidates.

@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Held for the owner's review of the decisions behind this change. The hold comes off when the owner confirms them.

…tions

The Before/After table now applies only when every capture maps to an
entry point through a drive report.json step. An e2e-summary.json input,
or any capture no step lists, keeps the collapsed layout with its Other
captures section, so every capture has a place. A new eval case covers
an e2e-summary.json input. The FDA citation points at the guidance's
HTML page, which automated link checks can load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 16, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Hold lifted: the owner reviewed and confirmed the decisions behind this change (2026-11-16).

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-security-review: 3 inline finding(s) on 30ffb8b3a16d06cbfb7ef244e53e0c97207bb6cb; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

… dead link

The fda.gov guidance URL now redirects to a 404, which fails the online
link check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 09cdae6 Nov 16, 2026
71 checks passed
@kyle-sexton
kyle-sexton deleted the feat/pr-evidence-size-rule branch November 16, 2026 21:52
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant