Skip to content
Merged
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
bump: minor
---

### Added

- **The evidence comment shows a small change inline.** `/source-control:pull-request evidence` lays out a change with at most two acceptance criteria and at most `evidence_inline_max_captures` captures (default 4), each mapped to an entry point by a drive `report.json`, as a Before/After table, one column per interaction, a caption under each image, and one line naming the capturing tool and what each row's state is. Any other change, an `e2e-summary.json` input included, keeps one collapsed section per acceptance criterion and `Other captures`. `--before <file>` takes the base commit's evidence file, governed by its own `publishable` list. A video sits in its own paragraph under the table.
- **Optional record block.** With `evidence_record_fields: true` (off by default) the comment carries who ran the verification (`--verified-by`), when (the evidence file's write time) and the issues found with their resolution (`--issues`, else the run's failed entries).
- `scripts/read-evidence-layout.sh` reads both keys from `docs/conventions/source-control.yaml` at the pull request's base, honored only on the default branch, with tests.

### Changed

- The evidence comment leads with the run's text result, one line per criterion or entry point; captures support it. The verifier-verdict and leak-scan gates are unchanged.
83 changes: 83 additions & 0 deletions plugins/source-control/evals/fixtures/evidence-pr.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# Seeds an evidence-comment case: a repository whose feature branch is one commit ahead of main, an
# offline forge (gh-forge) serving open PR 412 with an `## Acceptance criteria` body, a CONFIRMED
# verifier verdict recorded for the head, and two /app-control:drive report.json files, one from
# the head (evidence/head) and one from the base (evidence/base), each listing one PNG per
# interaction in `publishable` under the same file name on both sides. Run it from a case's
# scaffold.sh in the workspace:
# bash evidence-pr.sh <interactions> [--record-fields] [--e2e]
# <interactions> is the number of acceptance criteria and of driven entry points (one capture
# each per run). --record-fields commits `evidence_record_fields: true` to main's team config.
# --e2e writes each side's /testing:run-e2e e2e-summary.json in place of report.json; its flat
# `publishable` list maps no capture to an entry point.
set -euo pipefail

count="${1:?evidence-pr.sh needs an interaction count}"
record=0 e2e=0
for flag in "${@:2}"; do
case "$flag" in
--record-fields) record=1 ;;
--e2e) e2e=1 ;;
*) echo "evidence-pr.sh: unknown argument $flag" >&2; exit 2 ;;
esac
done
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
plugin="$(cd "$here/../.." && pwd)"
png='iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=='
names=(cart checkout receipt orders profile)

git init -q -b main
git config user.name eval
git config user.email eval@example.invalid
mkdir -p docs/conventions src
printf 'export const total = (items) => items.length;\n' >src/cart.ts
printf 'pr_body_required_sections:\n - Summary\n - Verification\n' >docs/conventions/source-control.yaml
((record)) && printf 'evidence_record_fields: true\n' >>docs/conventions/source-control.yaml
printf 'evidence/\n' >.gitignore
git add docs src .gitignore
git commit -q -m "chore: seed"

git checkout -q -b feat/cart-totals
printf 'export const total = (items) => items.reduce((s, i) => s + i.price, 0);\n' >src/cart.ts
git add src/cart.ts
git commit -q -m "feat: show the cart total in currency"
head_sha="$(git rev-parse HEAD)"

bash "$here/gh-forge/install.sh" 412 feat/cart-totals main "feat: show the cart total in currency" --open
forge="$(git rev-parse --path-format=absolute --git-common-dir)/eval-forge"

criteria=""
for ((i = 0; i < count; i++)); do
criteria+="- The ${names[i]} page shows the total in currency"$'\n'
done
jq -n --arg body "## Summary"$'\n\n'"Show the cart total in currency."$'\n\n'"## Acceptance criteria"$'\n\n'"$criteria" \
'{body: $body}' >"$forge/pr-overlay.json"

# One evidence file per side: one entry point and one screenshot per interaction.
for side in base head; do
dir="evidence/$side"
mkdir -p "$dir"
steps='[]' artifacts='[]' publishable='[]'
for ((i = 0; i < count; i++)); do
file="${names[i]}.png"
printf '%s' "$png" | base64 -d >"$dir/$file"
steps=$(jq -c --arg ep "${names[i]} page" --arg f "$file" \
'. + [{action: "open", entry_point: $ep, stated_result: null, observed: "page rendered", artifacts: [$f], outcome: "done"}]' <<<"$steps")
artifacts=$(jq -c --arg f "$file" '. + [{path: $f, kind: "screenshot", mask: "none-declared", publishable: true}]' <<<"$artifacts")
publishable=$(jq -c --arg f "$file" '. + [$f]' <<<"$publishable")
done
if ((e2e)); then
jq -n --argjson steps "$steps" --argjson publishable "$publishable" \
'{route: "drive", app: "shop", affected: ["cart"],
capture: {status: "ran", reason: null, count: ($publishable | length)},
entry_points: [$steps[] | "cart | \(.entry_point) | passed"], publishable: $publishable}' >"$dir/e2e-summary.json"
else
jq -n --argjson steps "$steps" --argjson artifacts "$artifacts" --argjson publishable "$publishable" \
'{app: "shop", control_skill: null, fallback: "playwright", resolved: null, mode: "unattended",
trusted_root: null, launch: [], steps: $steps, budgets: [], secrets_mark: [],
capture_count: ($artifacts | length), artifacts: $artifacts, values_none: [],
publishable: $publishable, capability_gaps: [], blocked: [], cleanup: []}' >"$dir/report.json"
fi
done

bash "$plugin/skills/pull-request/scripts/confirm-verdict.sh" record --head "$head_sha" --verdict CONFIRMED
8 changes: 7 additions & 1 deletion plugins/source-control/evals/fixtures/gh-forge/gh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@
# `pr close`, `pr update-branch`, `issue comment` and `issue edit`.
# reassess-state.sh seeds what the reassess cases need: pr-overlay.json (merged over the first pull request),
# comments.json (its conversation comments), reviews.json (its reviews) and merged-prs.json (pull requests
# already merged into the base). The last `pr comment` body is kept in comment-body.md for a grader.
# already merged into the base). The last `pr comment` body is kept in comment-body.md for a grader;
# `pr comment --help` lists the flags, `--attach` among them, that the evidence step checks for.
# shellcheck disable=SC2154 # PR_* come from the state file sourced below
set -uo pipefail

Expand Down Expand Up @@ -385,6 +386,11 @@ case "$cmd $sub" in
;;
"pr comment")
shift 2
if [[ " $* " == *" --help "* || " $* " == *" -h "* ]]; then
printf 'Usage: gh pr comment [<number> | <url> | <branch>] [flags]\n\nFlags:\n'
printf ' --attach file\n -b, --body text\n -F, --body-file file\n --create-if-none\n --delete-last\n --edit-last\n --yes\n'
exit 0
fi
parse "$@"
pr_matches "${args[0]:-}" || refuse pr comment "$@"
record "pr comment" "$@"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
schema_version: "1.1"
name: pull-request-evidence-e2e-summary-collapsed
context:
scaffold_script: scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"name": "gh-forge-path",
Comment thread
kyle-sexton marked this conversation as resolved.
Resolved
"version": "0.0.0",
Comment thread
kyle-sexton marked this conversation as resolved.
Resolved
"description": "Eval-only: puts the gh-forge stand-in first on the run's Bash PATH (fixtures/gh-forge/install.sh)",
"hooks": {
"SessionStart": [
{
"hooks": [
{
"type": "command",
"command": "g=\"$(git rev-parse --path-format=absolute --git-common-dir 2>/dev/null)\" || exit 0; d=\"$g/eval-forge/bin\"; if [ -n \"$CLAUDE_ENV_FILE\" ] && [ -x \"$d/gh\" ]; then printf 'export PATH=\"%s:$PATH\"\\n' \"$d\" >>\"$CLAUDE_ENV_FILE\"; fi"
}
]
}
]
Comment thread
kyle-sexton marked this conversation as resolved.
Resolved
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Write
input_match: '"content"\s*:\s*"(?=(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40})(?=(?:[^"\\]|\\.)*?<details(?:(?!</details>)(?:[^"\\]|\\.))*?!\[)(?!(?:[^"\\]|\\.)*?\\n\|\s*(?:\*\*)?Before(?:\*\*)?\s*\|)'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:source-control:)?pull-request"'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Write
input_match: '"content"\s*:\s*"(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40}(?:\\n)+(?!\||!\[|<details|<img)[^\\"]'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
description: Captures from e2e-summary.json files, which map no capture to an entry point, stay in collapsed sections even for a one-criterion change
tags: [pull-request, source-control, evidence]
plugins: ["../..", "gh-forge-path"]
runs: 3
max_turns: 40
timeout_seconds: 900
allowed_tools: [Read, Glob, Grep, Skill, Bash, Write]
expected_outcome: The evidence comment body written to a file opens with the text result, holds its captures inside a collapsed section, and has no Before/After table
---

The verifier passed PR 412's current head. Post its verification evidence comment. The head run's end-to-end summary is evidence/head/e2e-summary.json and the base run's is evidence/base/e2e-summary.json.
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
{
"pass": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n![Before: cart.png](/tmp/w/evidence/base/cart.png)\n\nAfter (head)\n\n![After: cart.png](/tmp/w/evidence/head/cart.png)\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "captures in the criterion's collapsed section"
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\n</details>\n\n<details>\n<summary>Other captures</summary>\n\n![After: cart.png](/tmp/w/evidence/head/cart.png)\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "captures under Other captures"
}
],
"fail": [
{
"answer": []
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n| | cart page |\n|---|---|\n| Before | ![cart.png](/tmp/w/evidence/base/cart.png)<br>cart page |\n| After | ![cart.png](/tmp/w/evidence/head/cart.png)<br>cart page |\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "a table for captures no step maps"
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n| | cart page |\n|---|---|\n| Before | ![cart.png](/tmp/w/evidence/base/cart.png)<br>cart page |\n| After | ![cart.png](/tmp/w/evidence/head/cart.png)<br>cart page |\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n![Before: cart.png](/tmp/w/evidence/base/cart.png)\n\nAfter (head)\n\n![After: cart.png](/tmp/w/evidence/head/cart.png)\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "a table beside the sections"
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>cart</summary>\n\npassed\n\n</details>\n\n![a](/tmp/a.png)\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "the capture sits outside every collapsed section"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"pass": [
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "source-control:pull-request"
}
}
]
}
],
"fail": [
{
"answer": []
},
{
"answer": [
{
"tool": "Skill",
"input": {
"skill": "source-control:commit"
}
}
],
"why": "wrong skill"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"pass": [
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n![Before: cart.png](/tmp/w/evidence/base/cart.png)\n\nAfter (head)\n\n![After: cart.png](/tmp/w/evidence/head/cart.png)\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "result line then the section"
}
],
"fail": [
{
"answer": []
},
{
"answer": [
{
"tool": "Write",
"input": {
"file_path": "/tmp/evidence-412.md",
"content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n![Before: cart.png](/tmp/w/evidence/base/cart.png)\n\nAfter (head)\n\n![After: cart.png](/tmp/w/evidence/head/cart.png)\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n"
}
}
],
"why": "a collapsed section first"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# Seeds PR 412 with one acceptance criterion and base and head e2e-summary.json files whose flat
# publishable lists map no capture to an entry point (fixtures/evidence-pr.sh).
set -euo pipefail

bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/evidence-pr.sh" 1 --e2e
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
schema_version: "1.1"
name: pull-request-evidence-large-change-collapsed
context:
scaffold_script: scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"name": "gh-forge-path",
"version": "0.0.0",
"description": "Eval-only: puts the gh-forge stand-in first on the run's Bash PATH (fixtures/gh-forge/install.sh)",
"hooks": {
"SessionStart": [
{
"hooks": [
{
"type": "command",
"command": "g=\"$(git rev-parse --path-format=absolute --git-common-dir 2>/dev/null)\" || exit 0; d=\"$g/eval-forge/bin\"; if [ -n \"$CLAUDE_ENV_FILE\" ] && [ -x \"$d/gh\" ]; then printf 'export PATH=\"%s:$PATH\"\\n' \"$d\" >>\"$CLAUDE_ENV_FILE\"; fi"
}
]
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Write
input_match: '"content"\s*:\s*"(?=(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40})(?=(?:[^"\\]|\\.)*?<details)(?=(?:[^"\\]|\\.)*?\\n(?:\*\*)?Before\b)(?=(?:[^"\\]|\\.)*?\\n(?:\*\*)?After\b)'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Write
input_match: '"content"\s*:\s*"(?=(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40})(?=(?:(?:[^"\\]|\\.)*?<details){3})(?!(?:[^"\\]|\\.)*?\\n\|\s*(?:\*\*)?Before(?:\*\*)?\s*\|)'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:source-control:)?pull-request"'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
type: tool_used
tool: Write
input_match: '"content"\s*:\s*"(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40}(?:\\n)+(?!\||!\[|<details|<img)[^\\"]'
---
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
description: A three-criterion change keeps one collapsed section per acceptance criterion, under the text result, each capture labeled with its run
tags: [pull-request, source-control, evidence]
plugins: ["../..", "gh-forge-path"]
runs: 3
max_turns: 40
timeout_seconds: 900
allowed_tools: [Read, Glob, Grep, Skill, Bash, Write]
expected_outcome: The evidence comment body written to a file opens with the text result under its heading and holds three collapsed sections, each capture labeled Before or After, and no Before/After table
---

The verifier passed PR 412's current head. Post its verification evidence comment. The head run's drive report is evidence/head/report.json and the base run's is evidence/base/report.json.
Loading
Loading