Repository navigation
feat(source-control): show small-change evidence inline and add an optional record block #16616
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+1,368
−27
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
1b085b1
feat(source-control): show small-change evidence inline and add an op…
kyle-sexton bc8bc43
test(source-control): serve pr comment --help in the eval forge
kyle-sexton 1e95a71
fix(source-control): tighten the evidence layout rules from review
kyle-sexton b66aa2b
fix(source-control): label collapsed evidence captures by run and poi…
kyle-sexton 1d26532
fix(source-control): keep unmapped evidence captures in collapsed sec…
kyle-sexton 30ffb8b
Merge branch 'main' into feat/pr-evidence-size-rule
kyle-sexton acb793e
docs(source-control): cite the FDA assurance guidance by title, not a…
kyle-sexton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
13 changes: 13 additions & 0 deletions
13
.changes/source-control/feat-evidence-inline-table-record-4e7a19c2.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| --- | ||
| bump: minor | ||
| --- | ||
|
|
||
| ### Added | ||
|
|
||
| - **The evidence comment shows a small change inline.** `/source-control:pull-request evidence` lays out a change with at most two acceptance criteria and at most `evidence_inline_max_captures` captures (default 4), each mapped to an entry point by a drive `report.json`, as a Before/After table, one column per interaction, a caption under each image, and one line naming the capturing tool and what each row's state is. Any other change, an `e2e-summary.json` input included, keeps one collapsed section per acceptance criterion and `Other captures`. `--before <file>` takes the base commit's evidence file, governed by its own `publishable` list. A video sits in its own paragraph under the table. | ||
| - **Optional record block.** With `evidence_record_fields: true` (off by default) the comment carries who ran the verification (`--verified-by`), when (the evidence file's write time) and the issues found with their resolution (`--issues`, else the run's failed entries). | ||
| - `scripts/read-evidence-layout.sh` reads both keys from `docs/conventions/source-control.yaml` at the pull request's base, honored only on the default branch, with tests. | ||
|
|
||
| ### Changed | ||
|
|
||
| - The evidence comment leads with the run's text result, one line per criterion or entry point; captures support it. The verifier-verdict and leak-scan gates are unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| #!/usr/bin/env bash | ||
| # Seeds an evidence-comment case: a repository whose feature branch is one commit ahead of main, an | ||
| # offline forge (gh-forge) serving open PR 412 with an `## Acceptance criteria` body, a CONFIRMED | ||
| # verifier verdict recorded for the head, and two /app-control:drive report.json files, one from | ||
| # the head (evidence/head) and one from the base (evidence/base), each listing one PNG per | ||
| # interaction in `publishable` under the same file name on both sides. Run it from a case's | ||
| # scaffold.sh in the workspace: | ||
| # bash evidence-pr.sh <interactions> [--record-fields] [--e2e] | ||
| # <interactions> is the number of acceptance criteria and of driven entry points (one capture | ||
| # each per run). --record-fields commits `evidence_record_fields: true` to main's team config. | ||
| # --e2e writes each side's /testing:run-e2e e2e-summary.json in place of report.json; its flat | ||
| # `publishable` list maps no capture to an entry point. | ||
| set -euo pipefail | ||
|
|
||
| count="${1:?evidence-pr.sh needs an interaction count}" | ||
| record=0 e2e=0 | ||
| for flag in "${@:2}"; do | ||
| case "$flag" in | ||
| --record-fields) record=1 ;; | ||
| --e2e) e2e=1 ;; | ||
| *) echo "evidence-pr.sh: unknown argument $flag" >&2; exit 2 ;; | ||
| esac | ||
| done | ||
| here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| plugin="$(cd "$here/../.." && pwd)" | ||
| png='iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==' | ||
| names=(cart checkout receipt orders profile) | ||
|
|
||
| git init -q -b main | ||
| git config user.name eval | ||
| git config user.email eval@example.invalid | ||
| mkdir -p docs/conventions src | ||
| printf 'export const total = (items) => items.length;\n' >src/cart.ts | ||
| printf 'pr_body_required_sections:\n - Summary\n - Verification\n' >docs/conventions/source-control.yaml | ||
| ((record)) && printf 'evidence_record_fields: true\n' >>docs/conventions/source-control.yaml | ||
| printf 'evidence/\n' >.gitignore | ||
| git add docs src .gitignore | ||
| git commit -q -m "chore: seed" | ||
|
|
||
| git checkout -q -b feat/cart-totals | ||
| printf 'export const total = (items) => items.reduce((s, i) => s + i.price, 0);\n' >src/cart.ts | ||
| git add src/cart.ts | ||
| git commit -q -m "feat: show the cart total in currency" | ||
| head_sha="$(git rev-parse HEAD)" | ||
|
|
||
| bash "$here/gh-forge/install.sh" 412 feat/cart-totals main "feat: show the cart total in currency" --open | ||
| forge="$(git rev-parse --path-format=absolute --git-common-dir)/eval-forge" | ||
|
|
||
| criteria="" | ||
| for ((i = 0; i < count; i++)); do | ||
| criteria+="- The ${names[i]} page shows the total in currency"$'\n' | ||
| done | ||
| jq -n --arg body "## Summary"$'\n\n'"Show the cart total in currency."$'\n\n'"## Acceptance criteria"$'\n\n'"$criteria" \ | ||
| '{body: $body}' >"$forge/pr-overlay.json" | ||
|
|
||
| # One evidence file per side: one entry point and one screenshot per interaction. | ||
| for side in base head; do | ||
| dir="evidence/$side" | ||
| mkdir -p "$dir" | ||
| steps='[]' artifacts='[]' publishable='[]' | ||
| for ((i = 0; i < count; i++)); do | ||
| file="${names[i]}.png" | ||
| printf '%s' "$png" | base64 -d >"$dir/$file" | ||
| steps=$(jq -c --arg ep "${names[i]} page" --arg f "$file" \ | ||
| '. + [{action: "open", entry_point: $ep, stated_result: null, observed: "page rendered", artifacts: [$f], outcome: "done"}]' <<<"$steps") | ||
| artifacts=$(jq -c --arg f "$file" '. + [{path: $f, kind: "screenshot", mask: "none-declared", publishable: true}]' <<<"$artifacts") | ||
| publishable=$(jq -c --arg f "$file" '. + [$f]' <<<"$publishable") | ||
| done | ||
| if ((e2e)); then | ||
| jq -n --argjson steps "$steps" --argjson publishable "$publishable" \ | ||
| '{route: "drive", app: "shop", affected: ["cart"], | ||
| capture: {status: "ran", reason: null, count: ($publishable | length)}, | ||
| entry_points: [$steps[] | "cart | \(.entry_point) | passed"], publishable: $publishable}' >"$dir/e2e-summary.json" | ||
| else | ||
| jq -n --argjson steps "$steps" --argjson artifacts "$artifacts" --argjson publishable "$publishable" \ | ||
| '{app: "shop", control_skill: null, fallback: "playwright", resolved: null, mode: "unattended", | ||
| trusted_root: null, launch: [], steps: $steps, budgets: [], secrets_mark: [], | ||
| capture_count: ($artifacts | length), artifacts: $artifacts, values_none: [], | ||
| publishable: $publishable, capability_gaps: [], blocked: [], cleanup: []}' >"$dir/report.json" | ||
| fi | ||
| done | ||
|
|
||
| bash "$plugin/skills/pull-request/scripts/confirm-verdict.sh" record --head "$head_sha" --verdict CONFIRMED |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 4 additions & 0 deletions
4
plugins/source-control/evals/pull-request-evidence-e2e-summary-collapsed/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| schema_version: "1.1" | ||
| name: pull-request-evidence-e2e-summary-collapsed | ||
| context: | ||
| scaffold_script: scaffold.sh |
17 changes: 17 additions & 0 deletions
17
...vals/pull-request-evidence-e2e-summary-collapsed/gh-forge-path/.claude-plugin/plugin.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "name": "gh-forge-path", | ||
| "version": "0.0.0", | ||
|
kyle-sexton marked this conversation as resolved.
Resolved
|
||
| "description": "Eval-only: puts the gh-forge stand-in first on the run's Bash PATH (fixtures/gh-forge/install.sh)", | ||
| "hooks": { | ||
| "SessionStart": [ | ||
| { | ||
| "hooks": [ | ||
| { | ||
| "type": "command", | ||
| "command": "g=\"$(git rev-parse --path-format=absolute --git-common-dir 2>/dev/null)\" || exit 0; d=\"$g/eval-forge/bin\"; if [ -n \"$CLAUDE_ENV_FILE\" ] && [ -x \"$d/gh\" ]; then printf 'export PATH=\"%s:$PATH\"\\n' \"$d\" >>\"$CLAUDE_ENV_FILE\"; fi" | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
|
kyle-sexton marked this conversation as resolved.
Resolved
|
||
| } | ||
| } | ||
5 changes: 5 additions & 0 deletions
5
...request-evidence-e2e-summary-collapsed/graders/captures-in-collapsed-section.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Write | ||
| input_match: '"content"\s*:\s*"(?=(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40})(?=(?:[^"\\]|\\.)*?<details(?:(?!</details>)(?:[^"\\]|\\.))*?!\[)(?!(?:[^"\\]|\\.)*?\\n\|\s*(?:\*\*)?Before(?:\*\*)?\s*\|)' | ||
| --- |
5 changes: 5 additions & 0 deletions
5
...ontrol/evals/pull-request-evidence-e2e-summary-collapsed/graders/skill-fired.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Skill | ||
| input_match: '"skill"\s*:\s*"(?:source-control:)?pull-request"' | ||
| --- |
5 changes: 5 additions & 0 deletions
5
...control/evals/pull-request-evidence-e2e-summary-collapsed/graders/text-first.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Write | ||
| input_match: '"content"\s*:\s*"(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40}(?:\\n)+(?!\||!\[|<details|<img)[^\\"]' | ||
| --- |
12 changes: 12 additions & 0 deletions
12
plugins/source-control/evals/pull-request-evidence-e2e-summary-collapsed/prompt.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| description: Captures from e2e-summary.json files, which map no capture to an entry point, stay in collapsed sections even for a one-criterion change | ||
| tags: [pull-request, source-control, evidence] | ||
| plugins: ["../..", "gh-forge-path"] | ||
| runs: 3 | ||
| max_turns: 40 | ||
| timeout_seconds: 900 | ||
| allowed_tools: [Read, Glob, Grep, Skill, Bash, Write] | ||
| expected_outcome: The evidence comment body written to a file opens with the text result, holds its captures inside a collapsed section, and has no Before/After table | ||
| --- | ||
|
|
||
| The verifier passed PR 412's current head. Post its verification evidence comment. The head run's end-to-end summary is evidence/head/e2e-summary.json and the base run's is evidence/base/e2e-summary.json. |
69 changes: 69 additions & 0 deletions
69
...ls/pull-request-evidence-e2e-summary-collapsed/samples/captures-in-collapsed-section.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n\n\nAfter (head)\n\n\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "captures in the criterion's collapsed section" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\n</details>\n\n<details>\n<summary>Other captures</summary>\n\n\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "captures under Other captures" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [] | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n| | cart page |\n|---|---|\n| Before | <br>cart page |\n| After | <br>cart page |\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a table for captures no step maps" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n| | cart page |\n|---|---|\n| Before | <br>cart page |\n| After | <br>cart page |\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n\n\nAfter (head)\n\n\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a table beside the sections" | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>cart</summary>\n\npassed\n\n</details>\n\n\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "the capture sits outside every collapsed section" | ||
| } | ||
| ] | ||
| } |
30 changes: 30 additions & 0 deletions
30
...source-control/evals/pull-request-evidence-e2e-summary-collapsed/samples/skill-fired.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Skill", | ||
| "input": { | ||
| "skill": "source-control:pull-request" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [] | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Skill", | ||
| "input": { | ||
| "skill": "source-control:commit" | ||
| } | ||
| } | ||
| ], | ||
| "why": "wrong skill" | ||
| } | ||
| ] | ||
| } |
33 changes: 33 additions & 0 deletions
33
.../source-control/evals/pull-request-evidence-e2e-summary-collapsed/samples/text-first.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n- cart | cart page | passed\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n\n\nAfter (head)\n\n\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "result line then the section" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": [] | ||
| }, | ||
| { | ||
| "answer": [ | ||
| { | ||
| "tool": "Write", | ||
| "input": { | ||
| "file_path": "/tmp/evidence-412.md", | ||
| "content": "<!-- evidence-verification -->\n### Verification evidence for 58acec248c73c4308349647924aaa337a9d02fb6\n\n<details>\n<summary>The cart page shows the total in currency</summary>\n\npassed\n\nBefore (base main)\n\n\n\nAfter (head)\n\n\n\n</details>\n\n<sub>One comment per pull request, updated in place for each new head.</sub>\n" | ||
| } | ||
| } | ||
| ], | ||
| "why": "a collapsed section first" | ||
| } | ||
| ] | ||
| } |
6 changes: 6 additions & 0 deletions
6
plugins/source-control/evals/pull-request-evidence-e2e-summary-collapsed/scaffold.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| #!/usr/bin/env bash | ||
| # Seeds PR 412 with one acceptance criterion and base and head e2e-summary.json files whose flat | ||
| # publishable lists map no capture to an entry point (fixtures/evidence-pr.sh). | ||
| set -euo pipefail | ||
|
|
||
| bash "$(dirname "${BASH_SOURCE[0]}")/../fixtures/evidence-pr.sh" 1 --e2e |
4 changes: 4 additions & 0 deletions
4
plugins/source-control/evals/pull-request-evidence-large-change-collapsed/case.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| schema_version: "1.1" | ||
| name: pull-request-evidence-large-change-collapsed | ||
| context: | ||
| scaffold_script: scaffold.sh |
17 changes: 17 additions & 0 deletions
17
...als/pull-request-evidence-large-change-collapsed/gh-forge-path/.claude-plugin/plugin.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "name": "gh-forge-path", | ||
| "version": "0.0.0", | ||
| "description": "Eval-only: puts the gh-forge stand-in first on the run's Bash PATH (fixtures/gh-forge/install.sh)", | ||
| "hooks": { | ||
| "SessionStart": [ | ||
| { | ||
| "hooks": [ | ||
| { | ||
| "type": "command", | ||
| "command": "g=\"$(git rev-parse --path-format=absolute --git-common-dir 2>/dev/null)\" || exit 0; d=\"$g/eval-forge/bin\"; if [ -n \"$CLAUDE_ENV_FILE\" ] && [ -x \"$d/gh\" ]; then printf 'export PATH=\"%s:$PATH\"\\n' \"$d\" >>\"$CLAUDE_ENV_FILE\"; fi" | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| } |
5 changes: 5 additions & 0 deletions
5
...pull-request-evidence-large-change-collapsed/graders/captures-labeled-by-run.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Write | ||
| input_match: '"content"\s*:\s*"(?=(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40})(?=(?:[^"\\]|\\.)*?<details)(?=(?:[^"\\]|\\.)*?\\n(?:\*\*)?Before\b)(?=(?:[^"\\]|\\.)*?\\n(?:\*\*)?After\b)' | ||
| --- |
5 changes: 5 additions & 0 deletions
5
...vals/pull-request-evidence-large-change-collapsed/graders/collapsed-sections.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Write | ||
| input_match: '"content"\s*:\s*"(?=(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40})(?=(?:(?:[^"\\]|\\.)*?<details){3})(?!(?:[^"\\]|\\.)*?\\n\|\s*(?:\*\*)?Before(?:\*\*)?\s*\|)' | ||
| --- |
5 changes: 5 additions & 0 deletions
5
...ntrol/evals/pull-request-evidence-large-change-collapsed/graders/skill-fired.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Skill | ||
| input_match: '"skill"\s*:\s*"(?:source-control:)?pull-request"' | ||
| --- |
5 changes: 5 additions & 0 deletions
5
...ontrol/evals/pull-request-evidence-large-change-collapsed/graders/text-first.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| type: tool_used | ||
| tool: Write | ||
| input_match: '"content"\s*:\s*"(?:[^"\\]|\\.)*?Verification evidence for [0-9a-f]{40}(?:\\n)+(?!\||!\[|<details|<img)[^\\"]' | ||
| --- |
12 changes: 12 additions & 0 deletions
12
...ins/source-control/evals/pull-request-evidence-large-change-collapsed/prompt.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| description: A three-criterion change keeps one collapsed section per acceptance criterion, under the text result, each capture labeled with its run | ||
| tags: [pull-request, source-control, evidence] | ||
| plugins: ["../..", "gh-forge-path"] | ||
| runs: 3 | ||
| max_turns: 40 | ||
| timeout_seconds: 900 | ||
| allowed_tools: [Read, Glob, Grep, Skill, Bash, Write] | ||
| expected_outcome: The evidence comment body written to a file opens with the text result under its heading and holds three collapsed sections, each capture labeled Before or After, and no Before/After table | ||
| --- | ||
|
|
||
| The verifier passed PR 412's current head. Post its verification evidence comment. The head run's drive report is evidence/head/report.json and the base run's is evidence/base/report.json. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.