Skip to content

feat(testing): add run-e2e findings.json schema and findings compare script - #16576

Open
melodic-automation-lanes[bot] wants to merge 2 commits into
mainfrom
work-resolve-issue/issue-16566-43911324164-1
Open

melodic-automation-lanes[bot] wants to merge 2 commits into
mainfrom
work-resolve-issue/issue-16566-43911324164-1

Conversation

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Closes #16566

Summary

Opened by the work-resolve-issue lane. The run's own summary:

Adds the first part of #15649's whole-map live verification pass to the testing plugin's run-e2e skill: a findings.json JSON Schema, a findings-compare.sh script (bash + jq, no model call) that labels every finding id NEW, PRE-EXISTING, GONE or not checked, and a co-located test suite that CI's test-bash lane runs. Design choices made on evidence: the files sit under plugins/testing/skills/run-e2e/{scripts,schemas}/, following autonomy/skills/setup/schemas, because plugins/<name>/schemas/<concern>.schema.json is the convention-YAML schema slot that scripts/check-convention-yaml.sh reads. An id is GONE only when the newer pass drove (passed or failed) every entry point the baseline finding lists, matched by feature, part and entry point. Any other baseline-only id is not checked, with a not_driven list giving each entry point and its reason. Finding kind is result-not-met for a stated finding and one of error, crash, refusal, no-output or garbled-output for an unstated one, and the schema enforces that pairing.

Not verified

  • findings-compare.test.sh was not run here: this lane allows no bash or jq execution outside the named wrappers. The assertions were desk-checked by hand and by the review subagent. The suite is left to the PR's CI test-bash lane.
  • The new .sh files were written without the executable bit (the lane has no chmod). The test calls the script with bash, and CI runs suites with bash &lt;file&gt;.
  • Tidy skipped: the change is about 490 changed lines, over the 300-line tidy cutoff.
  • typos was not run (not installed in this lane). It is left to CI.
  • The schema was not checked by a JSON Schema validator here, since none is available. The test checks its required fields and enums with jq.

Fix

  • plugins/testing/skills/run-e2e/schemas/findings.schema.json: new JSON Schema (2020-12). Per finding: id, feature, part, kind, where (step number, proof or run), entry_points, severity, stated/unstated label and evidence paths. Per pass: build, plus each entry's outcome (passed, failed, not checked…
  • plugins/testing/skills/run-e2e/scripts/findings-compare.sh: new script, findings-compare.sh \[--baseline &lt;findings.json&gt;\] &lt;findings.json&gt;. Prints JSON with both builds, counts per label and the labeled findings, ordered NEW, PRE-EXISTING, GONE, not checked, then by severity, then by…
  • plugins/testing/skills/run-e2e/scripts/findings-compare.test.sh: new suite. Covers each label, the no-baseline case, the not-checked rule (entry cut by the cap, one of two entry points blocked, entry missing from the newer pass, same entry point under another part), a GONE on an entry that failed…
  • .changes/testing/work-resolve-issue-issue-16566-43911324164-1-e42d3e50.md: minor changelog fragment for the testing plugin.

Verification

The runner ran each check from the base copy against 551ac10:

  • scripts/check-changelog-fragments.sh --check: passed
  • scripts/check-changed-skills.sh --merge-base: passed

The run reports:

  • check-changelog-fragments: all 8 fragments valid
  • check-changed-skills: run-e2e PASS with 0 errors. The 5 warnings are about the existing SKILL.md and context/e2e.md, which this change does not touch.
  • check-affected: the first run failed shellcheck (SC2016 on two single-quoted jq paths in the test). After the fix: 15 passed, 6 skipped, 0 failed over the 4 changed files.
  • review: a fresh-context subagent desk-checked the jq and every test assertion. It found that an empty or multi-value input file passed the input check, and that a baseline finding with an empty entry_points list would come out GONE. Both are fixed (slurped single-value check, non-empty…
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43911324164

Left out: the items under Not verified.

Related

#16566

Lane-Activity: work-resolve-issue/resolve-issue
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 491686f12684daf8d532c6a3289d4b4ff5078d8a: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16576 (incremental, head 491686f)

Scope: what changed since 551ac10. That is one file, plugins/testing/skills/run-e2e/scripts/findings-compare.test.sh, and I read every hunk of it. Because this lane already reviewed an earlier head, this round reports only CRITICAL and IMPORTANT findings.

What changed: the test helper nd() is renamed to not_driven(), and its four call sites are updated to match. The jq filter is still a constant single-quoted program. The id still reaches jq only through --arg, and the call sites are quoted the same way as before. The change does not touch a trust control, a merge hold, a plugin manifest, a standing instruction or any TRUSTED PATHS file.

Earlier summary comment: it had no line anchor and reported no findings, so there is nothing to restate.

Spend: none. The change makes no model call and uses no billable resource.

No security issues found.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 491686f12684daf8d532c6a3289d4b4ff5078d8a: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, blocking only): no findings

This lane already reviewed an earlier head, so this pass reports blocking findings only.

The push since 551ac109 changes one file, plugins/testing/skills/run-e2e/scripts/findings-compare.test.sh, and I read its whole hunk. The change renames the test helper nd() to not_driven() and updates its four call sites in the not-checked rule assertions. The body and expected values are unchanged.

The prior summary comment has no line anchor, and it carried no findings, so there is nothing to repeat. Spend: none.

Verifier dropped 0 of 0 candidates.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head 551ac109d31c94504f05a9b3b80d291cd8bcea21. No person needs to act: a lane clears this reason, and the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:missing)

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/fix-ci
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43917583125

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-unblock/apply-patch
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed 491686f
Run 43917583125

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run a whole-map live verification pass on demand and report what is new since the last pass - part 1 of 6

0 participants