Repository navigation
CI: real-tool smoke for review_checks.sh hub scanners - #2361
Merged
Merged
Conversation
Add a real-hub job that installs the real tfsec 1.28.11 (sha256-checked), ansible-lint 24.12.2, cfn-lint 1.20.0 and rubocop 1.69.2 and runs scripts/test-review-checks-real-hub.sh: one fictional known-bad canary per tool through review_checks.sh, asserting each is reported as RAN with its expected rule id. bundler-audit, kubescape and retire need a downloaded DB or rule set and stay stub-tested. The job joins the gates aggregate. Tests: 0/0 locally (the 4 real-tool canaries run only in the CI real-hub job; local gates: actionlint, bash -n, size, size-ratchet, fragments) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Match rows on the tool's canary path as well as rule id, run with set -euo pipefail while still capturing review_checks.sh's exit code, and restrict the tfsec download to https/TLS 1.2. Tests: 0/0 (canaries run in CI) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n failure Security/Eval ignores a plain string literal argument (cop pattern $!str), so the canary now evals a non-literal. On a failing tool the smoke reruns the hub argv and prints the first 40 lines of raw output so the next CI failure is diagnosable. The ansible-lint hub invocation is unchanged: upstream v24.12.2 prints -f sarif to stdout, so the cause of the missing JSON needs that raw output. Tests: 2/2 stub suites (hub-tf-ansible, hub-rubocop); real-hub runs in CI Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ansible-lint 24.12.2 crashed on import under the runner's Python 3.14 with a floating, newer ansible-core. Pin a compatible pair that both list Python 3.14. Tests: 21/21 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a real-hub CI job. It installs pinned real tfsec (sha256-checked binary), ansible-lint, cfn-lint and rubocop, then runs one fictional canary per tool through review_checks.sh. The job fails unless each tool is reported as run, the expected rule id appears on the canary file, and no NOT CHECKED or not-run line appears. The gates job depends on it.
Until now the hub tests used stub binaries only. This job is the first check against the real tools. The expected rule ids for tfsec and cfn-lint match by rule family because the exact ids are unverified; a wrong id makes the job fail rather than pass. Transitive pip and gem dependencies are not hash-locked yet; the job has read-only contents permission and no secrets. pmd and solhint are not covered until their PR lands.
Tests: canaries run in CI only (real tools are not installed locally). actionlint, bash -n, the size gate and the size ratchet pass. An independent review found no way for the job to pass without the tools running; its three minor findings are fixed.
🤖 Generated with Claude Code