Repository navigation
CI: real-tool smoke for cargo-audit, cargo-deny and brakeman - #2374
Merged
Merged
Conversation
Adds real semgrep 1.157.0 (repo rules, no registry pack) and njsscan 0.4.2 canaries to the real-hub job. They install from their own hash-locked scripts/real-hub-requirements-semgrep.txt venv because semgrep pins click 8.1.8, jsonschema 4.25.1 and wcmatch 8.5.2, older than the versions the other tools resolve to. The trufflehog canary matches the exact detector name Github. provision_tools.sh pins checkov 3.4.5. review_checks.sh runs njsscan with the sandboxed HOME and no semgrep metrics, asserted by the hub stub test. Tests: 21/21 (hub secrets-node stub test; k8s-js and js-iac-license hub tests pass; real-hub canaries run in CI) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The real-hub semgrep and njsscan venv now resolves semgrep 1.182.1, the version provision_tools.sh installs for users (njsscan 0.4.2 is compatible). Its Linux wheels are manylinux_2_34, so the lock is re-derived for that platform. test-provision-tools.sh now expects the checkov==3.4.5 pin. Tests: 61/61 (test-provision-tools.sh) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The real-hub job now installs tflint 0.66.0 and kics 2.1.20 (sha256 from each release checksums.txt; kics queries from the release commit fetched by full sha and verified), cppcheck 2.13.0-2ubuntu3 (exact version from the signed noble archive; job pinned to ubuntu-24.04) and DevSkim CLI 1.0.100 (nupkg sha512 = nuget.org catalog packageHash). Each gets a fictional canary asserted through review_checks.sh: tool ran, expected rule on the canary file, no not-run or NOT CHECKED line. syft is skipped: the hub never counts it as ran. Tests: kics canary passes locally; tflint, cppcheck and devskim canaries run only in CI (not runnable in the local sandbox). Gates: bash -n, actionlint, privacy, size, size-ratchet, test-provision-tools.sh all green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x comments cppcheck is now the noble .deb fetched from its stable Launchpad URL and checked with sha256 (an exact apt version breaks once superseded); runtime deps still come from apt. The DevSkim nupkg is also checked with `dotnet nuget verify` when the runner's dotnet has that verb. The real-hub job header no longer calls cppcheck stub-only, and the raw kics rerun prints results.json even when kics exits non-zero. Tests: 5/5 gates (bash -n, actionlint, privacy, size, size-ratchet); test-provision-tools.sh passes against the committed provision_tools.sh (the worktree copy is stale and unrestorable in this sandbox). Canaries unchanged: kics ran locally earlier, tflint/cppcheck/devskim run in CI only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extend the real-hub job to run govulncheck 1.9.0 and grype 0.121.3 through review_checks.sh on fictional canaries, asserting each RAN with its expected rule and no not-run or NOT CHECKED line. grype: linux_amd64 archive checked against the release checksums file (and a literal hash). The canary is a one-component CycloneDX SBOM (lodash 4.17.15), because the hub points grype at SBOM files only, never at a lockfile. govulncheck: no release binaries, so built with go install on the pinned Go 1.26.11 (GOTOOLCHAIN=local), module hash asserted from go version -m, source verified by the Go checksum DB. The canary calls yaml.Unmarshal of the real module gopkg.in/yaml.v2 2.2.2; go mod tidy fills go.sum and the module cache, then the hub runs with GOPROXY=off. Both vulnerability DBs are ONLINE and NOT pinned: govulncheck has no env override for -db, so it reads vuln.go.dev at run time; the grype DB comes from grype db update (checksum-verified against its listing; a dated archive pin would rot when upstream rotates it out). Locally both tools passed the hub canary against mirrored DBs; the real CI run is the first with the live endpoints. Tests: 7/7 (bash -n, actionlint, privacy grep, size gate, size-ratchet, fragments, scripts/test-provision-tools.sh 61 ok 0 fail) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d grype smoke Raise the real-hub timeout to 25 minutes. Retry up to 3 times with backoff around the govulncheck go install, grype db update, go mod tidy for the canary, and the hub run while govulncheck or grype reads as not run (vuln.go.dev is fetched at run time). The grype canary now also accepts the CVE aliases of its GHSA advisories. The canary go.mod stays at the repo root: the hub runs govulncheck only from a root go.mod, so moving it would make the tool NOT CHECKED. Instead the test now fails if gosec or staticcheck report any row on the govulncheck canary package; gosec's own canary rows are unchanged in a local run. Tests: 7/7 (bash -n, actionlint, privacy grep, size gate, size-ratchet, fragments, scripts/test-provision-tools.sh 61 ok 0 fail) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts in ci.yml and test-review-checks-real-hub.sh resolved to the smoke-7 side (main carried only the squashed smoke-6 content). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run real trivy 0.76.2 through review_checks.sh in the real-hub job on a Dockerfile misconfig canary (DS-0001/DS-0002) and an npm lockfile canary (lodash 4.17.15). The binary is pinned by sha256 from the release checksums file; the vulnerability DB is pulled by OCI digest with bounded retries; the misconfig checks are the bundle embedded in the pinned binary (TRIVY_SKIP_CHECK_UPDATE=true). The test also fails on a secret-only fallback. codeql is not added: its CLI bundle and per-language database builds are too heavy for a per-push job. Tests: 7/7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verify the release checksums file's sigstore bundle with cosign verify-blob (cosign pinned by sha256; identity pinned to the trivy release workflow at the release tag, issuer to GitHub Actions OIDC). After the DB download, assert the log shows the pinned digest reference and a DB was written. The trivy canary now asserts DS-0001 and DS-0002 each, the debug raw command passes --cache-dir, and the redundant checksum count grep is dropped. Tests: 7/7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The real-hub job now installs osv-scanner 2.7.1 (sha256 from the release SHA256SUMS), shellcheck 0.11.0 (release asset digest, re-verified against a fresh download) and guarddog 3.2.0 (hash-locked in its own venv), with bounded fetch retries. osv-scanner's offline DB is pre-seeded with the command the hub's NOT CHECKED message names and exported as REVIEW_OSV_DB_DIR. The canary test asserts each tool RAN with no "not run" or "NOT CHECKED" line and its rule on its canary file: a GHSA row on the lodash 4.17.15 lockfile, SC2086, and threat-process-cryptomining on a newly added lock package (a full scan, not a partial one). Tests: 30/30 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verify the seeded osv-scanner DB zip with unzip -tq instead of only checking it is non-empty, give the guarddog pip install --retries 5, and add osv-scanner and guarddog to the real-hub test's bounded hub-run retry loop. Tests: 30/30 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add cargo-audit 0.23.0, cargo-deny 0.20.2 and brakeman 8.1.0 to the real-hub job with fictional canaries (a pinned time 0.1.45 lock, a GPL-3.0-only crate against an MIT/Apache-2.0 deny.toml, and a Rails controller that evals a parameter). Binaries and the gem are sha256-pinned; the RustSec advisory DB is fetched by a full commit sha and checked against FETCH_HEAD. The osv-scanner seed now also holds a crates.io DB. pip-audit is dropped: it is only a fallback the seeded osv DB never reaches, and it queries PyPI live, so it cannot be integrity-pinned. Tests: 30/30 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Real kubescape appends .json to an --output path that lacks it, so review_checks.sh read an empty report and marked kubescape not run on every real run. The hub now writes and reads a path ending in .json. The k8s stub test mimics the suffix rule, so it fails on the old code. The real-hub CI job now runs kubescape 3.0.48 (sha256 from the release checksums) with the regolibrary v1.0.316 nsa and cis-v1.23-t1.0.1 frameworks (sha256 recorded at pin time; that release publishes none) on a fictional privileged-pod canary, asserting control C-0057. Tests: 6/6 (k8s stub test 23 ok, real kubescape canary through the hub, bash -n, actionlint, changelog fragments, test-provision-tools) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml # scripts/test-review-checks-real-hub.sh
# Conflicts: # .github/workflows/ci.yml # scripts/test-review-checks-real-hub.sh
# Conflicts: # scripts/test-review-checks-real-hub.sh
# Conflicts: # .github/workflows/ci.yml # scripts/test-review-checks-real-hub.sh
# Conflicts: # scripts/test-review-checks-real-hub.sh
The govulncheck canary's go.sum is a lockfile the hub also hands to
osv-scanner. With only the npm DB seeded, osv-scanner exits 127 ("no
offline version") on it and the real-hub smoke reads osv-scanner as not
run. Seed a go.mod alongside the npm lockfile and require one verified
zip per seeded ecosystem.
Tests: 61/61
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
osv-scanner 2.x reads Go dependencies from go.mod only; passing a go.sum errors (exit 127) and the hub reported osv-scanner as not run. The lock-file selector now matches go.mod, and the offline-DB test asserts go.mod is passed and go.sum is not. Tests: 165/165 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add cargo-audit 0.23.0, cargo-deny 0.20.2 and brakeman 8.1.0 to the real-hub job
with fictional canaries (a pinned time 0.1.45 lock, a GPL-3.0-only crate against
an MIT/Apache-2.0 deny.toml, and a Rails controller that evals a parameter).
Binaries and the gem are sha256-pinned; the RustSec advisory DB is fetched by a
full commit sha and checked against FETCH_HEAD. The osv-scanner seed now also
holds a crates.io DB. pip-audit is dropped: it is only a fallback the seeded osv
DB never reaches, and it queries PyPI live, so it cannot be integrity-pinned.
Tests: 30/30
🤖 Generated with Claude Code