Skip to content

feat: baseline / false-positive suppression (closes #88)#106

Open
assinchu wants to merge 1 commit into
NVIDIA:mainfrom
assinchu:feature/baseline-suppression
Open

feat: baseline / false-positive suppression (closes #88)#106
assinchu wants to merge 1 commit into
NVIDIA:mainfrom
assinchu:feature/baseline-suppression

Conversation

@assinchu

Copy link
Copy Markdown

Summary

Implements the baseline / false-positive suppression mechanism requested in #88.
Lets users suppress known/accepted findings so the risk score reflects only
un-triaged issues and re-scans surface only new findings — addressing the
"~95% false positives from framework patterns, every scan re-reports the same
issues" problem raised in that issue.

Without a baseline, behavior is unchanged.

What it adds

Two suppression mechanisms in one YAML/JSON baseline:

  • rules — glob-based, drift-tolerant: match on rule id, file path, and/or
    message (covers both global pattern suppression and skill/file-scoped
    suppression). Each rule carries a required reason.
  • fingerprints — exact per-finding hashes, machine-generated for incremental
    CI use.

CLI:

  • scan --baseline/-b PATH and --show-suppressed
  • new skillspector baseline <path> command to generate a baseline from a scan

Design

Suppression is applied in the report node (the single scoring/formatting
point, reused by CLI and any future REST API). Suppressed findings never count
toward the risk score and are excluded from SARIF results; they appear in
terminal/markdown reports only with --show-suppressed, and are always listed
in JSON output under suppressed/suppressed_count.

New module: skillspector/suppression.py (Baseline, SuppressionRule,
load_baseline, partition_findings, finding_fingerprint, build_baseline_dict).

Tests & docs

  • New tests/unit/test_suppression.py + suppression cases in test_report.py
    and test_cli.py. Full suite passes; ruff check/format clean.
  • New docs/SUPPRESSION.md, .skillspector-baseline.example.yaml, and
    README/DEVELOPMENT.md updates.

Example

# Accept current findings, then re-scan against the baseline
skillspector baseline ./my-skill/ -o .skillspector-baseline.yaml
skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-suppressed

Closes #88

Adds a baseline mechanism so known/accepted findings can be suppressed,
letting the risk score reflect only un-triaged issues and re-scans surface
only new findings. Suppressed findings never count toward the score and are
excluded from SARIF; they appear in reports only with --show-suppressed and
are always listed in JSON output.

Two complementary suppression mechanisms in one YAML/JSON baseline:
- rules: glob-based (rule id / file path / message), drift-tolerant
- fingerprints: exact per-finding hashes, machine-generated

CLI:
- scan --baseline/-b PATH, --show-suppressed
- new `skillspector baseline <path>` command to generate a baseline

Suppression is applied in the report node (the single scoring/formatting
point) so the CLI and any future REST API behave identically. Includes a
new skillspector.suppression module, unit/report/CLI tests, an example
baseline, and docs (docs/SUPPRESSION.md).

Closes NVIDIA#88

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Aravinda Sharma <7734009+assinchu@users.noreply.github.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really solid, well-architected feature. Approving.

  • Applying suppression in the single report node (the one scoring/formatting point) is the right call — CLI and any future API stay consistent, suppressed findings are excluded from the risk score and SARIF, always machine-readable in JSON, and shown in terminal/markdown only under --show-suppressed.
  • suppression.py is clean: glob rules (drift-tolerant) + exact fingerprints, the all-wildcard rule guard prevents accidentally suppressing everything, and partition_findings keeps everything when no baseline is supplied. Backward compatibility is preserved and covered by test_report_no_baseline_unchanged.
  • Good test depth across test_suppression.py, report tests, and a CLI generate→scan round-trip; docs (SUPPRESSION.md, README, DEVELOPMENT.md, example file) are thorough.

Minor / optional (non-blocking):

  1. SARIF: instead of dropping suppressed results entirely, consider emitting them with the SARIF suppressions property on the result. That keeps an auditable trail in the SARIF while still excluding them from scoring (consumers like code-scanning understand suppressions).
  2. The example baseline / docs use identifiers, file paths, and an env-var name that read as references to specific real-world skills/projects. For a public example, consider neutral placeholders (e.g. example-skill/SKILL.md, $EXAMPLE_TOKEN) so the docs stay generic.
  3. _match_glob uses fnmatch, so a rule_id/message containing literal glob metacharacters ([, ?) would be interpreted as a pattern. Edge case only; worth a comment or fnmatch.translate escaping if you want literal matching.

@assinchu

Copy link
Copy Markdown
Author

Thanks for the review, @rng1995 . Please merge as needed.

@rng1995

rng1995 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

@assinchu - Please fix the Minor / optional (non-blocking) issues and resolve conflicts to merge this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: False-positive suppression / baseline mechanism

2 participants