fix(report): filter empty LLM findings and add SARIF rules[] array#158
Open
mimran-khan wants to merge 1 commit into
Open
fix(report): filter empty LLM findings and add SARIF rules[] array#158mimran-khan wants to merge 1 commit into
mimran-khan wants to merge 1 commit into
Conversation
Two SARIF output issues fixed: 1. Empty findings (missing rule_id or message) from LLM meta-analyzer failures are now filtered before SARIF serialization. Previously these could produce invalid SARIF results with empty ruleId. 2. SARIF output now includes the tool.driver.rules[] array containing reportingDescriptor entries for each unique rule referenced by results. This brings output closer to SARIF 2.1.0 spec compliance and enables IDE integrations that require rule metadata. Added SarifReportingDescriptor model to sarif_models.py. Fixes NVIDIA#146, NVIDIA#148
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two SARIF output correctness issues:
Empty findings from LLM failures inflate results — When the LLM meta-analyzer fails or returns partial data, findings with empty
rule_idormessageleak into the SARIF output, producing invalid results that confuse downstream tooling (IDEs, CI integrations).Missing
tool.driver.rules[]array — The SARIF 2.1.0 spec recommendsrules[]containingreportingDescriptorentries for each rule referenced by results. Many consumers (GitHub Code Scanning, VS Code SARIF Viewer) use this to display rule metadata.Changes
src/skillspector/sarif_models.pySarifReportingDescriptormodel withidandshortDescriptionfieldsrulesfield toSarifDriversrc/skillspector/nodes/report.py(_build_sarif)rule_idormessageis falsy before building resultsrules[]array from seen rule IDsTesting
12 new tests covering:
rule_idfilteredmessagefilteredrules[]present with correct id and shortDescriptionExisting 44 tests in
test_sarif.pyandtest_report.pycontinue to pass.Fixes #146, Fixes #148