Skip to content

fix(meta_analyzer): store None-end fallback key so static findings are not dropped#94

Open
Shrotriya-lalit wants to merge 2 commits into
NVIDIA:mainfrom
Shrotriya-lalit:fix/issue-67-apply-filter-end-line
Open

fix(meta_analyzer): store None-end fallback key so static findings are not dropped#94
Shrotriya-lalit wants to merge 2 commits into
NVIDIA:mainfrom
Shrotriya-lalit:fix/issue-67-apply-filter-end-line

Conversation

@Shrotriya-lalit

Copy link
Copy Markdown
Contributor

Problem

LLMMetaAnalyzer.apply_filter() stored LLM-confirmed findings keyed by:

(file, rule_id, start_line, end_line)

Static analyzers almost never populate end_line — it defaults to None.
The LLM, however, fills in end_line for every finding it confirms.

This key mismatch meant every static finding with end_line=None was silently
dropped after LLM enrichment, even when the LLM had correctly confirmed it.
In practice, the meta-analyzer produced zero filtered findings for entire files.

Root Cause

# Before fix — only exact key stored:
confirmed_granular[(file, rule_id, int(start), int(end))] = enrichment
# Static finding has end_line=None → no match → finding dropped

Fix

When storing a granular LLM confirmation with an explicit end_line, also store
a fallback key with end_line=None via setdefault:

confirmed_granular[(file, rule_id, int(start), exact_end)] = enrichment
confirmed_granular.setdefault(
    (file, rule_id, int(start), None), enrichment
)

setdefault is used so that if the LLM also reports a finding at the same
start_line with end_line=None, its enrichment takes precedence. Lookup order
is unchanged: exact key → start-only key → coarse key.

Tests

  • test_static_end_line_none_confirmed_when_llm_returns_end_line — reproduces the bug exactly
  • test_static_end_line_none_multiple_same_rule — multiple findings same rule, different start lines
  • test_end_line_used_when_provided — regression: exact end_line match still works

Closes #67

Checklist

  • make lint passes
  • Tests added for new behaviour (including bug reproduction test)
  • DCO sign-off on commit (git commit -s)

…e not dropped

apply_filter stored LLM confirmations keyed by (file, rule_id, start, end_line).
Static analyzers almost never record end_line (it defaults to None), while the
LLM fills it in for every finding it confirms.  This key mismatch caused every
static finding with end_line=None to be silently dropped when the LLM returned
an explicit end_line, effectively discarding all static results after LLM
enrichment.

Fix: when storing a granular LLM confirmation with an explicit end_line N, also
store a fallback key with end_line=None via setdefault so that a static finding
at the same (file, rule_id, start_line) with end_line=None still matches.

Closes NVIDIA#67

Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct fix for the dropped-findings bug. Approving.

Static analyzers almost always leave end_line=None, while the LLM fills it in, so the exact (file, rule, start, end) key never matched and confirmed findings were silently dropped. Storing the extra (file, rule, start, None) fallback via setdefault (so an explicit LLM end_line=None still takes precedence) is the minimal, correct change, and the lookup order is unchanged. Good that the reproduction test, the multi-finding test, and the updated test_end_line_used_when_provided all pin the new behavior — and I confirmed rejected findings (is_vulnerability=False) are never stored, so they stay rejected.

Minor / optional (non-blocking): the None-end fallback means two static findings at the same (file, rule, start_line) with different end_lines will now both be confirmed if the LLM confirms that start_line — a slight over-match. For a security tool that's the safe direction (avoids false negatives), just noting the behavior.

@Shrotriya-lalit

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review @rng1995!

On the over-match observation: you're right that two static findings at the same (file, rule, start_line) with different end_line values will both be confirmed via the None fallback when the LLM confirms that start_line. This is an intentional trade-off — for a security scanner, a false negative (silently dropping a real finding) is worse than a slight over-match on an edge case that only arises when the same rule fires twice at the same line with different end boundaries, which is uncommon in practice.

If the over-match becomes a problem, a follow-up could track confirmed end_lines per (file, rule, start_line) and only apply the None fallback when no exact end_line match exists. Happy to do that as a separate PR if the maintainers prefer tighter semantics.

@rng1995

rng1995 commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Please resolve the merge conflicts and do separate PR for tighter semantics

…t for tighter semantics

Upstream main introduced a cleaner solution to issue NVIDIA#67: instead of storing a
None-end fallback key in confirmed_granular via setdefault, it keeps a separate
confirmed_by_start dict keyed by (file, rule_id, start_line) and checks it only
when f.end_line is None.  This prevents the over-match the reviewer flagged: a
static finding with an explicit end_line that differs from the LLM's end_line is
no longer erroneously confirmed.

Also resolve a duplicate TestRunStaticPatternsAgentSnooping class (our branch
had our original copy; main already landed the canonical upstream version with
two additional tests: test_no_same_line_duplicate and test_node_runs_over_state).

Fix the agent snooping AS1 patterns to match absolute paths (e.g.
open("/Users/x/.claude/settings.json")) in addition to relative paths, and add
per-(rule_id, line) deduplication so a single line matching two patterns only
produces one finding.

Update test_end_line_used_when_provided to use a real end_line=None Finding
(the _make_finding helper converts None→line via 'or') and assert the correct
two-result behaviour under the tighter semantics.

876 tests pass, ruff clean.

Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>
@Shrotriya-lalit

Copy link
Copy Markdown
Contributor Author

Conflicts resolved and pushed.

The merge also brought in the upstream's cleaner confirmed_by_start approach (separate dict, only applied when f.end_line is None) which directly addresses the over-match concern — a static finding with an explicit end_line that differs from the LLM's end_line is now correctly dropped rather than over-matched. This is cleaner than the setdefault fallback in the original PR and implements the "tighter semantics" mentioned in your review.

Updated test_end_line_used_when_provided to use a real end_line=None Finding (the test class's _make_finding helper converts None→line via or, so we construct the Finding directly) and assert the three-way behaviour: exact match confirmed, end_line=None confirmed via confirmed_by_start, different explicit end_line dropped.

880 tests pass, ruff clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(meta-analyzer): LLM-confirmed findings dropped when model returns end_line

2 participants