[Feature] `--no-llm` mode bypasses meta-analyzer entirely, presenting unfiltered raw findings as final results

[mimran-khan]

Summary

Think of a spam filter that only works when connected to the internet. Offline, every email goes straight to your inbox — including the obvious spam that any simple rule could catch (like "email from xyz-prince-money@scam.com"). You'd expect the filter to still block the obvious stuff locally, even if it can't do the sophisticated AI analysis.

SkillSpector has the same problem. Its --no-llm mode (which is the default when no LLM provider is configured) completely skips the meta-analyzer — the component that filters out false positives. With no LLM, raw regex matches go directly into the final report with zero post-processing. The same skill that scores 35/100 with LLM analysis scores 100/100 without it, because the meta-analyzer would have correctly removed 70%+ of findings as documentation examples or non-exploitable patterns.

The issue isn't that --no-llm can't be as good as LLM-enabled mode — of course it can't. The issue is that it applies zero heuristic filtering, not even the basic checks that already exist in the codebase (like is_code_example()). Users who evaluate SkillSpector without an LLM configured — which is likely most first-time users and CI/CD pipelines — see wildly inflated, noisy results and conclude the tool is unreliable.

Reproduction

Create a skill with a mix of genuine issues and documentation:

mixed-skill/
├── SKILL.md
├── tool.py
└── docs/
    └── examples.md

tool.py:

import subprocess
subprocess.run(["deploy", "--env", "prod"])  # Genuine: executable code calling subprocess

docs/examples.md:

# Examples

```bash
curl -k https://api.example.com/health  # Documentation: code block example
rm -rf /tmp/cache                        # Documentation: cleanup example
eval "$DEPLOY_CMD"                       # Documentation: showing a pattern
```

# With LLM — meta-analyzer filters documentation findings
skillspector scan ./mixed-skill/ --verbose
# Raw findings: ~8, after meta-analysis: ~2-3 (documentation examples removed)

# Without LLM — no filtering at all
skillspector scan ./mixed-skill/ --no-llm
# Findings: ~8 (ALL raw pattern matches reported, including documentation)

The meta-analyzer (when active) correctly identifies that findings in fenced code blocks within docs/examples.md are not exploitable. Without it, those same findings are reported as genuine vulnerabilities.

Root Cause

The LangGraph workflow routes based on whether an LLM provider is configured:

def should_run_meta_analyzer(state: SkillspectorState) -> str:
    if state.get("llm_client") is None:
        return "report"  # Skip meta-analyzer entirely
    return "meta_analyzer"

In --no-llm mode, findings go straight from static analyzers to report generation. There is no heuristic-based fallback filter to:

• Apply the is_code_example() check retrospectively to all findings
• Remove findings from documentation-only files
• Apply confidence thresholds (e.g., drop findings below 0.5)
• Deduplicate obvious repeated patterns

Impact

• --no-llm is the effective default when no LLM provider is configured — the likely case for CI/CD pipelines and first-time evaluators
• Users without LLM access see dramatically worse results with no way to reduce false positives
• Makes the tool appear unreliable/noisy when evaluated without LLM first (bad first impression)
• Inconsistent scoring: Same skill can score 100 without LLM but 35 with LLM, with no indication of why

Suggested Fix

1. Heuristic fallback filter for --no-llm mode:

   • Apply is_code_example() to ALL findings (not just the 2 analyzers that currently use it)
   • Apply confidence threshold filtering (drop findings below 0.5)
   • Reduce weight for findings in non-executable file types (.md outside of SKILL.md)
   • Deduplicate same-rule same-file findings

2. Report transparency: When --no-llm is active, clearly indicate in the output that results are unfiltered static analysis and recommend LLM-enabled scanning for production decisions.

3. JSON output metadata: Include "meta_analysis_applied": false in the output so consumers know results are unfiltered.

4. --confidence-threshold flag: Allow users to set a minimum confidence threshold even in --no-llm mode, giving some control over noise without requiring LLM.

Affected Version

SkillSpector v2.2.3