Skip to content

fix(scoring): apply 1.3x multiplier only to findings from executable files#122

Open
tcconnally wants to merge 1 commit into
NVIDIA:mainfrom
tcconnally:fix/risk-score-per-file-weight
Open

fix(scoring): apply 1.3x multiplier only to findings from executable files#122
tcconnally wants to merge 1 commit into
NVIDIA:mainfrom
tcconnally:fix/risk-score-per-file-weight

Conversation

@tcconnally

Copy link
Copy Markdown
Contributor

Summary

Addresses the scoring concern in #103: the 1.3x executable-script multiplier was applied globally to ALL findings when any executable file existed. This meant documentation, teaching content, deny-lists, and anti-examples in .md files received the same score inflation as actual executable code — making thoroughly documented security skills appear MORE dangerous.

Fix

_compute_risk_score() now accepts optional component_metadata and builds a file-path lookup. The 1.3x multiplier is applied only to findings whose file is marked as executable in the component metadata. Findings from markdown, text, json, yaml, and toml files receive base severity weight (no multiplier).

Before

score = sum(severity_weights)
if has_executable_scripts:
    score = int(score * 1.3)  # inflates ALL findings

After

for each finding:
    weight = severity_weight
    if has_executable_scripts AND finding's file is executable:
        weight = int(weight * 1.3)  # only executable-file findings
    score += weight

Testing

  • 622 tests pass (all existing + new test_report_doc_findings_no_multiplier)
  • Backward compatible: when component_metadata is None, behavior is unchanged

…files

Previously the risk score multiplied ALL findings by 1.3x if any
executable script existed in the skill, punishing documentation
and teaching content that mentions attack patterns for educational
or deny-list purposes.

This change:
- Builds a file-executable lookup from component_metadata
- Applies 1.3x only to findings from executable files (.py, .sh, .js, etc.)
- Leaves non-executable-file findings at base severity weight
- No change when component_metadata is unavailable (backward compatible)

Added test: doc_findings_no_multiplier verifies markdown findings
are not inflated by the executable multiplier.

Addresses the scoring concern in NVIDIA#103

Signed-off-by: Perseus Computing <51974392+tcconnally@users.noreply.github.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — applying the 1.3x multiplier per-finding based on component_metadata executable flags (rather than to the whole score) is a sensible refinement, and the two new tests cover both the executable and the documentation cases.

Minor / optional (non-blocking):

  • Path matching: the multiplier relies on file_executable.get(f.file) matching the path in component_metadata exactly. If findings ever store a path with different normalization than the metadata (absolute vs relative, leading ./, OS separators), the lookup silently misses and the multiplier will not apply. A comment or a normalization step would keep them aligned.
  • Behavior when metadata is None: in that case no finding gets the multiplier even if has_executable_scripts is true — a minor behavior change from the previous whole-score multiply. Fine since report() always passes metadata, but worth noting for any other callers of this private helper.
  • The score math changed from int(sum * 1.3) to sum(int(weight * 1.3)) (per-finding flooring), so totals can differ slightly (e.g. 65 → 64). The test update reflects this; just confirming it is intentional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants