Skip to content

fix(schemas): normalize confidence from 0-100 scale before Pydantic validation#93

Merged
rng1995 merged 2 commits into
NVIDIA:mainfrom
Shrotriya-lalit:fix/issue-89-confidence-normalization
Jun 23, 2026
Merged

fix(schemas): normalize confidence from 0-100 scale before Pydantic validation#93
rng1995 merged 2 commits into
NVIDIA:mainfrom
Shrotriya-lalit:fix/issue-89-confidence-normalization

Conversation

@Shrotriya-lalit

Copy link
Copy Markdown
Contributor

Problem

LLMFinding and MetaAnalyzerFinding both declare confidence: float = Field(ge=0.0, le=1.0). When a local model (e.g. Ollama-hosted Llama or Mistral) returns confidence as an integer on a 0–100 scale ("confidence": 85), Pydantic raises a ValidationError at the le=1.0 boundary and crashes the meta-analyzer for the entire file — all findings for that file are silently dropped.

Root Cause

The constraint is evaluated after type coercion but before any user logic, so there is no opportunity to rescale at the model level.

Fix

Add a mode="before" @field_validator on confidence in both schemas:

@field_validator("confidence", mode="before")
@classmethod
def _normalize_confidence(cls, v: object) -> float:
    v = float(v)          # raises on non-numeric
    if v > 1.0:
        v = v / 100.0     # rescale 0-100 → 0-1
    return max(0.0, min(1.0, v))

This also clamps negative values and values above 100 (e.g. 110) gracefully instead of crashing.

Tests

  • Confidence 0.85 (already valid) passes through unchanged
  • Confidence 85 (0-100 integer) is rescaled to 0.85
  • Confidence 1001.0, 00.0
  • Confidence -50.0, 1101.0
  • Non-numeric input raises ValueError
  • Both LLMFinding and MetaAnalyzerFinding validated

Closes #89

Checklist

  • make lint passes
  • Tests added for new behaviour
  • DCO sign-off on commit (git commit -s)

…alidation

LLMFinding and MetaAnalyzerFinding both hard-fail with le=1.0 when Ollama
(or other local models) return confidence as an integer on a 0-100 scale.
Add a mode="before" field_validator that: converts to float, divides by 100
if the value exceeds 1.0, then clamps to [0.0, 1.0].  This also handles
negative values and values above 100 gracefully rather than crashing the
meta-analyzer for the entire file.

Closes NVIDIA#89

Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good defensive fix — a single out-of-range confidence shouldn't crash the meta-analyzer and drop a whole file's findings. Approving.

The mode="before" validator runs prior to the ge/le Field constraint, so rescaling 0–100 values (e.g. 85 -> 0.85, 100 -> 1.0) and clamping negatives works correctly, applied symmetrically to both LLMFinding and MetaAnalyzerFinding. float(v) cleanly rejects non-numeric input, and the tests cover the rescale/clamp/non-numeric paths.

Minor / optional (non-blocking): values just above 1 are ambiguous — 1.5 becomes 0.015 (divided by 100) rather than clamped toward 1.0. That's an inherently ambiguous input and the 0–100 assumption is documented, so this is fine; just flagging that a model emitting 1.5 on a 0–1 scale would be under-counted rather than rejected.

@rng1995

rng1995 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

@Shrotriya-lalit - Please resolve the conflicts, minor issues and merge the PR.

… confidence rescaling

Upstream main added _clamp_start_line (LLMFinding) and a simple
_clamp_confidence on both schemas while removing ge/le Field bounds for
JSON-schema compatibility with OpenAI-compatible endpoints.

Merge resolution keeps all three upstream changes and replaces the simple
clamp with the mode="before" _normalize_confidence validator from this
branch: it both rescales 0-100 integer values (e.g. 85 → 0.85) and clamps
out-of-range floats, which the clamp-only approach cannot do.

Update test_llm_finding_clamps_confidence to match the rescaling semantics.

Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>
@Shrotriya-lalit

Copy link
Copy Markdown
Contributor Author

Thanks for the review @rng1995!

Conflicts have been resolved — merged origin/main into the branch and combined both changes:

  • Kept upstream's _clamp_start_line and the removal of ge/le Field bounds (for OpenAI-compatible endpoint compatibility)
  • Replaced upstream's _clamp_confidence with the mode="before" _normalize_confidence from this PR, which handles both the 0–100 rescaling and clamping in one pass

Regarding the 1.5 ambiguity you flagged: agreed it's an inherent edge case of the threshold approach. A model emitting 1.5 on a 0–1 scale is operating outside spec; the rescaling assumption is documented in the validator docstring. If this becomes a problem in practice a two-pass approach (try clamping first, rescale only when the value looks like an integer percentage) could be added as a follow-up.

727 tests pass, lint clean. Ready to merge.

@rng1995 rng1995 merged commit cc7dd87 into NVIDIA:main Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM stage crashes on OpenAI-compatible/local endpoints (Ollama): models return confidence 0-100 but output schema validates 0-1

2 participants