fix(schemas): normalize confidence from 0-100 scale before Pydantic validation#93
Conversation
…alidation LLMFinding and MetaAnalyzerFinding both hard-fail with le=1.0 when Ollama (or other local models) return confidence as an integer on a 0-100 scale. Add a mode="before" field_validator that: converts to float, divides by 100 if the value exceeds 1.0, then clamps to [0.0, 1.0]. This also handles negative values and values above 100 gracefully rather than crashing the meta-analyzer for the entire file. Closes NVIDIA#89 Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>
rng1995
left a comment
There was a problem hiding this comment.
Good defensive fix — a single out-of-range confidence shouldn't crash the meta-analyzer and drop a whole file's findings. Approving.
The mode="before" validator runs prior to the ge/le Field constraint, so rescaling 0–100 values (e.g. 85 -> 0.85, 100 -> 1.0) and clamping negatives works correctly, applied symmetrically to both LLMFinding and MetaAnalyzerFinding. float(v) cleanly rejects non-numeric input, and the tests cover the rescale/clamp/non-numeric paths.
Minor / optional (non-blocking): values just above 1 are ambiguous — 1.5 becomes 0.015 (divided by 100) rather than clamped toward 1.0. That's an inherently ambiguous input and the 0–100 assumption is documented, so this is fine; just flagging that a model emitting 1.5 on a 0–1 scale would be under-counted rather than rejected.
|
@Shrotriya-lalit - Please resolve the conflicts, minor issues and merge the PR. |
… confidence rescaling Upstream main added _clamp_start_line (LLMFinding) and a simple _clamp_confidence on both schemas while removing ge/le Field bounds for JSON-schema compatibility with OpenAI-compatible endpoints. Merge resolution keeps all three upstream changes and replaces the simple clamp with the mode="before" _normalize_confidence validator from this branch: it both rescales 0-100 integer values (e.g. 85 → 0.85) and clamps out-of-range floats, which the clamp-only approach cannot do. Update test_llm_finding_clamps_confidence to match the rescaling semantics. Signed-off-by: Lalit Shrotriya <shrotriya.lalit@outlook.com>
|
Thanks for the review @rng1995! Conflicts have been resolved — merged
Regarding the 727 tests pass, lint clean. Ready to merge. |
Problem
LLMFindingandMetaAnalyzerFindingboth declareconfidence: float = Field(ge=0.0, le=1.0). When a local model (e.g. Ollama-hosted Llama or Mistral) returns confidence as an integer on a 0–100 scale ("confidence": 85), Pydantic raises aValidationErrorat thele=1.0boundary and crashes the meta-analyzer for the entire file — all findings for that file are silently dropped.Root Cause
The constraint is evaluated after type coercion but before any user logic, so there is no opportunity to rescale at the model level.
Fix
Add a
mode="before"@field_validatoronconfidencein both schemas:This also clamps negative values and values above 100 (e.g.
110) gracefully instead of crashing.Tests
0.85(already valid) passes through unchanged85(0-100 integer) is rescaled to0.85100→1.0,0→0.0-5→0.0,110→1.0ValueErrorLLMFindingandMetaAnalyzerFindingvalidatedCloses #89
Checklist
make lintpassesgit commit -s)