Skip to content

fix(analyzer): report result coverage and reject misaligned inputs - #198

Open
plox-sumit wants to merge 1 commit into
agentscope-ai:mainfrom
plox-sumit:fix/analyzer-result-coverage
Open

plox-sumit wants to merge 1 commit into
agentscope-ai:mainfrom
plox-sumit:fix/analyzer-result-coverage

Conversation

@plox-sumit

Copy link
Copy Markdown

Fixes #195.

Analyzers skipped failed grader results without saying so. With three samples where two grading calls failed, DistributionAnalyzer reported a mean of 1.0 and AccuracyAnalyzer reported 100% accuracy, and nothing in the returned object showed that only one result was used. The validation analyzers also paired dataset and grader_results with zip, so a shorter results list was silently truncated instead of raising.

What changed:

  • BaseAnalyzer gets two helpers. _validate_alignment raises ValueError when the two lists have different lengths. _coverage_metadata adds total_samples, valid_results, error_results, skipped_results and coverage_rate to the result metadata, extends the explanation when coverage is below 100%, and logs a warning.
  • The seven validation analyzers (accuracy, precision, recall, F1, false negative, false positive, correlation) call the length check after their existing empty-input guard and report coverage.
  • DistributionAnalyzer reports coverage. Its total_samples now means results submitted, not results used, which is what the name says.
  • PairwiseAnalyzer checks alignment and sets total_comparisons to the number of comparisons it actually analyzed. The submitted count is still available as total_samples.
  • F1ScoreAnalyzer never excluded failed results: a result without predictions counts as an empty prediction set and adds false negatives. That behavior is unchanged. Its valid_results therefore counts every result that entered the calculation, and error_results still tells you how many of those were errors.
  • ConsistencyAnalyzer is untouched. It also pairs with zip, but test_analyze_mismatched_lengths asserts that truncation on purpose, so I left it for you to decide.

Failed results are still excluded from the metric itself, so existing numbers do not change when every call succeeds. The error_policy option suggested in the issue is left out to keep this small; it can be a follow-up if there is interest.

Tests: added to each analyzer's existing test file (plus a new tests/analyzer/test_pairwise_analyzer.py): the example from the issue, a full-coverage run with the explanation unchanged, a length-mismatch case for every validation analyzer and pairwise, the F1 error case, and the pairwise total. All 66 analyzer tests pass, and the full suite passes apart from three pre-existing Windows symlink tests in tests/harness/test_sandbox.py. black, isort, flake8 and pylint pass with the repo's pre-commit config.

🤖 Generated with Claude Code

Analyzers dropped failed grader results without reporting it, so a run
where most grading calls failed could still report a perfect score.
Validation analyzers also paired dataset and results with zip, silently
truncating misaligned inputs.

Add _validate_alignment and _coverage_metadata to BaseAnalyzer, use them
in the validation, distribution and pairwise analyzers, and add tests.

Fixes agentscope-ai#195

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]:Analyzers silently ignore failed or missing grader results and may report misleading metrics

1 participant