fix(analyzer): report result coverage and reject misaligned inputs - #198
Open
plox-sumit wants to merge 1 commit into
Open
plox-sumit wants to merge 1 commit into
plox-sumit wants to merge 1 commit into
Conversation
Analyzers dropped failed grader results without reporting it, so a run where most grading calls failed could still report a perfect score. Validation analyzers also paired dataset and results with zip, silently truncating misaligned inputs. Add _validate_alignment and _coverage_metadata to BaseAnalyzer, use them in the validation, distribution and pairwise analyzers, and add tests. Fixes agentscope-ai#195 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #195.
Analyzers skipped failed grader results without saying so. With three samples where two grading calls failed,
DistributionAnalyzerreported a mean of 1.0 andAccuracyAnalyzerreported 100% accuracy, and nothing in the returned object showed that only one result was used. The validation analyzers also paireddatasetandgrader_resultswithzip, so a shorter results list was silently truncated instead of raising.What changed:
BaseAnalyzergets two helpers._validate_alignmentraisesValueErrorwhen the two lists have different lengths._coverage_metadataaddstotal_samples,valid_results,error_results,skipped_resultsandcoverage_rateto the result metadata, extends the explanation when coverage is below 100%, and logs a warning.DistributionAnalyzerreports coverage. Itstotal_samplesnow means results submitted, not results used, which is what the name says.PairwiseAnalyzerchecks alignment and setstotal_comparisonsto the number of comparisons it actually analyzed. The submitted count is still available astotal_samples.F1ScoreAnalyzernever excluded failed results: a result without predictions counts as an empty prediction set and adds false negatives. That behavior is unchanged. Itsvalid_resultstherefore counts every result that entered the calculation, anderror_resultsstill tells you how many of those were errors.ConsistencyAnalyzeris untouched. It also pairs withzip, buttest_analyze_mismatched_lengthsasserts that truncation on purpose, so I left it for you to decide.Failed results are still excluded from the metric itself, so existing numbers do not change when every call succeeds. The
error_policyoption suggested in the issue is left out to keep this small; it can be a follow-up if there is interest.Tests: added to each analyzer's existing test file (plus a new
tests/analyzer/test_pairwise_analyzer.py): the example from the issue, a full-coverage run with the explanation unchanged, a length-mismatch case for every validation analyzer and pairwise, the F1 error case, and the pairwise total. All 66 analyzer tests pass, and the full suite passes apart from three pre-existing Windows symlink tests intests/harness/test_sandbox.py. black, isort, flake8 and pylint pass with the repo's pre-commit config.🤖 Generated with Claude Code