Repository navigation
Keep scores of differently configured metrics apart in get_metrics - #106
Merged
Merged
Conversation
Metric.name() returned only the class name, so get_metrics() stored F1Score(one_to_one=True) and F1Score(one_to_one=False) under the same key and one score overwrote the other. METRICS_ALL returned 6 of its 9 scores, and which variant survived depended on set iteration order. name() now appends every dataclass field that differs from its default, e.g. "F1Score(one_to_one=False)". Instances with default arguments keep their old names, so METRICS_CORE output is unchanged. PrecisionTopNPercent keeps n in the name and appends only the other fields. get_metrics() computes equal metric instances once and raises ValueError when two different metrics report the same name, instead of silently overwriting a score. Fixes #105 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #106 +/- ##
==========================================
+ Coverage 95.17% 95.24% +0.06%
==========================================
Files 53 53
Lines 2820 2839 +19
Branches 440 446 +6
==========================================
+ Hits 2684 2704 +20
Misses 83 83
+ Partials 53 52 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
On GitHub's macOS runners, torch now warns "Skipping device Apple Paravirtual device that does not support Metal 2.0" when transformers probes for MPS. It appeared between the 2026-10-14 master run and now, alongside torch 2.14.1 -> 2.15.0 and a new macos-26-arm64 runner image. With filterwarnings = ["error"] it failed TestJaccardEmbeddingIntegration on every macOS job. The warning is about the runner's virtual GPU, so ignore that one message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kPsarakis
force-pushed
the
fix/105-metric-name-collisions
branch
from
October 30, 2026 11:22
b759eaa to
c931ee6
Compare
The docs show users defining their own dataclass metrics. Check that instances differing in a required field or a default_factory field get separate keys in get_metrics(), and that instances differing only in a repr=False field, which the name can't show, raise instead of overwriting each other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All Documentation runs shared the "pages" concurrency group with cancel-in-progress, so a push to one PR cancelled the docs build of every other open PR. Give each PR its own group; pushes to master and manual runs, which deploy, still share one group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #105.
Stack: 1 of 3. This PR → #107 (tied scores in 1:1 selection) → #109 (metric input validation and docs).
Problem
Metric.name()returned only the class name, soget_metrics()storedF1Score(one_to_one=True)andF1Score(one_to_one=False)under the same key and one score overwrote the other.METRICS_ALLreturned 6 of its 9 scores. Which variant survived depended on set iteration order: in one run"Precision"held theone_to_one=Falsevalue.Change
Metric.name()appends every dataclass field that differs from its default, e.g."F1Score(one_to_one=False)". Metrics built with default arguments keep their old names ("F1Score","PrecisionTop10Percent"), soMETRICS_COREoutput, the README and the benchmark scripts are unchanged.PrecisionTopNPercentkeepsnin the name and appends only the other fields:"PrecisionTop70Percent(one_to_one=False)".get_metrics()computes equal metric instances once, and raisesValueErrorwhen two different metrics report the same name (e.g. customname()overrides), instead of silently overwriting.I didn't use
str(self)as suggested in the issue, because it would rename every key, including the defaults ("F1Score"→"F1Score(one_to_one=True)").Breaking change
Code that reads a non-default metric by its bare class name now gets a different key:
get_metrics(gt, {F1Score(one_to_one=False)})returns"F1Score(one_to_one=False)"instead of"F1Score". This is recorded with a migration note in the changelog's Unreleased section, so it's better suited to a minor release than a patch.Tests
one_to_one=Falsevariants.METRICS_ALLreturning every score, duplicates computed once, and the name-clash error.docs/metrics.md) get separate keys, and that instances differing only in arepr=Falsefield raise.CI fixes (separate commits)
Both are unrelated to #105, but this PR's CI needed them, and they sit at the bottom of the stack so #107 and #109 get them too.
TestJaccardEmbeddingIntegrationwithUserWarning: Skipping device Apple Paravirtual device that does not support Metal 2.0. torch emits it when transformers probes for MPS on GitHub's macOS runners, andfilterwarnings = ["error"]turns it into a failure. It appeared since the 2026-10-14masterrun, alongside torch 2.14.1 → 2.15.0 and a newmacos-26-arm64image, somasterwould fail the same way now.pyproject.tomlignores that one message.pagesconcurrency group withcancel-in-progress, so a push to one PR cancelled the docs builds of the other open PRs. PR runs now get a group per branch;masterpushes and manual runs, which deploy, still share one.🤖 Generated with Claude Code