Repository navigation
feat(evaluation): add relative_tolerance to static_json scoring - #649
Merged
ShuxinLin merged 1 commit intoOct 19, 2026
Merged
Conversation
static_json passes only on a strict exact match, so a structured answer
of rounded rates (health-409/410: five per-unit failure rates to one
decimal) fails on any rounding or labelling difference. Its delta-1 and
range checks are reported but never feed the pass decision.
A scenario can now opt in through scenario_meta.json:
{"scoring_method": "static_json", "match": "numeric", "relative_tolerance": 0.1}
Every flattened gold value must be present. A numeric value passes when
|answer - gold| <= relative_tolerance x |gold|; any other value, such as
an asset id, must match exactly. Keys stay positional, so a list must
keep the gold order, and an extra key fails the answer. Comparisons are
inclusive and made on values rounded to 6 decimals, as the fcc and fmea
scorers do. The score is F1 over accepted keys, and the usual
static_json details are kept alongside the per-value tolerance results.
A match other than "numeric", or a missing, negative or non-numeric
relative_tolerance, fails the scenario with a rationale instead of
raising. Scenarios without a match key are scored exactly as before.
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
static_jsonpasses only on a strict exact match. Its delta-1 and range checks are reported but never feed the pass decision. This lets a scenario opt in to a relative tolerance on numeric values throughscenario_meta.json, following thematchvocabulary thefccscorer uses:{"scoring_method": "static_json", "match": "numeric", "relative_tolerance": 0.1}|answer - gold| <= relative_tolerance x |gold|. The check is inclusive and uses values rounded to 6 decimals, as infccandfmea.tolerance_values.matchother thannumeric, or a missing, negative or non-numericrelative_tolerance, fails the scenario with aninvalid_scenario_metarationale instead of raising.matchkey are scored exactly as before.Why
health-409 and health-410 ask for five per-unit failure rates to one decimal. The key comes from the expert failure-mode tag in the excavator dataset, which the agent has to rebuild from the job text. Under exact matching, a reader who agrees with the expert on 98% of jobs still fails (D 122.4 against 125.2), so the scenarios couldn't be passed by a correct method.
The companion scenario change (in the scenario repo) adds
relative_tolerance: 0.1to both scenarios. It also states the rate window in the question: "measured from each unit's first to its last breakdown".Effect on recorded runs
I regraded every recorded 409/410 answer (143) with this branch, loading the scenario files through
load_scenarios:No recorded pass/fail result changes.
Tests
uv run pytest src/evaluation -k "not integration"gives 186 passed. The new tests cover:matchis absent