Skip to content

feat(evaluation): add relative_tolerance to static_json scoring - #649

Merged
ShuxinLin merged 1 commit into
aafeedback_changesfrom
feat/static-json-relative-tolerance
Oct 19, 2026
Merged

ShuxinLin merged 1 commit into
aafeedback_changesfrom
feat/static-json-relative-tolerance

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Summary

static_json passes only on a strict exact match. Its delta-1 and range checks are reported but never feed the pass decision. This lets a scenario opt in to a relative tolerance on numeric values through scenario_meta.json, following the match vocabulary the fcc scorer uses:

{"scoring_method": "static_json", "match": "numeric", "relative_tolerance": 0.1}
  • Every flattened gold value must be present, and an extra key fails the answer.
  • A number passes when |answer - gold| <= relative_tolerance x |gold|. The check is inclusive and uses values rounded to 6 decimals, as in fcc and fmea.
  • Any other value, such as an asset id, must match exactly. Keys stay positional, so a list must keep the gold order.
  • The score is F1 over accepted keys. The existing static_json details are kept, and the per-value results are added under tolerance_values.
  • A match other than numeric, or a missing, negative or non-numeric relative_tolerance, fails the scenario with an invalid_scenario_meta rationale instead of raising.
  • Scenarios without a match key are scored exactly as before.

Why

health-409 and health-410 ask for five per-unit failure rates to one decimal. The key comes from the expert failure-mode tag in the excavator dataset, which the agent has to rebuild from the job text. Under exact matching, a reader who agrees with the expert on 98% of jobs still fails (D 122.4 against 125.2), so the scenarios couldn't be passed by a correct method.

The companion scenario change (in the scenario repo) adds relative_tolerance: 0.1 to both scenarios. It also states the rate window in the question: "measured from each unit's first to its last breakdown".

Effect on recorded runs

I regraded every recorded 409/410 answer (143) with this branch, loading the scenario files through load_scenarios:

Answer 409 410
Recorded runs (every job counted, 2–3x the key) 0/104 pass 0/39 pass
Expert labels, stated window pass pass
Expert labels, n−1 convention pass pass
Expert labels plus jump starts, or plus tooth/lip replacements fail fail
5% of jobs misread at random 88% pass 88% pass

No recorded pass/fail result changes.

Tests

uv run pytest src/evaluation -k "not integration" gives 186 passed. The new tests cover:

  • inclusive boundaries
  • the every-job answer failing
  • order and asset ids held exact
  • an extra key failing
  • a fenced answer
  • five invalid metadata shapes
  • strict matching when match is absent

static_json passes only on a strict exact match, so a structured answer
of rounded rates (health-409/410: five per-unit failure rates to one
decimal) fails on any rounding or labelling difference. Its delta-1 and
range checks are reported but never feed the pass decision.

A scenario can now opt in through scenario_meta.json:

    {"scoring_method": "static_json", "match": "numeric", "relative_tolerance": 0.1}

Every flattened gold value must be present. A numeric value passes when
|answer - gold| <= relative_tolerance x |gold|; any other value, such as
an asset id, must match exactly. Keys stay positional, so a list must
keep the gold order, and an extra key fails the answer. Comparisons are
inclusive and made on values rounded to 6 decimals, as the fcc and fmea
scorers do. The score is F1 over accepted keys, and the usual
static_json details are kept alongside the per-value tolerance results.

A match other than "numeric", or a missing, negative or non-numeric
relative_tolerance, fails the scenario with a rationale instead of
raising. Scenarios without a match key are scored exactly as before.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin
ShuxinLin merged commit f906f25 into aafeedback_changes Oct 19, 2026
5 checks passed
@ShuxinLin
ShuxinLin deleted the feat/static-json-relative-tolerance branch October 19, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant