Skip to content

fix(metrics): compute TEDS over the actual ordered tree - #77

Closed
lanyue-llk wants to merge 2 commits into
opendatalab:mainfrom
lanyue-llk:codex/fix-teds-ordered-tree
Closed

lanyue-llk wants to merge 2 commits into
opendatalab:mainfrom
lanyue-llk:codex/fix-teds-ordered-tree

Conversation

@lanyue-llk

Copy link
Copy Markdown

Root cause

TEDSMetric._tree_edit_distance serializes each parsed table tree into a string and passes that string directly to APTED. The default Config.children accessor does not parse the serialization, so APTED sees each whole table as one leaf. Nested row and cell differences are therefore not evaluated.

The exception fallback also returns only the absolute node count difference. Equal-size trees with different structure can consequently receive an optimistic distance of zero.

Fix

  • pass the existing parsed node dictionaries directly to APTED;
  • teach TableConfig.children how to traverse their children arrays;
  • keep the existing tag and normalized text rename costs;
  • remove the node-count fallback so calculation errors are surfaced by the metric's existing error result;
  • add regressions for a missing child and changed cell text, and update expectations that previously encoded the leaf-only result.

Verification

  • tests/test_teds.py: 24 tests passed.
  • The package's top-level import eagerly requires optional torch; the test file was run in the locked browser-eval environment with the metric package loaded directly, so the TEDS suite itself ran without installing the unrelated LLM extractor dependency.

This changes scorer output. Existing published benchmark evidence should retain its pinned scorer commit and original values; future benchmark profiles can pin the fixed commit explicitly.

@lanyue-llk

Copy link
Copy Markdown
Author

Closing this upstream submission. The correction is maintained in the lexmount fork at lexmount#1.

@lanyue-llk lanyue-llk closed this Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant