docs: NeMo TN parity ceiling + exact-parity FST route - #75
Merged
Conversation
Add docs/NEMO_PARITY.md documenting: - current en TN parity (506/566, 89%) with the per-class breakdown - the measured ceiling: NeMo's own normalizer scores only 480/508 (94.5%) on its own test files, so 100% is impossible for any single implementation (the files aggregate deterministic / non-deterministic / punctuation_match_ input / normalize_with_audio modes) - a validated exact-parity route: NeMo's compiled ClassifyFst/VerbalizeFst/ PostProcessingFst loaded via rustfst + a normalize.py driver port, prototyped end-to-end at 488/508 (96.1%), i.e. NeMo-parity - the three non-obvious fidelity fixes (rm_epsilon, custom tropical shortest-path, quote-aware parser) and the size/dependency tradeoff
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the outcome of the NeMo TN parity investigation so future work has the full picture in-repo.
What this documents (
docs/NEMO_PARITY.md)test_cases_*.txtfiles are a union of expected outputs across different NeMo modes (deterministic, non-deterministic, the separatepunctuation_match_inputexact-spacing mode, and the multi-candidatenormalize_with_audio). Confirmed empirically: NeMo's own Python normalizer scores 480/508 (94.5%) on these files — it cannot reproduce them either. So the practical ceiling is ~94–96%.ClassifyFst147K states / 7.7 MB,VerbalizeFst,PostProcessingFst) loaded in Rust viarustfst+ a port of thenormalize.pydriver reaches 488/508 (96.1%) — genuine NeMo parity — prototyped end-to-end.rm_epsilonbefore shortest-path, a custom tropical DAG shortest-path becauserustfst::shortest_pathdiverges from OpenFST on complex compositions, and a quote-aware token parser), and the size/dependency tradeoff (~10 MB grammars +rustfstvs the current tiny pure-Rust crate).Docs-only; no code or behavior change.
🤖 Generated with Claude Code