LLM-Based ASR Error Correction

Post-editing raw ASR transcripts with an LLM to fix recognition errors, without re-transcribing.

Approach

Framing. Measuring raw against ref on the dev set shows the "errors" fall into three classes, and only the first is what we should own:

Class	Examples	Stance
Genuine ASR errors	Sharks→Shocks, Wiens share→Wythenshawe, cob life→Co-op Live, Ana Sarwar→Anas Sarwar	fix
Formatting	sentence punctuation, capitalisation	fix lightly
Reference style	numbers spelled out (23.5 thousand→23 and a half thousand), disfluency rewrites	leave alone

A meaningful share of the WER is punctuation/casing rather than word substitutions (word WER 0.113 vs word+punct 0.153 on dev), and chasing the reference's stylistic conventions mostly inflates over-correction. So the system is deliberately conservative: fix clear word errors + light punctuation, do not paraphrase or reformat.

Pipeline (asrec/correct.py):

1. Chunk long transcripts (up to ~8k words) on paragraph/sentence boundaries (asrec/chunk.py). Chunks concatenate back to the exact original, so document structure is preserved.
2. Single conservative LLM pass per chunk (Claude, default claude-sonnet-5) with a prompt that enumerates what to fix and — more importantly — what not to touch (no paraphrasing, no number spelling, no dialect changes).
3. The tail of the previous corrected chunk is passed as read-only context so proper-noun spellings stay consistent across chunk boundaries.

Evaluation — beyond aggregate WER

The evaluator (asrec/metrics.py) aligns both raw and hyp to ref with an optimal edit-distance alignment (Levenshtein via rapidfuzz; stdlib difflib as a fallback) and anchors every metric on reference token positions. For each ref token we know whether the raw ASR had it right and whether our output has it right, which separates fixing errors from breaking correct tokens:

• WER before/after, in two views — words (lowercase, no punctuation) and words+punct — so the source of any gain is explicit.
• Correction recall — of ref tokens the ASR got wrong, the fraction we fixed.
• Over-correction rate — of ref tokens the ASR got right, the fraction we broke. This is the key guardrail metric.
• Net tokens improved — fixed − broken.

Results (claude-sonnet-5, dev, 20 docs — reproduce with eval --hyp outputs/dev):

view	WER raw→hyp	WERR	corr. recall	over-corr.	net tokens
words	0.113 → 0.104	+8.1%	0.110	0.0033	+395
words+punct	0.153 → 0.144	+6.1%	0.091	0.0040	+496

19/20 documents improve; the one mild regression (72540297, net −39, WER 0.116→0.119) is a doc where the model over-reaches on garbled content. (Numbers vary ~±1pp between runs — the LLM is non-deterministic, and a length-drift rollback can leave one chunk uncorrected.)

Two iterations got here, both metric-driven:

1. Prompt. The first prompt over-corrected — flipping en-US→en-GB spelling (center→centre, though the ref keeps center) and confidently rewriting badly-garbled proper nouns/quotes toward real-world-correct forms that differ from the reference — which regressed entity-dense documents. Hardening the prompt (never change dialect spelling, don't guess heavily-garbled names, never delete/reorder words) plus a content-preservation guard (roll a chunk back to raw on large length drift) removed those regressions and cut over-correction to 0.33%.

2. Metric. The first evaluator used difflib, a greedy matcher that over-counts edits: it scored 72540297 at WER 0.66 and manufactured phantom "deletions" (137 for a 12-word net change) that inflated broken. Switching to an optimal Levenshtein alignment corrected this — that same doc is really WER 0.116, and the true corpus baseline is 0.113 (not 0.198). The numbers above are on the optimal metric.

Still open: fixed/broken come from two independent optimal alignments to ref, so a single joint three-way alignment (or aligning raw↔hyp directly) would be the cleaner fidelity fix — though the headline WER delta already comes from one reliable alignment.

Production without ground truth

In production there is no ref. The over-correction guardrail above becomes a set of reference-free monitors:

• Edit rate raw↔corrected: flag docs where the model changed far more than the dev-calibrated distribution — the signature of over-correction/hallucination.
• Semantic drift: embedding similarity between raw and corrected; low similarity means meaning moved, which correction should never do.
• Fluency delta: external LM perplexity of corrected vs raw (should improve modestly, not spike).
• Entity consistency: the same proper noun should be spelled the same way throughout a document and across a speaker/series.
• Human-in-the-loop sampling + canary A/B: score a small sampled slice, and watch the distribution of edit types/volume over time to catch regressions (e.g. after a model or prompt change) as distribution drift.

Usage

python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env                    # put ANTHROPIC_API_KEY here (loaded automatically)
# or: export ANTHROPIC_API_KEY=...      # only `run` needs a key, `eval` does not

# baseline: raw ASR vs reference
python -m asrec.cli eval --split dev

# correct dev, then score the corrected outputs
python -m asrec.cli run  --split dev  --out outputs/dev
python -m asrec.cli eval --split dev  --hyp outputs/dev

# qualitative view of one document — the exact tokens fixed and broken
python -m asrec.cli inspect --id 69930104 --hyp outputs/dev

# produce the test-set deliverable (same per-file layout as input)
python -m asrec.cli run  --split test --out outputs/test

pip install -e . also exposes an asrec console command (e.g. asrec eval --split dev). run --glossary enables an optional agentic first pass that extracts a document-wide proper-noun glossary and feeds it to every chunk, so entity spellings stay consistent across chunk boundaries (costs one extra call per document).

eval writes a full per-document eval_report.json. run requires the anthropic SDK and an API key; eval needs neither (optimal alignment via rapidfuzz, or a stdlib difflib fallback).

Tests:

pip install -r requirements-dev.txt
pytest            # chunker reconstruction, metric decomposition, guard, tokenizer