How we measure accuracy
Word error rate is the industry’s accuracy number, and it is easier to get wrong than to get right. One regular expression in our own scoring moved a French fixture by 9.65 points — and the same fault is present in a good deal of published ASR evaluation. Here is the method we use, the twelve fixtures we run it on, and every number it produced.
The apostrophe fault: two errors for a word the model got right
An early run of this benchmark put French at 19.31% on MLS while every other language sat near 10%. An outlier that large is normally blamed on the corpus, and we blamed it on the corpus. It was not the corpus. It was the word-error-rate normaliser, and the same fault is present in a good deal of published ASR scoring.
The WER normaliser stripped punctuation with a regular expression that deleted apostrophes rather than separating the tokens around them. French runs on elision — d’, l’, qu’, c’ — and the reference corpora write those as two tokens. Recognition output writes them joined. Delete the apostrophe and the hypothesis collapses to one token against the reference’s two, so every elision scored a deletion and a substitution: two errors for a word the model got right.
Replacing the apostrophe with a space took mls_fr from 19.31% to 9.66%. It was never French-only — English contractions were mis-scored identically, just less often. Our macro average across twelve fixtures moved from 10.57% to 9.31%.
If you compute WER on French, Italian or any elision-heavy language, check what your normaliser does with apostrophes before you trust the number. It is a two-character fix and it moved one of our fixtures by nine points.
What was held constant
One batch pipeline, one segmentation strategy, one decoder setting, the same audio and the same reference texts throughout. Corpora were VoxPopuli (parliamentary speech) and MLS (audiobook narration), plus one 141-second English clip.
The point of holding everything else constant is that published comparisons usually do not. They vary the decoder settings, the segmentation and the audio all at once — which is why their numbers disagree with each other, and with anyone's production system.
Caveats
- Twelve fixtures. Single-clip differences under roughly 1pp are not distinguishable from noise at this corpus size.
- One run per configuration. Run-to-run variance is unmeasured.
voxpopuli_fris excluded from the macro average. Recognition returns words the reference does not contain, and two independent runs agree with each other roughly twice as well as either agrees with the corpus — which is consistent with an edited parliamentary record rather than a verbatim transcript. Flagged rather than quietly dropped.- Numbers are spelled out in the references and returned as digits by recognition, worth about three errors per number. Not fixed, quantified.
- Batch pipeline only. Streaming segments on VAD in real time and was not measured here.
Common questions
What exactly does a WER normaliser have to do?+
Lower-case, expand or standardise numbers, strip punctuation that carries no lexical meaning — and, critically, treat an apostrophe as a token boundary rather than a character to delete. Deleting it merges two reference tokens into one hypothesis token, which scores as a deletion plus a substitution. Pin the elision cases with unit tests; they are the ones that silently drift.
Why is your French number so high?+
On MLS it is not, any more: 9.66%, in line with the other languages, after fixing a normaliser bug that deleted apostrophes and charged two errors for every French elision. On VoxPopuli it remains near 30% for both models, and we believe that reference is not a verbatim transcript of the audio. We have flagged it rather than quietly dropping it.
Can I reproduce this?+
Most of it, from this page alone. Both corpora are public — VoxPopuli and Multilingual LibriSpeech — the fixture list and the settings are stated above, and the normaliser fix is one substitution: replace the apostrophe with a space instead of deleting it, then check your elision cases against a reference. Our harness is not open source; if you are reproducing this seriously, write and tell us what you get.
Does a word error rate tell me which product to buy?+
Less than you would hope. WER measures one pipeline on one corpus. What a product actually feels like depends at least as much on segmentation, on how it handles the pauses in real speech, on post-processing and on latency — none of which a single accuracy figure captures. Treat a published WER as a floor on plausibility, not as a ranking.
We run this measurement against every change to the transcription pipeline, in six languages, before it reaches anyone’s audio. It is the reason we can put numbers on a page at all.
What VibeVoice isRelated reading
Last reviewed . Competitor pricing and platform support change; check theirs before buying.