How we measure accuracy

The apostrophe fault: two errors for a word the model got right

An early run of this benchmark put French at 19.31% on MLS while every other language sat near 10%. An outlier that large is normally blamed on the corpus, and we blamed it on the corpus. It was not the corpus. It was the word-error-rate normaliser, and the fault is the kind that survives review because it looks like tidying up.

The WER normaliser stripped punctuation with a regular expression that deleted apostrophes rather than separating the tokens around them. French runs on elision — d’, l’, qu’, c’ — and the reference corpora write those as two tokens. Recognition output writes them joined. Delete the apostrophe and the hypothesis collapses to one token against the reference’s two, so every elision scored a deletion and a substitution: two errors for a word the model got right.

Replacing the apostrophe with a space took mls_fr from19.31% to 9.66%. It was never French-only — English contractions were mis-scored identically, just less often. Our macro average across twelve fixtures moved from 10.57% to 9.31%.

If you compute WER on French, Italian or any elision-heavy language, check what your normaliser does with apostrophes before you trust the number. It is one substitution and it moved one of our fixtures by 9.65 points.

What was held constant

One batch pipeline, one segmentation strategy, one decoder setting, the same audio and the same reference texts throughout. Corpora were VoxPopuli (parliamentary speech) and MLS (audiobook narration), plus one 141-second English clip.

The point of holding everything else constant is that published comparisons usually do not. They vary the decoder settings, the segmentation and the audio all at once — which is why their numbers disagree with each other, and with anyone’s production system.

What is typing’s word error rate?

Nobody asks. Speech recognition is the only text-input method that gets audited, and the number above is ours. The keyboard has never been asked for one — but it has been measured, on a scale this benchmark cannot match.

The larger dataset agrees on the direction. Across 136 million keystrokes, participants pressed correction keys on about 6.3% of them — one in sixteen.^[2] Counting slips that get overwritten before anyone notices, a first-pass mistype rate of seven to eight per cent is a fair reading of that data. If those slips fell independently of each other, which they do not exactly, seven per cent per keystroke puts an initial mistake in roughly one five-letter word in three.

Set beside that, our 9.31% is not the outlier it is usually treated as. It is simply the one of the two that somebody was asked for.

The units are not the same and the difference matters: those are errors per keystroke, ours are errors per word. What makes the comparison fair is not the arithmetic but the question it answers — not which method is accurate, but who does the correcting, and when.

Typing corrects continuously and invisibly. The backspaces are inside the act, so the cost is real and never counted, and the finished text looks clean. Dictation’s errors survive into the text, where you can see all of them at once and fix them at reading speed rather than at typing speed.

And a word error rate does not measure the part dictation does for you. Capitalisation, full stops, commas, quotation marks and sentence breaks arrive already applied — the things a typist skips under time pressure and repairs afterwards. None of it appears in any WER figure, which means the measurement understates the difference rather than flattering it.

Both methods make mistakes. Only one of them makes you fix them by hand while you are still thinking, and only one of them hands back text that is already punctuated.