Speech recognition
From sound wave to text: feature extraction, current models, word error rate, and what may go into a transcript.
The idea
A microphone delivers a sequence of numbers: air pressure, measured sixteen thousand times a second. A model can do little with that. So it is first turned into a picture: time on one axis, pitch on the other, loudness as brightness.
A model then works on that picture, much as in image recognition.
What determines accuracy
| Factor | Effect |
|---|---|
| Microphone distance | very large |
| Background noise | very large |
| Room reverberation | large |
| Several people speaking at once | large |
| Dialect and accent | large |
| Technical terms and names | medium, fixable with a dictionary |
| Model choice | medium |
The ordering is notable: a better microphone almost always beats a better model.
The chain
- 01
Preprocess
Resample to 16 kHz, convert to mono, normalise the level.
- 02
Extract features
Short-time Fourier transform, then conversion to the mel scale.
- 03
Recognise
An encoder-decoder model produces text, usually in 30-second segments.
- 04
Post-process
Punctuation, capitalisation, spelling out numbers, correcting terminology.
- Supply a dictionary of names and technical terms as context where the model supports it.
- Cut long recordings at speech pauses, not at fixed time marks.
- Record per-word timestamps; they are indispensable for later review.
- Store per-segment confidence and route low values to review.
The metric
In practice raw WER is often misleading. A transcript at 5 percent WER can be useless if the errors fall on numbers and names. An entity error rate is therefore collected alongside: the share of misrecognised names, amounts and dates.
Mel spectrogram
Usual parameters: 25 ms window, 10 ms hop, 80 mel bands. That yields 100 feature vectors per second: a data reduction of about a factor of 160 against the 16 kHz waveform.
The data protection frame
A voice recording contains more than words: a voice is a biometric attribute as soon as it is used for identification.
- The recording, the transcript and any speaker identification are three separate processing operations, each with its own legal basis.
- Everyone must be informed before the recording, not after.
- In employment, works council rights are routinely engaged.
- Emotion recognition from the voice in the workplace is prohibited under the AI Act.
- Set retention periods for recording and transcript separately; the recording can usually be deleted much earlier.
See GDPR and AI and Risk classes.
Related courses and sources
Hugging Face audio course
From the signal through spectrograms to recognition and synthesis, with code throughout. The practical companion to the speech and audio articles.
For anyone building speech technology themselves; assumes Python and some signal knowledge.
Robust Speech Recognition
Speech recognition that copes with noise, accents and language switching. The benchmark dictation solutions are measured against.
For anyone introducing dictation or transcription who needs a benchmark.
Speech and Language Processing
Jurafsky and Martin, the standard work on language processing, free chapter by chapter. Covers classical methods and language models in one arc.
For anyone learning language processing systematically, classical and modern in one arc.