Speech synthesis
From text to voice: how current systems sound, what voice cloning means technically, and which labelling duties apply.
The idea
Text is first translated into phonetics and stress, from that a time-frequency image is produced, and from that image a second model generates the sound wave. The split into two stages is why voice, delivery and content can be steered separately.
What it is good for
- Read-aloud for documents and accessibility.
- Announcements and voice dialogues.
- Training material and subtitling.
- Multilingual versions without a new recording.
The structure
- 01
Text normalisation
Spell out numbers, abbreviations and dates. "§ 3 para. 2" must become "section three paragraph two".
- 02
Phonetics
Conversion to phonemes, with a dictionary for names and technical terms.
- 03
Acoustic model
Produces a mel spectrogram with duration and stress.
- 04
Vocoder
Produces the actual waveform from the spectrogram.
The first stage is routinely underestimated and causes most audible errors. An amount like "1,250.50 EUR" has to become "one thousand two hundred and fifty euros fifty", and a wrong rule for that is obvious to every listener.
What drives quality
| Factor | Effect |
|---|---|
| Text normalisation | very large, the most common source of errors |
| Pronunciation dictionary | large for technical text |
| Stress and pauses | large for intelligibility |
| Vocoder quality | medium, usually sufficient today |
| Sample rate | small above 22 kHz |
Voice cloning, technically
Two routes with very different effort:
| Route | Material | Effort | Quality |
|---|---|---|---|
| Speaker embedding | 3 to 30 seconds | no adaptation needed | recognisable, not perfect |
| Fine-tuning | 10 minutes to hours | training required | very close to the original |
In the first, an encoder turns a short sample into a vector describing the voice, and that vector steers the synthesis. It is exactly the idea behind image embeddings, see Embeddings.
Computed: the mel scale and the real-time factor
Speech synthesis almost never produces a waveform directly. It produces a mel spectrogram first. The mel scale maps frequencies the way the ear resolves them: finely at the bottom, coarsely at the top.
At 1,000 Hz this gives 2595 · log₁₀(2.4286) ≈ 1000 mel. At 2,000 Hz it gives
2595 · log₁₀(3.8571) ≈ 1521 mel. Twice the frequency is therefore not twice the
perceived pitch, which is exactly why the computation happens in mel.
A typical mel spectrogram uses 80 bands at a hop of 256 samples and a sample rate
of 22,050 Hz. That yields 22050 / 256 ≈ 86 frames per second. One second of
speech is therefore a matrix of 80 × 86 = 6,880 numbers the model has to
produce.
A model that produces 10 seconds of speech in 0.4 seconds has an RTF of 0.04. For a dialogue system, though, what counts is not RTF but time to first audio. Computing the whole sentence before speaking is noticeably slow; generating and emitting in chunks is not.
Labelling
- Provider duty. Outputs must be marked machine-readably as artificially generated, for instance by a watermark in the signal or metadata.
- Deployer duty. Anyone imitating a voice and publishing it must disclose that. Exceptions exist for obviously artistic or satirical works, with limits.
- Dialogue systems. Anyone speaking to a system must be able to tell it is not a person. That duty applies regardless of voice quality.
- Documentation. Consent of the voice provider, scope of use, revocation route and deletion of the voice data belong in writing.
Watermarking
Methods embedding an inaudible marker in the generated signal exist and are used by several providers. Their limits are known:
- They survive light compression but not every kind of post-processing.
- They prove origin from a particular system, not the absence of manipulation.
- They help with cooperative providers, not with deliberate misuse.
In practice a watermark is a duty-of-care measure, not protection. Organisational safeguards such as call-back procedures for payment instructions are more effective against voice fraud than any technical detection.
Related courses and sources
Hugging Face audio course
From the signal through spectrograms to recognition and synthesis, with code throughout. The practical companion to the speech and audio articles.
For anyone building speech technology themselves; assumes Python and some signal knowledge.