Audio as a signal

The idea#

A sound is a vibration. A microphone measures it many thousand times a second. From those measurements you can compute which pitches are present, and from that comes an image a model can process.

The basic quantities#

Computing a spectrogram#

import numpy as np

def spectrogram(signal, sr=16000, window_ms=25, hop_ms=10):
    n_win = int(sr * window_ms / 1000)     # 400 samples
    n_hop = int(sr * hop_ms / 1000)        # 160 samples
    window = np.hanning(n_win)             # without a window function
                                           # edge artefacts appear
    frames = [signal[i:i + n_win] * window
              for i in range(0, len(signal) - n_win, n_hop)]
    spec = np.abs(np.fft.rfft(np.array(frames), axis=1))
    return 20 * np.log10(spec + 1e-10)     # in decibels

sr = 16000
t = np.arange(sr) / sr
# Two superimposed tones, 440 Hz and 880 Hz
signal = np.sin(2*np.pi*440*t) + 0.5*np.sin(2*np.pi*880*t)
S = spectrogram(signal, sr)
print(S.shape)     # (98, 201): 98 time frames, 201 frequency bands

The output shape shows the data reduction: 16,000 samples per second become 98 by 201 values, about an eighth of the volume, in a form far more usable for models.

The sampling theorem#

At a 16 kHz sample rate, frequencies up to 8 kHz are representable. Speech carries its intelligibility overwhelmingly below 4 kHz, which is why even telephone quality at 8 kHz works. Sibilants sit higher, and they are what suffers first at a low sample rate.

The trade-off#

Worked through: a 25 ms window gives a frequency resolution of about 40 Hz. To separate two tones 10 Hz apart you would need a 100 ms window, and would then no longer resolve events within those 100 ms. For speech, 25 ms is the usual compromise, because phonemes change on that scale.

From hertz to mel#

The mel scale reflects that the ear resolves low frequencies more finely. Eighty mel bands between 0 and 8 kHz is the standard for speech models. The effect: about 30 bands lie below 1 kHz and only about 15 between 4 and 8 kHz. That uneven spacing is not imprecision but an adaptation to perception.

Practical preprocessing#