Speech and Audio AI

Machines that listen and speak

Two days before the January 2024 New Hampshire primary, thousands of voters picked up the phone and heard what sounded like President Biden telling them to save their vote for November. He never said it. A political consultant had paid for a cloned voice, and within weeks the US telecoms regulator ruled unanimously that AI-generated voices in robocalls are illegal without consent.

What is speech and audio AI?

It is the family of systems that turn sound into meaning and meaning back into sound. The main members are speech recognition (audio to text, also called ASR), speech synthesis (text to speech, TTS), speaker recognition (who is talking), audio understanding (a dog barking, a machine about to fail, a song’s genre) and, newest, speech-to-speech models that hear you and answer out loud without writing anything down in between.

The same technology that cloned a president’s voice also writes live captions for deaf viewers, lets people who cannot type dictate email, reads screens aloud to blind users, and translates a conversation while it happens. In July 2026 OpenAI said about 150 million people use ChatGPT’s voice mode every week. Voice is becoming a default way to talk to computers.

How does it work, in one paragraph?

A microphone measures air pressure thousands of times a second, which gives a long list of numbers. Those numbers are converted into a picture of which frequencies are loud at each moment (a spectrogram), and a neural network reads that picture the way a language model reads tokens. Synthesis runs the same road backwards: text becomes a spectrogram or a sequence of audio tokens, and a second network turns that into a waveform your speaker can play. Modern voice assistants either chain these steps or fold them into a single model.

Why does it matter?

Speech is the interface humans learned first and use most. Getting it right opens computing to people who cannot read, type or see a screen, and to everyone whose hands are busy. Getting it wrong has costs of its own: a misheard dose, a caption that invents a sentence, a cloned voice that empties a bank account. This lesson builds the whole chain from the physics of sound up, with real signal processing running in your browser at every step.

Key takeaways

• Speech AI covers recognition, synthesis, speaker and sound understanding, and speech-to-speech conversation.
• Audio is turned into a time-frequency picture that neural networks read like a sequence; synthesis runs the chain in reverse.
• The same tools power captions, dictation and translation, and also voice-cloning fraud.

Sound as numbers

Sound is air being squeezed and stretched, hundreds or thousands of times per second. To a computer it is simply a list of numbers, and the first question is how many numbers per second you need.

What is a digital waveform?

A microphone turns pressure changes into a voltage, and an analogue-to-digital converter measures that voltage at regular instants. Each measurement is a sample. The sampling rate is how many samples are taken per second, and the bit depth is how finely each one is measured. CD audio uses 44,100 samples per second at 16 bits. Most speech models, including Whisper, work at 16,000 samples per second.

Do the arithmetic and the scale of the problem appears. One second of 16 kHz speech is 16,000 numbers. A ten-second voice note is 160,000 numbers, and it might contain 25 words. Text arrives already chopped into meaningful units; audio does not, which is why so much of speech AI is about compressing that river of samples into something shorter and more meaningful.

How fast is fast enough? The Nyquist limit

To capture a wave you need at least two samples per cycle: one near a peak and one near a trough. So a sampling rate of f_s can faithfully represent frequencies only up to f_s/2, called the Nyquist frequency. Anything higher does not simply vanish. Its samples line up perfectly with a lower-frequency wave, and that lower wave is what you hear on playback. This impostor is called an alias.

The folded frequency you hear when a tone f is sampled at rate fs. Example: a 3,000 Hz tone sampled at 4,000 Hz plays back as 1,000 Hz.

Past the Nyquist line the dots cannot tell a fast wave from a slow one. Real converters put a low-pass anti-aliasing filter in front of the sampler for exactly this reason: removing high frequencies is better than having them come back as fake low ones.

Why does it matter?

Almost all the information that distinguishes speech sounds lies below about 8 kHz, which is why 16 kHz sampling is the standard for speech models. Classic telephone audio samples at 8 kHz and filters out everything above roughly 3.4 kHz. That removes much of the hiss that separates “s” from “f”, which is why spelling a name over the phone needs “S for Sierra”. A model trained on wideband audio and deployed on phone calls sees a spectrogram with its top half missing, and its accuracy drops unless it was trained for that.

Key takeaways

• Digital audio is pressure sampled thousands of times per second: 16,000 numbers per second is standard for speech models.
• A sampling rate fs captures frequencies only up to fs/2; higher frequencies fold back as false lower tones called aliases.
• Sampling rate is a hidden source of distribution shift: phone audio lacks the upper band that wideband models expect.

Every sound is a sum of sines

Play a violin and a flute on the same note and you hear the difference instantly, even though both vibrate 440 times a second. The difference lives in ingredients you cannot see in the waveform but can measure with one of the most useful algorithms ever written.

What is a spectrum?

In 1807 Joseph Fourier argued that any repeating signal can be built by adding sine waves. For a pitched sound those sines sit at whole-number multiples of a fundamental frequency f₀: the harmonics 2f₀, 3f₀ and so on. The fundamental sets the pitch you hear. The relative strength of the harmonics sets the colour, or timbre. A list of how much of each frequency a sound contains is its spectrum.

How is it computed?

The discrete Fourier transform measures, for every frequency bin k, how well a frame of N samples lines up with a sine and cosine at that frequency. Written out it looks heavier than it is: each output is a weighted sum of the inputs.

The discrete Fourier transform. The fast Fourier transform (FFT) computes all N outputs in about N log N steps instead of N², which is what makes real-time audio analysis possible.

A worked example: take N = 4,096 samples at 16 kHz, which is 256 ms of audio. The bins are spaced 16,000 / 4,096 ≈ 3.9 Hz apart, so harmonics of a 220 Hz voice land about 56 bins apart and show up as clean separate peaks. Before the transform the frame is multiplied by a smooth window that fades its edges to zero; without it, the abrupt cut at the frame boundary smears energy into every bin.

Scrambling phases changes the waveform beyond recognition but leaves the spectrum, and almost all of what you hear, unchanged. That is the practical reason speech systems work with magnitudes: they keep what the ear cares about and discard what it mostly ignores.

Why does it matter?

The spectrum is where speech becomes legible. Pitch, the melody of a sentence, is the spacing of the harmonics. Vowels are patterns in their strengths. Hiss sounds like “s” are broad noise at high frequency. Almost every speech system, from 1950s digit recognisers to today’s audio language models, starts by moving from samples to frequencies.

Key takeaways

• Any periodic sound is a sum of sines at multiples of a fundamental; the fundamental is pitch, the harmonic mix is timbre.
• The FFT computes a spectrum in N log N time; windowing each frame keeps the spectrum clean.
• The ear mostly ignores phase, so speech systems keep magnitudes and throw phase away.

Spectrograms and the Mel scale

A single spectrum describes a moment. Speech changes every few tens of milliseconds, so we take a spectrum, slide along a little, take another, and stack them side by side. The result is a picture you can learn to read.

What is a spectrogram?

A spectrogram is the short-time Fourier transform drawn as an image: time runs left to right, frequency bottom to top, and brightness shows energy. In speech you can see the harmonics of the voice as fine horizontal stripes, formants as broad bright bands where the mouth and throat resonate, fricatives like “s” and “sh” as fuzzy noise high up, and stop consonants like “t” as a short silence followed by a sharp vertical burst. Phoneticians learned to read words off spectrograms by eye decades before computers could.

How does it work?

Cut the audio into overlapping frames, typically 25 ms long, starting every 10 ms. Window each frame, take its FFT, keep the magnitudes and put them on a log (decibel) scale, because loudness is perceived roughly logarithmically. The window length is a real choice with a real cost: a long window resolves frequencies finely but blurs quick events; a short one does the reverse. This is the time-frequency uncertainty principle, and no amount of computation escapes it.

Speech models then usually squeeze the frequency axis onto the Mel scale, a warping that matches how pitch is perceived: equal steps in Mels sound like equal steps in pitch. Below about 1 kHz it is nearly linear; above that it is roughly logarithmic, so the high frequencies are pooled into fewer, wider bands. Whisper uses 80 Mel bands computed from 25 ms windows every 10 ms, which turns each second of audio into 100 frames of 80 numbers.

The common Mel formula. 1,000 Hz is about 1,000 Mel; 8,000 Hz is only about 2,840 Mel.

The synthesised vowels are made the way a voice makes them: a buzzy source full of harmonics, shaped by resonances. As the vowel moves from “ee” to “ah” to “oo”, the harmonic stripes barely change but the bright formant bands slide. That pattern, not the pitch, is what tells you which vowel you heard. The live view uses your browser’s built-in FFT analyser; nothing is recorded or sent anywhere.

Why does it matter?

The log-Mel spectrogram is the standard input for speech recognition, speaker identification and audio classification, and the standard intermediate target for many speech synthesisers. It shrinks the data by an order of magnitude while keeping what humans use to understand speech. Once audio is an image-like grid, the whole toolkit of convolutional networks and transformers applies.

Key takeaways

• A spectrogram stacks short-time spectra: harmonics, formants, hiss and bursts are all visible in it.
• Window length trades time precision against frequency precision; 25 ms windows every 10 ms are the speech default.
• The Mel scale spends resolution where hearing does, and log-Mel spectrograms are the usual input to speech models.

From sound to text

Say “recognise speech” quickly and it can sound a lot like “wreck a nice beach”. Audio has no spaces between words, every speaker sounds different, and each sound bends toward its neighbours. Speech recognition took seventy years to get good.

What makes recognition hard?

• No boundaries. Fluent speech is continuous; the pauses you think you hear between words are mostly supplied by your brain.
• Coarticulation. The “k” in “key” and the “k” in “coo” are made in different places in the mouth because the lips and tongue are already preparing the vowel.
• Variation. Accents, ages, speaking rates, microphones, rooms and background noise all change the spectrogram.
• Alignment. A second of audio gives 100 spectrogram frames but only two or three words. Transcripts do not say which frames belong to which letter.

Seventy years in eight steps

1. 1952Digits, one speaker

   Bell Labs’ Audrey recognised the spoken digits 0 to 9 for a single talker by matching formant patterns.

2. 1980s to 2000sHidden Markov models

   Phones as hidden states, Gaussian mixtures scoring each 10 ms frame, a pronunciation dictionary and an n-gram language model searched together.

3. 2006CTC

   A loss that lets a network learn from audio and a transcript alone, with no frame-by-frame alignment.

4. 2012Deep neural networks

   Neural networks replace the Gaussian mixtures inside the HMM and cut error rates sharply across several benchmarks.

5. 2014 to 2019End to end

   One network from spectrogram to characters: CTC, attention encoder-decoders and RNN-transducers for streaming on phones.

6. 2020Self-supervised audio

   wav2vec 2.0 learns from unlabelled audio first, then needs only minutes of transcribed speech to fine-tune.

7. 2022Weak supervision at scale

   Whisper trains on 680,000 hours of web audio with imperfect transcripts and works across languages and accents without fine-tuning.

8. 2024 onwardSpeech inside language models

   Audio becomes tokens for a multimodal LLM, which can transcribe, translate, answer or reply in speech directly.

How does CTC solve the alignment problem?

Connectionist Temporal Classification (Graves et al., 2006) lets a network output one symbol per frame, including a special blank meaning “nothing new here”. To read the output, merge repeated symbols, then delete the blanks. Many different frame-level paths collapse to the same text, and during training CTC adds up the probability of every one of them with a dynamic programme. The network is never told where each letter is; it only has to make the total probability of the right transcript high.

The count at the bottom is computed with the same forward recursion CTC uses in training, with every symbol given equal weight. Even a five-letter word over twelve frames has thousands of valid alignments; real utterances have astronomically many, which is why the dynamic programme matters.

How do modern recognisers work?

A modern recogniser has an encoder, usually a transformer or a Conformer (a transformer with convolution layers added), that turns log-Mel frames into contextual vectors. On top sits one of three decoders: a CTC head (fast and simple), a transducer that emits text as audio streams in (what phone keyboards and live captions use), or an attention decoder that writes the transcript token by token like a language model.

Two ideas about data made the biggest difference. wav2vec 2.0 (2020) pre-trains the encoder on unlabelled audio by masking parts of it and learning to identify the missing piece, in the same spirit as masked language modelling. After that, fine-tuning on just ten minutes of transcribed speech reached 4.8% word error on clean LibriSpeech audio, which matters enormously for the thousands of languages with little transcribed data. Whisper (2022) went the other way: 680,000 hours of web audio paired with the imperfect transcripts found alongside it. Special tokens let one model detect the language, transcribe, translate into English and add timestamps. On LibriSpeech it did not beat models fine-tuned on that dataset, but across a wide range of other test sets it made about 55% fewer errors on average than a fine-tuned model with the same LibriSpeech score: it was robust to accents, noise and domains it had never been tuned for.

Why does it matter?

Recognition is the front door for captions, meeting notes, voice search, call-centre analytics and every voice assistant. The design choices are real trade-offs: CTC and transducers stream with low latency and cannot invent long passages; attention decoders read context better but, being language models, can produce fluent text the audio never contained. You will meet that failure again in the risks section.

Key takeaways

• Speech recognition went from template matching to HMMs, to deep networks, to end-to-end transformers trained on vast audio.
• CTC solves alignment with a blank symbol and a collapse rule, summing over every path that spells the transcript.
• Self-supervision (wav2vec 2.0) and weak supervision at scale (Whisper) made recognition robust and multilingual.

Measuring recognition: word error rate

“97% accurate” sounds impressive until you notice that the 3% includes the one word in your sentence that mattered. The field’s standard metric is simple, useful, and easy to misread.

What is word error rate?

Line up the recogniser’s output (the hypothesis) with a human transcript (the reference) using the fewest possible word edits. Count substitutions S, deletions D and insertions I, and divide by the number of reference words N.

Because insertions count, WER can exceed 100%: a recogniser that hallucinates a paragraph over a two-word reference scores far worse than one that outputs nothing.

How is the alignment found?

With the same dynamic programme as spelling correction, the Levenshtein edit distance, applied to words instead of letters. Fill a table where each cell holds the cheapest way to turn the first i reference words into the first j hypothesis words, then trace back the cheapest route. The demo does exactly that; open the table to watch it.

Normalisation matters as much as the model. “Dr.” versus “doctor” and “3 PM” versus “three pm” are formatting choices, not recognition errors, and the Whisper authors built a dedicated text normaliser so that such differences would not dominate their comparisons.

Why does it matter, and where does it mislead?

• All words weigh the same. Missing “the” and turning “fifteen milligrams” into “fifty milligrams” each cost one error. For safety-critical uses, count critical-term errors separately.
• Averages hide groups. Koenecke et al. (2020) tested five commercial recognisers and found an average WER of 35% for Black speakers against 19% for white speakers. A single headline number would never have shown it.
• Benchmarks are not your audio. Clean read audiobooks (LibriSpeech) are far easier than accented, overlapping, noisy conversation. Always measure on a sample of your own recordings.
• Words are not universal. Chinese, Japanese and Thai do not put spaces between words, so they are usually scored with character error rate instead.

Key takeaways

• WER = (substitutions + deletions + insertions) / reference words, found with a word-level edit-distance alignment.
• Text normalisation can change WER as much as the model does; compare systems only under the same normaliser.
• WER treats every word equally and hides group differences: measure on your own audio and break results down by speaker group.

Teaching machines to speak

For decades synthetic speech was instantly recognisable: the voice Stephen Hawking used came from a 1980s formant synthesiser. Today a few seconds of someone’s voice are enough to make a model speak as them, with their accent and their laugh.

What does a text-to-speech system have to do?

1. Normalise the text. “Dr. Lee lives on Lee Dr.” needs “doctor” and then “drive”; “1998” is a year, “$19.98” is a price.
2. Choose pronunciations. “Read” rhymes with “reed” or “red” depending on tense; names and loanwords are notoriously hard.
3. Plan prosody. Where to pause, which word to stress, whether the pitch rises or falls. “I didn’t say she stole it” means seven different things depending on which word you stress.
4. Generate audio. Produce a spectrogram or codec tokens, then a waveform.

How does it work? The source-filter model

Your voice has two independent parts. The source is the vocal folds buzzing at a fundamental frequency, producing a harmonic-rich tone; its pitch contour carries intonation. The filter is your throat, mouth and lips, whose shape creates resonances called formants. The first two formants, F1 and F2, identify vowels: F1 falls as the tongue rises, F2 rises as the tongue moves forward. Every speech synthesiser, from 1950s electronics to neural networks, has to model both parts, whether explicitly or not.

Moving the point changes what is said; changing the contour changes how it is said. The reference vowels come from Peterson and Barney (1952). Flat pitch sounds robotic immediately, which is why prosody became the hardest and most important part of natural-sounding synthesis.

Four generations of synthetic voices

From WaveNet to zero-shot cloning

WaveNet (DeepMind, 2016) was the turning point. It generated raw audio one sample at a time, each prediction conditioned on thousands of previous samples through stacked dilated convolutions, and listeners rated it far more natural than the best existing systems. It was also slow: 16,000 sequential predictions for every second of speech. The following years were spent keeping the quality while generating in parallel, which led to the fast neural vocoders used today.

The next leap reframed the problem. VALL-E (Microsoft, 2023) treated speech synthesis as language modelling over discrete audio tokens from a neural codec, trained on 60,000 hours of English speech. Given text and a three-second recording of a stranger, it continues “in their voice”, copying timbre, accent and even the acoustics of the room. That capability is the foundation of both today’s natural assistants and today’s voice-cloning scams.

Why does it matter?

Good synthesis powers screen readers, audiobooks, navigation, dubbing and voice assistants, and it can give a voice back to people who have lost theirs to illness, trained from old recordings. The same zero-shot capability means a voice is no longer proof of identity. That trade-off shapes the rest of this lesson.

Key takeaways

• Speech is a source (pitch and its contour) shaped by a filter (formants); F1 and F2 identify vowels, prosody carries meaning.
• Synthesis went from rules, to spliced recordings, to neural spectrogram models with neural vocoders, to codec language models.
• Zero-shot models like VALL-E clone a voice from seconds of audio, which makes voices useless as proof of identity.

Audio tokens and talking machines

Language models are superb at predicting the next token. If audio could be written as tokens, the same machinery could listen, speak, sing and hold a conversation. Neural audio codecs made that possible.

What is a neural audio codec?

An autoencoder for sound. An encoder compresses the waveform into a short sequence of vectors, a quantiser replaces each vector with entries from learned codebooks, and a decoder turns the codes back into audio. The indices are the tokens. SoundStream (Google, 2021) and EnCodec (Meta, 2022) established the design, and both rely on a trick called residual vector quantisation (RVQ).

How does residual quantisation work?

One codebook with 1,024 entries can only describe 1,024 different sounds per frame, which is far too coarse. RVQ uses a chain of codebooks instead. The first picks the nearest entry; the second quantises what the first got wrong; the third quantises what is still left; and so on. Each stage adds detail, and the number of distinct outputs multiplies: eight codebooks of 1,024 entries can describe 1,024⁸ different frames with just eight tokens.

The codebooks here are trained by k-means in your browser, exactly as RVQ codebooks can be initialised. Real codecs use 128-dimensional vectors and train the codebooks jointly with the encoder and decoder, but the stage-by-stage refinement is the same. Dropping later stages lowers quality gracefully, which is how one EnCodec model serves bitrates from 1.5 to 24 kbps.

EnCodec at 24 kHz: 75 frames per second, 8 codebooks, 1,024 entries each (10 bits).

From codecs to conversation

Once audio is tokens, a transformer can be trained on sequences that mix text tokens and audio tokens. That is how GPT-4o (2024) handled voice: one network trained end to end on text, images and audio. OpenAI reported audio replies in 320 ms on average, against 2.8 and 5.4 seconds for its previous voice mode, which chained three separate models.

Human conversation is faster still, and messier. Stivers et al. (2009) found that across ten languages, replies most often begin within about 200 ms of a question ending, and people overlap, interrupt and say “mm-hm” while listening. Moshi (Kyutai, 2024) was the first open model built for this full-duplex style. It models two audio streams at once, the user’s and its own, so it can listen while speaking, and it predicts a time-aligned text “inner monologue” alongside its audio tokens to keep what it says coherent. Its Mimi codec compresses speech to 12.5 frames per second, and the paper reports about 200 ms latency in practice. By mid-2026 the approach reached mass-market products: OpenAI replaced ChatGPT’s voice mode with a full-duplex model that can make turn-taking decisions several times per second.

Music, briefly

The same recipe generates music. Text-conditioned models predict codec tokens for a few seconds or minutes of audio, and commercial song generators now produce full tracks with vocals from a one-line prompt. Music raises the questions of speech synthesis at larger scale: whose recordings were used for training, whether a generated song that imitates a living artist’s voice is allowed, and how listeners can tell. Record labels and artists have taken these questions to court, and the answers are still being settled.

Why does it matter?

Audio tokens turned speech from a separate engineering stack into one more modality for foundation models. Assistants can now hear sarcasm, hesitation and background context, answer with emotion, and be interrupted like a person. But cascades have not disappeared: they are easier to audit (there is a transcript at every step), easier to plug into tools, and many production voice agents still use them.

Key takeaways

• Neural codecs turn audio into a short sequence of discrete tokens using residual vector quantisation.
• With audio as tokens, language models can synthesise speech, clone voices, make music and hold spoken conversations.
• Full-duplex models listen while they speak, reaching response times close to the 200 ms humans expect.

Voice cloning, trust and access

For all of history, hearing a familiar voice was good evidence of who was speaking. That assumption has quietly expired, and people, companies and laws are still catching up.

What goes wrong?

• Fraud. In early 2024 an employee of the engineering firm Arup in Hong Kong joined a video call with what looked and sounded like the company’s CFO and colleagues, and transferred $25 million to fraudsters. Every other participant was fake. Smaller versions of the same trick, a cloned relative asking for emergency money, are now common.
• Election interference. The New Hampshire robocall led to a $6 million FCC fine for the consultant who commissioned it. In June 2025 a state jury acquitted him of the criminal voter-suppression charges, a reminder that existing laws fit new harms imperfectly.
• Invented transcripts. Koenecke et al. (2024) found that about 1% of Whisper transcriptions contained phrases or sentences that were never spoken, and 38% of those included harmful content such as violence or false authority. Rates were higher for people with aphasia, whose speech has longer pauses.
• Unequal accuracy. Recognisers that work worse for some accents and dialects push those speakers out of voice interfaces, and into worse outcomes when transcripts feed hiring, healthcare or legal records.

How are people responding?

• Consent law. The FCC’s 2024 ruling treats AI voices as “artificial” under the Telephone Consumer Protection Act, so robocalls using them need the recipient’s prior consent. In the EU, Article 50 of the AI Act applies from 2 August 2026: AI-generated audio must be marked in a machine-readable way, and deepfakes must be disclosed.
• Watermarking. Systems such as AudioSeal hide an inaudible signal in generated speech that a detector can find even in short or edited clips. The limit is structural: a watermark only exists if the generator adds it, and open-source models can be run without one.
• Detection. Classifiers trained to spot synthetic speech work on the generators they were trained against and often fail on new ones. Treat a detector’s “real” verdict as weak evidence.
• Process. The defences that work regardless of quality are procedural: call back on a known number, agree a family code word, require a second channel for any payment request.

Why does it matter? Because the upside is real too

Speech technology is one of the clearest cases in AI of a capability that is simultaneously an accessibility tool and a fraud tool. Responsible deployment means measuring accuracy for the people who depend on it most, and assuming that anything which can be heard can be forged.

Key takeaways

• Voice cloning has already enabled multimillion-dollar fraud and election robocalls; a familiar voice is no longer evidence of identity.
• Law (FCC 2024, EU AI Act Article 50), watermarking and detection help but each has gaps; procedural verification is the robust defence.
• Speech AI is also a major accessibility technology, so accuracy for underserved speakers and hallucination rates deserve direct measurement.

Check your understanding

Seven situations from real speech projects. Choose the answer you would stake the project on.

You have 5,000 hours of call-centre audio recorded at 8 kHz and a recogniser trained on 16 kHz audio. You resample the calls to 16 kHz and accuracy is still poor. Why, and what helps most?

References and further reading

Papers, rulings and reports cited in this lesson. For a single deep source, Jurafsky and Martin’s Speech and Language Processing has free chapters on phonetics, ASR and TTS. To keep going on this path, try the RNN lab or read Multimodal AI.

References

1. [1]

   Declaratory Ruling: Implications of Artificial Intelligence Technologies on Protecting Consumers from Unwanted Robocalls and Robotexts (FCC 24-17)(opens in a new tab)

   Federal Communications Commission, 2024

   Unanimous ruling of 8 February 2024 that AI-generated voices count as “artificial” under the Telephone Consumer Protection Act.

2. [2]

   OpenAI launches GPT-Live, a full-duplex voice upgrade that lets ChatGPT talk more like a person(opens in a new tab)

   VentureBeat, 2026

   July 2026: ChatGPT’s voice moves to a full-duplex model that listens while it speaks; OpenAI cites 150 million weekly voice users.

3. [3]

   Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks(opens in a new tab)

   Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber, 2006

   ICML 2006. The CTC loss: train a sequence model from unaligned input and label sequences using a blank symbol.

4. [4]

   wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations(opens in a new tab)

   Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli, 2020

   Self-supervised pre-training on unlabelled audio; with ten minutes of labelled data it reached 4.8 / 8.2 WER on LibriSpeech clean / other.

5. [5]

   Robust Speech Recognition via Large-Scale Weak Supervision(opens in a new tab)

   Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever, 2022

   Whisper: an encoder-decoder transformer trained on 680,000 hours of weakly labelled multilingual web audio.

6. [6]

   Racial disparities in automated speech recognition(opens in a new tab)

   Allison Koenecke, Andrew Nam, Emily Lake, et al., 2020

   Five commercial recognisers averaged a WER of 0.35 for Black speakers and 0.19 for white speakers.

7. [7]

   Control Methods Used in a Study of the Vowels(opens in a new tab)

   Gordon E. Peterson, Harold L. Barney, 1952

   The classic measurements of vowel formant frequencies for men, women and children (JASA 24, 175).

8. [8]

   WaveNet: A Generative Model for Raw Audio(opens in a new tab)

   Aäron van den Oord, Sander Dieleman, Heiga Zen, et al., 2016

   An autoregressive network that generates audio one sample at a time with dilated causal convolutions.

9. [9]

   Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers(opens in a new tab)

   Chengyi Wang, Sanyuan Chen, Yu Wu, et al., 2023

   VALL-E: text-to-speech as language modelling over neural codec tokens; imitates a voice from a 3-second recording.

10. [10]

    High Fidelity Neural Audio Compression(opens in a new tab)

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi, 2022

    EnCodec: a convolutional autoencoder with residual vector quantisation, producing discrete audio tokens at 1.5 to 24 kbps.

11. [11]

    Hello GPT-4o(opens in a new tab)

    OpenAI, 2024

    One model trained end to end across text, vision and audio; audio responses in 320 ms on average versus 2.8 s and 5.4 s for the earlier cascaded Voice Mode.

12. [12]

    Universals and cultural variation in turn-taking in conversation(opens in a new tab)

    Tanya Stivers, N. J. Enfield, Penelope Brown, et al., 2009

    Across 10 languages, answers most often begin within about 200 ms of the end of a question.

13. [13]

    Moshi: a speech-text foundation model for real-time dialogue(opens in a new tab)

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil Zeghidour, 2024

    A full-duplex spoken dialogue model with a theoretical latency of 160 ms and about 200 ms in practice, built on the Mimi codec.

14. [14]

    Arup revealed as victim of $25 million deepfake scam involving Hong Kong employee(opens in a new tab)

    CNN Business, 2024

    A finance employee paid out $25 million after a video call in which the other participants were deepfaked, voices included.

15. [15]

    N.H. jury acquits consultant behind AI robocalls mimicking Biden on all charges(opens in a new tab)

    WBUR News (Associated Press), 2025

    The New Hampshire primary robocall case: the criminal trial outcome and the separate $6 million FCC fine.

16. [16]

    Careless Whisper: Speech-to-Text Hallucination Harms(opens in a new tab)

    Allison Koenecke, Anna Seo Gyeong Choi, Katelyn X. Mei, Hilke Schellmann, Mona Sloane, 2024

    About 1% of Whisper transcriptions contained invented phrases; 38% of those included explicit harms. Rates were higher for speakers with aphasia.

17. [17]

    Quick facts: transparency rules for AI systems (AI Act, Article 50)(opens in a new tab)

    European Commission, 2026

    From 2 August 2026, AI-generated audio must be marked as such in a machine-readable way, and deepfakes disclosed.

18. [18]

    Proactive Detection of Voice Cloning with Localized Watermarking(opens in a new tab)

    Robin San Roman, Pierre Fernandez, Alexandre Défossez, Teddy Furon, Tuan Tran, Hady Elsahar, 2024

    AudioSeal: an imperceptible watermark that can be detected sample by sample, so edited clips can still be flagged.