text to speech

Text-to-speech turns written text into a waveform, and the whole field is a sequence of fixes to one original sin: speech is continuous and text is discrete, so something has to invent all the acoustic detail — pitch, duration, timbre — that the text alone never specifies. The modern pipeline splits that problem in two: an acoustic model predicts an intermediate representation (usually a mel spectrogram) from text, and a vocoder turns that spectrogram into an actual audio waveform. Most of TTS research since 2016 is the history of making each of those two stages faster and more natural, and the newest systems are starting to collapse them back into one model.

Tacotron 2 was the system that proved neural TTS could sound almost human, and it did so with an autoregressive bottleneck. Shen and colleagues at Google (2018) built an encoder-decoder with attention that reads text character by character and generates mel spectrogram frames one at a time, each frame conditioned on everything generated before it — the same recurrent structure a language model uses for text. In their evaluation, human raters scored Tacotron 2's output 4.53 out of 5 on mean opinion score, against 4.58 for real recorded human speech, which was close enough that the result is widely cited as the moment neural TTS crossed into near-human quality. The cost of that quality was speed: generating one frame at a time means synthesis time scales with utterance length, and nothing about the architecture allows parallel generation.

FastSpeech 2 removed the autoregressive bottleneck by predicting how long each sound should last before generating it. Ren and colleagues at Microsoft (2020) replaced the frame-by-frame decoder with a non-autoregressive one: a duration predictor estimates how many mel frames each phoneme should occupy, and the model expands the entire sequence and generates every frame in parallel rather than waiting on the previous one. The reported result was roughly 30 times faster synthesis than Tacotron 2 with comparable quality, and the explicit duration and pitch predictors it introduced — collectively called the variance adaptor — also gave users a direct, controllable lever over prosody that an implicit autoregressive model never exposed. Making duration an explicit prediction rather than an emergent property of attention turned out to be the key unlock for both speed and controllability at once.

The vocoder's job — turning a spectrogram into a waveform — went through the same slow-but-accurate to fast-and-good-enough arc. WaveNet (van den Oord and colleagues, DeepMind, 2016) generates raw audio sample by sample using dilated causal convolutions and was the first vocoder to sound convincingly natural, but at roughly half of real-time speed it was far too slow to deploy in an interactive product. HiFi-GAN (Kong and colleagues, 2020) replaced that sample-by-sample generation with a GAN trained to produce the whole waveform in one forward pass, reaching real-time factors well under 0.01 — generating audio roughly a hundred times faster than it plays back — while matching WaveNet-level perceptual quality, which is why it became the default production vocoder across the field. Diffusion-based vocoders such as WaveGrad and DiffWave push quality further still but reintroduce the multi-step, slower-than-real-time cost that HiFi-GAN had eliminated.

VITS collapsed the two-stage pipeline into one model, trained end to end. Kim and colleagues (2021) combined an acoustic model and a vocoder into a single network trained jointly with variational inference and adversarial training, removing the mel-spectrogram handoff between two separately trained components entirely. Text goes in, a waveform comes out of the same model, and the reported naturalness is competitive with the best two-stage systems while the training setup is simpler, since there's no longer a risk of the acoustic model and vocoder being mismatched because they were trained on different data or objectives.

The newest systems treat voice as something to be sampled and conditioned on, not just synthesized. Meta's VoiceBox (2023) uses flow matching and can clone a speaker's voice from a few seconds of reference audio without any speaker-specific fine-tuning, generating in a new voice purely from in-context conditioning the way a large language model generates in a new style from a prompt. Commercial systems like ElevenLabs and open systems like Bark layer emotion and style control on top of the same basic idea, treating a voice as a conditioning signal the model can be steered by rather than a fixed, separately-trained target.

Read the history of text-to-speech as a running trade between autoregression and parallelism at every layer of the stack: Tacotron 2 proved the quality bar could be hit one frame at a time, FastSpeech 2 showed almost all of that quality survives predicting every frame at once, WaveNet proved a vocoder could sound real sample by sample, and HiFi-GAN showed the entire waveform could be generated in a single parallel pass instead — and each of those parallelism wins is what actually made neural TTS cheap enough to run in a live product rather than a research demo.