Robust Speech Recognition via Large-Scale Weak Supervision
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever
eess.AS, cs.CL, cs.LG, cs.SD
2022-12-07
OpenAI trains an encoder-decoder Transformer on 680k hours of web captions; zero-shot it cuts average WER 55.2% versus a matched LibriSpeech-supervised model across 12 datasets.
Self-supervised speech encoders such as Wav2Vec 2.0 can soak up a million hours of raw audio, then still need a supervised decoder fine-tuned on each target distribution. Fine-tuning is good at harvesting dataset-specific shortcuts. A vision analogue is blunt: a 9.2-point ImageNet gain from fine-tuning can vanish on seven other natural-image sets.
Speech has the same crack. Deep Speech 2 already matched a 5.8% human WER on LibriSpeech test-clean in 2015; later supervised systems pushed that number to 1.4%. Those models still fall apart off clean read speech. SpeechStew pooled 5,140 hours of gold sets and still sat two orders of magnitude below unlabeled audio collections. Whisper scales weak supervision the rest of the way: 680,000 hours of internet audio-caption pairs, including 117,000 hours in 96 non-English languages and 125,000 hours of X-to-English translation. The aim is one system that works out of the box, without a decoder fine-tune per deployment.
The architecture is an off-the-shelf encoder-decoder Transformer, kept plain on purpose so scale is the variable. Audio is resampled to 16 kHz and turned into an 80-channel log-Mel spectrogram (25 ms window, 10 ms hop), then a two-layer convolutional stem and Transformer blocks. English models reuse GPT-2's byte-level BPE; multilingual models refit the vocabulary. Sizes run from Tiny at 39M parameters to Large at 1.55B.
The real work is filtering and the multitask token format. Web captions are full of ASR-generated "transcript-ese": all caps, no commas, no paragraph breaks. Heuristics drop machine transcripts. An audio language detector checks that spoken language matches caption language; English captions on non-English audio become translation examples. Audio is sliced into 30-second chunks. Silent chunks, downsampled, train voice activity detection.
Special decoder tokens specify language, transcribe versus translate, and whether timestamps are required. One set of weights covers recognition, translation, language ID, and speech/non-speech. Training runs about 220 updates, two to three epochs, batch 256 segments, almost no augmentation. A short fine-tune on transcripts without speaker names stops the model from hallucinating who is talking. Long-form decoding slides the 30-second window using predicted timestamps, with beam search, temperature fallback on repetition and log-prob, and VAD heuristics.
Zero-shot is the point. The largest Whisper is about 2.5% WER on LibriSpeech test-clean, merely mid-2019 supervised quality. Against a wav2vec 2.0 model matched on that number, Whisper averages 12.8% WER on 12 other academic sets versus 29.3%, a 55.2% relative error cut. The 39M Tiny model is already competitive out of domain with the best LibriSpeech-supervised systems. Under additive pub noise below 10 dB SNR, Whisper overtakes NVIDIA STT models that win in the clean regime.
| Setting | Metric | Whisper | Baseline |
| 12 out-of-domain English sets | mean WER | 12.8 | wav2vec 2.0: 29.3 |
| CoVoST2 X→en | BLEU | 29.1 | prior SOTA lower; +6.7 on the low-resource group |
| Fleurs LID (82 overlapping langs) | accuracy | 80.3% | 20 unseen Fleurs languages cap the full set |
| Kincaid46, 25 recordings | vs humans | near pure-human; 1.15 points behind computer-aided | four commercial ASR APIs |
Multilingual WER tracks pre-training hours tightly: R² about 0.83 on log-log Fleurs, WER roughly halves per 16× data. Hebrew, Telugu, Chinese, and Korean sit above that line. Translation hits a new overall CoVoST2 SOTA at 29.1 BLEU, yet still trails Maestro/mSLAM on high-resource pairs. Dataset ablations: English WER falls from 30.5 at 3,000 hours to 9.9 on the full set, with clear diminishing returns past 54,000 hours.
What practitioners get is open weights and inference code that do not need per-dataset fine-tuning. Noise, accents, rare words, and hour-long audio are where it beats LibriSpeech-tuned models. Packing VAD, language ID, transcription, and translation into one decoder also simplifies the classic pipeline.
English ASR is already near human on the authors' long-form sample. Further scale mostly buys multilingual and translation headroom. Weak supervision only works after aggressive filtering of machine captions and language mismatches.
VoxPopuli is a miss: Whisper lags Maestro, XLS-R, and mSLAM. The authors suspect those systems pre-trained on this distribution and that VoxPopuli has about 10× more supervised hours per language than MLS. Language ID on Fleurs is 13.6 points behind supervised SOTA, partly because 20 Fleurs languages never appear in training. A custom text normalizer can cut WER by up to 50%; they admit it may overfit Whisper's writing style. Purported Welsh translation data (9,000 hours) is mostly English mis-tagged by the language detector.
Long-form decoding is a pile of heuristics. Greedy decoding loops; beam search, temperature fallback, and a 0–1 s constraint on the first timestamp are load-bearing. Sliding 30-second windows paper over shaky timestamp predictions. The "near human" claim rests on 25 Kincaid46 recordings and should not be generalized.