Chinchilla showed smaller models on more data beat GPT-3-scale giants
alex_verem · x · 2026-10-13
Thread entry #2: DeepMind's Training Compute-Optimal Large Language Models (Chinchilla). Training 400+ models from 70M to 16B parameters, the authors found compute-optimal training scales model size and tokens equally — most large models were significantly undertrained. Chinchilla (70B, 4× data, same compute as Gopher) uniformly beat Gopher (280B), GPT-3, Jurassic-1 and Megatron-Turing NLG, proving size isn't everything.
More from Research
- Graph theorist spends 20 hours on OpenAI Math Dump, finds solutions to his decade-old conjectures — burny_tech · 2026-10-13
- tszzl predicts mechinterp progress in coming months unlike anything in years — tszzl · 2026-10-13
- Free ML Workshop on the Math Discord Server Covers ML, Deep Learning and Graphs — zeroalephzeta · 2026-10-13
- sansreplace: a novel O(k log k) sampling-without-replacement algorithm with zero extra memory — burny_tech · 2026-10-13
- New Paper Explains Mamba-3's State Rotations as Underdamped Second-Order Dynamics — tri_dao · 2026-10-13
- New Paper Gives Exact Pricing and Analytic Risk Attribution for First-to-Default Basket Swaps — DrCirillo · 2026-10-13