Cerebras paper: layer dropout saves up to 25% training FLOPs and yields 1.55x faster decoding
burny_tech · x · 2026-09-09
A Cerebras research paper, "Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference," argues that layer dropout should return to state-of-the-art pretraining recipes, countering the belief that it hurts accuracy at scale.
Method
- Apply layer dropout (stochastic depth) at the whole Transformer-block level, sampled per sequence
- Use an increasing dropout distribution across depth and a schedule that decays to zero over training
Findings
- Across 2,400+ runs spanning 271M–8.2B parameters and up to 160B tokens, layer dropout yields lower loss at equal FLOPs
- LLMs can save up to 25% of training FLOPs at similar validation loss
- Post-training benefits: depth-elastic models enable early exit, layer skipping, and self-speculative decoding for up to 1.55x faster inference with negligible accuracy loss
All pretraining experiments ran on Cerebras CS-3 systems.
Related event: Cerebras paper: layer dropout cuts training compute by 25%(2 posts)→
More from Infra
- Homemade animation walks through TSMC's CoWoS-S advanced packaging flow, bugs admitted — jwt0625 · 2026-10-24
- Muon optimizer may nearly halve LLM training compute, challenging a decade of AdamW — bycloud · 2026-10-24
- Infographic: why AI is an energy problem, from computation and cooling to the grid — CurieuxExplorer · 2026-10-24
- Engineer shares how he runs infra ops with Codex agents — kuza55 · 2026-10-24
- Running a 115B MoE with 262K context on an 8GB GPU at 70 T/s — jonathanx37 · 2026-10-24
- Physical AI Weekly: force-sensing gloves, Isaac Lab 3.0, humanoids in Audi plants — AIFlow_ML · 2026-10-24