Cerebras paper: layer dropout saves up to 25% training FLOPs and yields 1.55x faster decoding
burny_tech · x · 2026-09-09
A Cerebras research paper, "Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference," argues that layer dropout should return to state-of-the-art pretraining recipes, countering the belief that it hurts accuracy at scale.
Method
- Apply layer dropout (stochastic depth) at the whole Transformer-block level, sampled per sequence
- Use an increasing dropout distribution across depth and a schedule that decays to zero over training
Findings
- Across 2,400+ runs spanning 271M–8.2B parameters and up to 160B tokens, layer dropout yields lower loss at equal FLOPs
- LLMs can save up to 25% of training FLOPs at similar validation loss
- Post-training benefits: depth-elastic models enable early exit, layer skipping, and self-speculative decoding for up to 1.55x faster inference with negligible accuracy loss
All pretraining experiments ran on Cerebras CS-3 systems.
More from Infra
- PyTorch highlights cross-community collaboration at KubeCon + PyTorchCon China 2026 — PyTorch · 2026-09-09
- GLM 5.3 Flash goes live on W&B serverless inference: 1M context, vision, $0.50/M output — wandb · 2026-09-09
- Podcast Dives Into Broadcom Custom ASICs, 2027 Supply Bottleneck, and Nvidia's Hugging Face Deal — BenBajarin · 2026-09-09
- Magnitude open-sources Apple silicon inference server that auto-tunes local models for your Mac — nickbaumann_ · 2026-09-09
- Viettel unifies GPU fleet into Token-as-a-Service platform with three open source layers — PyTorch · 2026-09-09
- Put per-turn action schemas in the last user message to preserve prompt caching — Low_Bad_6585 · 2026-09-09