2,400 experiments show layer dropout can match dense baselines in LLM pretraining
burkov · x · 2026-09-10
A systematic study revisits layer dropout—randomly skipping transformer blocks during training—long abandoned due to reported accuracy loss at scale. Researchers ran 2,400+ controlled experiments on decoder-only transformers from 271M to 8.2B parameters, training on up to 160B tokens with Cerebras CS-3 systems, jointly optimizing the configuration to see if layer dropout can match or beat dense baselines while cutting training FLOPs and inference latency.
Related event: Cerebras paper: layer dropout cuts training compute by 25%(2 posts)→
More from Infra
- turbovec: Rust vector index fits 10M document vectors in 4GB RAM and outpaces FAISS — tom_doerr · 2026-09-10
- LM Studio 0.4.24 adds advanced llama.cpp argument overrides for GGUF model loading — solyarisoftware · 2026-09-10
- tszzl wraps up: efficiency gains only amplify hunger for hardware — tszzl · 2026-09-10
- Trimming MTP draft vocab to 47k boosts DGX Spark code decoding by 21.5% on same hardware — MaziyarPanahi · 2026-09-10
- Is 5 tokens/s usable for local LLMs? Redditor runs 27B model off an iGPU — Zombiecidialfreak · 2026-09-10
- Deep-Dive Speculative Decoding Blog Incoming: Drafter Training to vLLM Serving — auto_grad_ · 2026-09-10