Researchers derive a dropout scaling law from active parameter count
gordic_aleksa · x · 2026-10-16
gordicaleksa highlighted new research deriving a scaling law for dropout: Dropout = sigmoid(0.526(log ActiveParams − 27.2)), meaning the optimal dropout rate can be predicted from a model's active parameter count via this sigmoid form. His quip: you either die trying to reach the frontier, or live long enough to create dropout scaling laws.
More from Research
- ORL pushes diffusion LMs to a new speed-intelligence Pareto frontier with ~6.9x throughput gains — james_y_zou · 2026-10-16
- Why CoT is the key: classic paper shows transformers need it for serial problems — willcb · 2026-10-16
- Circuit view may dissolve much of deep learning's mystery, argues researcher thread — willcb · 2026-10-16
- Terence Tao: OpenAI's 300-paper dump wiped out junior research projects overnight — Hesamation · 2026-10-16
- Stanford launches CS312 'Deep Learning Alchemy': hands-on LLM experiments plus prediction quizzes — stanfordnlp · 2026-10-16
- DOT-MoE: converting dense models into sparse MoE via optimal transport — burkov · 2026-10-16