MAI-Base-2 adopted the 1/(weight_decay·lr) timescale hparam scaling rule
SeunghyunSEO7 · x · 2026-10-16
Seunghyun highlights that Microsoft's MAI-Base-2 training report derived hyperparameter scaling rules by treating 1/(weightdecay × lr) as a timescale — a perspective he and @laurenceai proposed earlier, later extended to LLM pre-training by Shane Bergsma. A case of an academic scaling insight being adopted in frontier model training.
More from Research
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16
- MP-Bench: 12 voice agents no better than coin flip at knowing when to speak in group chats — solyarisoftware · 2026-10-16
- Neuron-level GWAS + PheWAS on Qwen3-8B pinpoints safety-critical neurons — Sauers_ · 2026-10-16
- Tencent squeezes 770B-parameter Hunyuan Hy4 Preview into 214 GiB at ~2.38 bits per weight — coolmum18 · 2026-10-16
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16