Scale learning rate by 1/sqrt(f) when sampling rows, Microsoft finds
JohnCLangford · x · 2026-08-14
John Langford shares a rule of thumb: using a fraction f of rows implies scaling the learning rate by 1/sqrt(f); joint tuning yields statistical convergence gains of roughly 1 centinat on some models, part of the Dion optimizer's statistical improvements.
Related event: Microsoft researchers improve convergence with row norm and sampling tuning(2 posts)→
More from Research
- Prime Flash MoE: Blackwell-Optimized CUDA Kernels Speed Up MoE Inference by 2.4x — pbaylies · 2026-08-14
- Why MiniMax Music3 needs an RVQ tokenizer: a technical explainer — ostrisai · 2026-08-14
- Philosopher Chalmers analyzes Anthropic's J-space: not a global workspace — burny_tech · 2026-08-14
- Chinese chemogenetics trial not on ClinicalTrials.gov; watch ChiCTR for FIH trials — MWCvitkovic · 2026-08-14
- Universal hand action space enables cross-embodiment robot hand skill learning — chris_j_paxton · 2026-08-14
- Fisher-Rao Distance Between Multivariate Normals Can Be Approximated Arbitrarily Finely — FrnkNlsn · 2026-08-14