Microsoft Researcher Highlights Key ICML Papers on Training and Fine-Tuning

JohnCLangford · x · 2026-08-03

John Langford from Microsoft Research highlights several recent AI training and optimization papers, primarily from ICML, that he found interesting:

FAC: Achieves post-training with 1/100th the examples by building synthetic examples to cover an induced sparse feature representation.
Bounded Log loss: Provides substantial improvements in supervised fine-tuning for reasoning.
Maximum Likelihood RL: Offers a better bias for chain-of-thought RL by overemphasizing low probability successes.
MuonSSM: Integrates Newton-Schulz into the associative operator of SSMs, costing an order of magnitude in compute but yielding significant performance benefits.
SDFT: A distillation-based fine-tuning approach that heavily reduces forgetting.
ReQAT: Enables 4-bit training through a carefully studied conjunction of techniques, resulting in large speedups.
Datamixing scaling laws: Explores the mechanics and understanding of datamixture scaling.

More from Research