Weight decay scaling was in the original AdamW paper — first author Loshchilov is on X
SeunghyunSEO7 · x · 2026-10-18
A post points out that the now-frequently-discussed scaling rule for weight decay was already present in the original AdamW paper. The author also notes that Ilya Loshchilov — first author of both AdamW and NGPT — has an X account, sharing screenshots of him replying to the discussion. The exchange sparked a re-read of overlooked details in classic optimizer papers.
More from Research
- Detecting Speech AI Hallucinations from Attention Patterns: Tested on Qwen-2-Audio and Voxtral-3B — burkov · 2026-10-18
- Teen solves years-old math problem in a week with AI and Lean, writes 15-page paper — IgorCarron · 2026-10-18
- SFT + simple search tool lets Qwen3-4B match a ~1T-param model on diagnosis benchmark — Tim_Dettmers · 2026-10-18
- Automating Experiments Makes Premise Choice the Real Bottleneck in Recursive Self-Improvement — Gauri_the_great · 2026-10-18
- Papers with Code Launches Following Feed to Track AI Tasks, Benchmarks and Authors — NielsRogge · 2026-10-18
- Moritz Hardt: AI benchmarks are fundamentally broken but fixable — asusarla · 2026-10-18