Experiments confirm AdamC's optimal weight decay (0.5) exceeds AdamW's typical 0.1–0.2
aaron_defazio · x · 2026-11-17
New runs confirm that AdamC's optimal weight decay can be larger than AdamW's: sweeping wd from 0 to 1 on a qwen3-0.5m training run, AdamC bottoms out at wd=0.5 (val loss 3.1725), beating the 0.1–0.2 range typical for AdamW. Author Aaron Defazio notes it matches theoretical expectations, with full loss curves shared.
More from Research
- PyTorch to ship new stateless RNG API after giffmana's year-long design push — giffmana · 2026-11-17
- 250 agents train a single algorithm: 'program learning' recreates world maps efficiently — willdepue · 2026-11-17
- Security researcher argues peer review is broken, vetted blog posts may replace it — EarlenceF · 2026-11-17
- Fields Medalist Maynard stunned by OpenAI's quasi-Riemann hypothesis proof — emax · 2026-11-17
- Vismatch migrates to Kornia 0.9.2: LoFTR 47% faster, xfeat-star accuracy jumps — ducha_aiki · 2026-11-17
- 100,000 hours of egocentric skill data open-sourced on Hugging Face, free for all — lukas_m_ziegler · 2026-11-17