Experiments confirm AdamC's optimal weight decay (0.5) exceeds AdamW's typical 0.1–0.2

aaron_defazio · x · 2026-11-17

New runs confirm that AdamC's optimal weight decay can be larger than AdamW's: sweeping wd from 0 to 1 on a qwen3-0.5m training run, AdamC bottoms out at wd=0.5 (val loss 3.1725), beating the 0.1–0.2 range typical for AdamW. Author Aaron Defazio notes it matches theoretical expectations, with full loss curves shared.

Original post →

More from Research

Research channel →