Adam vs AdamW: why decoupled weight decay isn't the same as adding λ||w||² to the loss
cneuralnetwork · x · 2026-10-30
Framed as an ML interview question, the author explains the difference between Adam and AdamW. The easy answer is decoupled weight decay; the harder follow-up asks why adding λ||w||² directly to Adam's loss is not equivalent to the decoupled weight decay AdamW uses.
More from Research
- Epoch data: leading chips get ~40% more efficient yearly, but AI power draw keeps climbing — teortaxesTex · 2026-10-30
- Soft-MTP: a one-line fix removes noisy gradients from multi-token prediction — pmttyji · 2026-10-30
- Vicki Boykis on ONNX inference for retrieval and ranking: slides from DataPhilly talk — vboykis · 2026-10-30
- Agentic MRI review takes 9th place out of 6299 teams in Kaggle RSNA knee abnormality detection — kagglingdieter · 2026-10-30
- Meta's RoboJEPA: First Scaling Laws for Multi-Embodiment Robotic World Models — apsarathchandar · 2026-10-30
- Agent harness engineering: lessons from 57 papers — same model, 6x performance gap — Marmelab · 2026-10-30