SkewAdam cuts MoE optimizer memory by 97.4% and fits 6.78B on a 40GB GPU

Kooky-Ad-4124 · reddit · 2026-07-22

A new preprint introduces SkewAdam, a tiered optimizer designed to slash the memory cost of Mixture-of-Experts training.

• The core idea is to allocate optimizer state by parameter type instead of treating everything equally.
• In the proposed setup, the backbone gets momentum plus factored second moment, experts get factored second moment only, and the router gets exact second moment.
• The paper reports optimizer-state memory falling from 50.6 GB to 1.29 GB, a 97.4% reduction.
• Peak training memory drops from 81.4 GB to 31.3 GB, allowing a 6.78B MoE to fit on a single 40GB GPU without hurting convergence or router stability.

Original post →

More from Infra

• LLM inference benchmarks can mislead teams before production traffic hits — Suspicious_Orchid770 · 2026-07-22
• Tokenizers v1 heads to SIMD refactors after claims of 500–1000x speedups — vanstriendaniel · 2026-07-22
• Cloudflare-style infra is making agent-first apps feel radically easier to build — threepointone · 2026-07-22
• OpenFPM CUDA-style kernels now run on Apple Silicon GPUs via Metal — Scobleizer · 2026-07-22
• Samsung is said to weigh a €1B Mistral investment at a €20B valuation — rohanpaul_ai · 2026-07-22
• NVIDIA and ETH Zürich cut small-message AllReduce latency by deleting barriers — thoefler · 2026-07-22

Infra channel →