New ApexW optimizer claims lower loss than AdamW with half the VRAM of bitsandbytes
omendUUde · reddit · 2026-11-18
A Redditor released ApexW, a from-scratch training optimizer claiming to beat both AdamW on convergence and bitsandbytes on memory:
- Methodology: deterministic seeds, side-by-side runs against torch AdamW and bnb 8-bit across multiple learning rates on an A100 (bf16, llama 3.1 8B) and a free T4 (fp16, llama 3.2 3B)
- Results: the compact formulation beats AdamW on loss at lower memory; 8-bit mode cuts active VRAM roughly in half vs bnb (0.67GB vs 1.48GB on 8B, optimizer state 120MB vs 320MB) with slightly faster steps and lower training loss at 1e-4/2e-4/5e-4
- Ships compact / 8bit / hybrid modes; full benchmark logs are in reproducible Colab notebooks
The author admits these are only 60-100 step sanity checks; full multi-epoch fine-tunes and downstream evals are next. Promising early numbers, but unproven over long training runs.
More from Infra
- Smuggled Blackwell fingerprints found in Chinese open-weight models; Jensen Huang faces scrutiny — Difficult-Limit-7551 · 2026-11-18
- Local gesture recognition with Liquid AI's d1-2B: ~70ms per frame, zero tokens generated — helloiamleonie · 2026-11-18
- First SGLang Summit draws 2,000+: v1.0 ships, consortium forms with NVIDIA, Google, Meta — aliscodes · 2026-11-18
- TokScale leaderboard: #1 user burned 4.65T tokens, est. $2.43M in cost — olcan · 2026-11-18
- MOSS-Image FP16-fixed weights bring speed boost to aging Turing GPUs — mkrGG · 2026-11-18
- Inside Microsoft's massive Outlook outage: what broke and why fixes took so long — tomwarren · 2026-11-18