HybridApexW optimizer: 2x faster than torch.compile AdamW+Muon baseline
omendUUde · reddit · 2026-11-19
A Reddit user shares clean 1000-step benchmarks of their drop-in PyTorch optimizer HybridApexW (APEXW + Muon hybrid) against a torch.compile AdamW+Muon baseline on a 103M-param GPT-2 with WikiText-103:
- AdamW+Muon (torch.compile): 1,076.55s | 16.22GB VRAM | loss 5.52
- HybridApexW (Compact): 533.50s | 16.11GB | loss 4.49
- HybridApexW (8-bit): 537.67s | 15.90GB | loss 4.47
Fusing projection updates and 8-bit state tracking into custom Triton/CUDA kernels yields roughly 2x wall-clock speedup over torch.compile, though VRAM savings are modest (320MB) after fixing an allocator leak. The author candidly admits the first script warmed up JIT on the live model, inflating the baseline to 23.2GB/2,707s; the fix warms up on a clone and clears CUDA cache. HybridApexW handles parameter-group splitting (AdamW for embeddings/norms, Muon for 2D weights) with a single 6e-4 LR. Colab and GitHub links provided; the author asks for feedback on baselines.
More from Research
- Google's GUARD paper: frozen base model, self-editing agent harness gains +8.89 points — omarsar0 · 2026-11-19
- Kezek splits mono two-person conversations into per-speaker tracks for full-duplex dialogue training data — manifestro · 2026-11-19
- Elie Bakouch: current AI research benchmarks overfit and can't track RSI — eliebakouch · 2026-11-19
- Hutter Prize text compression jumps ~10% by overfitting small transformers for arithmetic coding — ricklamers · 2026-11-19
- Sandia Labs' VASt model predicts high-entropy alloy properties from tens of samples — CurieuxExplorer · 2026-11-19
- FoodTruck Bench, an LLM agent business-sim benchmark, lands on Papers with Code — NielsRogge · 2026-11-19