HybridApexW optimizer: 2x faster than torch.compile AdamW+Muon baseline

omendUUde · reddit · 2026-11-19

A Reddit user shares clean 1000-step benchmarks of their drop-in PyTorch optimizer HybridApexW (APEXW + Muon hybrid) against a torch.compile AdamW+Muon baseline on a 103M-param GPT-2 with WikiText-103:

Fusing projection updates and 8-bit state tracking into custom Triton/CUDA kernels yields roughly 2x wall-clock speedup over torch.compile, though VRAM savings are modest (320MB) after fixing an allocator leak. The author candidly admits the first script warmed up JIT on the live model, inflating the baseline to 23.2GB/2,707s; the fix warms up on a clone and clears CUDA cache. HybridApexW handles parameter-group splitting (AdamW for embeddings/norms, Muon for 2D weights) with a single 6e-4 LR. Colab and GitHub links provided; the author asks for feedback on baselines.

Original post →

More from Research

Research channel →