ApexW: a Triton PyTorch optimizer that halves optimizer-state memory, 1.5x faster steps
Ready_Parking_5435 · reddit · 2026-10-27
A developer released ApexW, a PyTorch optimizer using compact optimizer states and a Triton execution path. On workloads with a few large tensors it's 1.5x faster than PyTorch's fused AdamW and cuts optimizer-state memory roughly in half — useful for LLM fine-tuning workloads hitting OOM, enabling larger batches or models.
Honest trade-offs disclosed:
- Many small tensors suffer from Python launch latency: at 128M params, PyTorch native 16.1 ms/step vs ApexW 76.0 ms/step (0.21x)
- End-to-end, PyTorch still wins on the MLP test due to deep C++ integration
- Auto execution mode isn't consistently fast yet
Code on GitHub; author seeks feedback on benchmarks and implementation.
More from Infra
- Lovable invokes 753.6B Cloudflare Workers monthly, migrated from Next.js to TanStack Start — ritakozlov · 2026-10-27
- Prague lab BottleCapAI releases ThinkingCap models cutting thinking tokens up to 69% with negligible accuracy loss — JFPuget · 2026-10-27
- Local CPU vector mesh beats BM25 routing 100k tools: 70.7% top-1 at 9.6ms — Key_Coconut_9269 · 2026-10-27
- 4 coding agents on one RTX 3090: custom CUDA engine hits ~290 tok/s with Qwen3.8-27B — Adorable_Weakness_39 · 2026-10-27
- Google ships llm-d 1.0, an open Kubernetes-native inference stack, demoed on 1,000 agents — dl_weekly · 2026-10-27
- Transformers v5.21 ships FP8 KV-cache, MoE sharding, and new TTS models — LysandreJik · 2026-10-27