Fused linear DPO loss merged into NVIDIA TileGym: 7.2x less memory, 1.27x faster
blelbach · x · 2026-10-20
A community PR (#211) has been merged into NVIDIA's official TileGym repo: a cuTile-based fused linear DPO loss (sigmoid).
- Performance: peak memory drops from 61.5 GB to 8.5 GB at 16 pairs (up to 7.2x less), and runs 1.13–1.27x faster than unfused PyTorch (measured at T=1024, H=4096, V=128256, bf16 on an RTX PRO 6000 Blackwell).
- How it works: it never materializes the full logits — the gradient is a scalar times (softmax − onehot), which NVIDIA's tuned cross-entropy kernel already writes in place, so the whole op is one kernel launch plus the GEMMs. At 1 pair memory is slightly worse; savings start at 2 pairs, with a chunked path taking over at 16.
- Debugging story: v1 was 1.5x slower than baseline; profiling showed an fp32 GEMM on CUDA cores eating over half the runtime. The fix (PR #212, fusedlinearcrossentropy) delivered 2.4x speedup and 3.2 GB savings.
More from Infra
- Value grows log-log with compute: why scaling past demand may yield zero returns — DimitrisPapail · 2026-10-20
- Q-CTRL: over 99.5% of qubits sit idle in one fault-tolerant Shor's algorithm blueprint — MJBiercuk · 2026-10-20
- Nscale and Anthropic reportedly in talks for another $45 billion compute deal — TechCrunch AI · 2026-10-20
- Qwen3.8 quant testing on 5090/Strix Halo: Gufo engine fixes tool-calling failures — awitod · 2026-10-20
- SGLang Summit sponsors jump from 11 to 36 in a week; Lip-Bu Tan, Lilian Weng to speak — satnam6502 · 2026-10-20
- If GitHub goes down, how does GitHub push code to GitHub to bring GitHub back up? — VeryWellVersed · 2026-10-20