Fused kernels cut GB200 forward latency up to 20% in Beam serving microbenchmarks
vllm_project · x · 2026-10-16
Inferact's long-context serving optimizations for Beam include fused all-reduce + norm, packed QKV paths and a tanh softcap kernel. On GB200 forward microbenchmarks, TP4 batch 1 fell from 4.23ms to 3.89ms; TP8 batch 128 from 8.15ms to 6.47ms; the softcap kernel on 1,024 rows dropped from 976.43µs to 110.05µs (8.9×). The authors note these are forward microbenchmarks with dummy weights, not end-to-end serving results.
More from Infra
- China bans nerfed GPU variants while allowing full RTX 5090s, pressuring Nvidia — gandamu_ml · 2026-10-16
- RTX 5090 banned from export to China, but buyers there still want it — gandamu_ml · 2026-10-16
- Neocloud stocks sink on Gartner's "80% will fail" report; analyst calls it garbage — McDonaghMatthew · 2026-10-16
- JapanFold Claims One Tenstorrent Server Beats an Nvidia DGX B200 on Bio Models at a Third of the Price — DavidBennett__ · 2026-10-16
- vLLM ships Day-0 RL support for Beam in SkyRL, +62% LoRA decode throughput — vllm_project · 2026-10-16
- DFlash2 speeds Beam decode up to 3.03× across benchmarks, vLLM reports — vllm_project · 2026-10-16