vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128
vllm_project · x · 2026-10-16
vLLM showcases kernel-level work for Beam long-context serving: fused all-reduce + norm, packed QKV paths, and a tanh softcap kernel.
In GB200 forward microbenchmarks, TP4 batch 1 fell from 4.23 to 3.89 ms, and TP8 batch 128 from 8.15 to 6.47 ms; softcap dropped from 8.51µs to 2.21µs (1 row) and 976µs to 110µs (1,024 rows). These are forward microbenchmarks with dummy weights, not end-to-end serving results.
Related event: vLLM Details Beam Architecture and Kernel-Level Long-Context Optimizations(2 posts)→
More from Infra
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- China rolls out 'token factories' and a 2026-28 token economy action plan — nordicinst · 2026-10-16
- Vercel Sandbox Drives go GA: attach TBs of data in sub-second, fork instantly — cramforce · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16