vLLM Adds Hardware & Performance Optimizations
vllm_project · x · 2026-07-12
Supplementary hardware and performance notes for vLLM 0.25.0, highlighting cross-platform optimizations:
- NVIDIA / GB300: FlashInfer's fused all-reduce tuned for GB300, Blackwell's NVFP4 decode throughput restored, and FA4-MLA warmup & XQA decode kernels added.
- GLM-5.2 / DeepSeek: fused indexer kernel brings 1.9–3.3% end-to-end throughput boost, reduce-scatter MoE all-reduce improves 3%, and DSv4's tokentoreqindices cache accelerates certain kernels by 5–6x.
- AMD ROCm: torch 2.11 stable ABI, AITER FlashAttention MLA prefill backend, and shared-expert fused kernel.
- Intel XPU: W8A8 FP8 linear kernel, multi-granularity quantization, and uniform-batch CUDA graphs for FA2.
- CPU / Other architectures: Faster unquantized MoE on AArch64, Apple Silicon hang fixed, and support for RISC-V RVV INT4 GEMM & PowerPC fp16.
More from Infra
- Anthropic S-1 reveals $800B+ in off-books future compute commitments — GaryMarcus · 2026-10-27
- ~42% of Anthropic's ~11GW contracted compute is NVIDIA GPUs — ns123abc · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27
- Neon rebuilds its docs search entirely on its own stack: Lakebase Search, Functions, AI Gateway — matei_zaharia · 2026-10-27