vLLM Adds Hardware & Performance Optimizations
vllm_project · x · 2026-07-12
Supplementary hardware and performance notes for vLLM 0.25.0, highlighting cross-platform optimizations:
- NVIDIA / GB300: FlashInfer's fused all-reduce tuned for GB300, Blackwell's NVFP4 decode throughput restored, and FA4-MLA warmup & XQA decode kernels added.
- GLM-5.2 / DeepSeek: fused indexer kernel brings 1.9–3.3% end-to-end throughput boost, reduce-scatter MoE all-reduce improves 3%, and DSv4's tokentoreqindices cache accelerates certain kernels by 5–6x.
- AMD ROCm: torch 2.11 stable ABI, AITER FlashAttention MLA prefill backend, and shared-expert fused kernel.
- Intel XPU: W8A8 FP8 linear kernel, multi-granularity quantization, and uniform-batch CUDA graphs for FA2.
- CPU / Other architectures: Faster unquantized MoE on AArch64, Apple Silicon hang fixed, and support for RISC-V RVV INT4 GEMM & PowerPC fp16.
More from Infra
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Direct link: Hugging Face's Ultra Scale Playbook on GPU-cluster LLM training — mdancho84 · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11