Use Flashinfer for VLLM on Ampere Hardware
mayo551 · reddit · 2026-08-17
Through trial and error, a user found that Flashinfer is the best attention backend for VLLM on Ampere hardware. Unlike FA2 and tritonattn, Flashinfer maintains speed at high context lengths (>70k) and works seamlessly with MTP. It achieves 100+ T/S on a 4x3090 setup with Qwen 3.8 27B. Users should verify the backend in the logs and use --language-model-only for Gemma models.
More from Infra
- First AA-AgentPerf results: NVIDIA Vera Rubin NVL72 delivers up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- Vera Rubin NVL72 first benchmarks: up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- NVIDIA Vera Rubin NVL72 hits up to 5.7x GB300 throughput per MW on agent benchmark — ArtificialAnlys · 2026-10-16
- 14 open-source LLM observability tools to know in 2026, from Langfuse to Laminar — TheTuringPost · 2026-10-16
- Local AI is a routing problem, not a privacy feature: where each task should run — bigdata · 2026-10-16
- Moving agent work from GPUs to CPUs for durable, predictable actions — cramforce · 2026-10-16