Custom vLLM INT8 stack hits 972 tok/s on Qwen 27B with 4x MI100 ($6.5k rig)
1ncehost · reddit · 2026-08-27
A developer released a complete INT8 serving stack for older GPUs without native FP8: built on vLLM, AITER, and a GPTQ INT8 quant (DFlash2), it runs Qwen3.8 27B at 972 tok/s TG / 5,680 tok/s PP on a 4× MI100 rig ($6.5k), versus 15 tok/s on stock vLLM.
The work includes:
- A complete W8A8 INT8 GEMM library tuned for MI100, used everywhere
- INT8 KV cache and INT8 AITER Unified Attention (with Triton fallback, faster than Flash Attention)
- INT8 Mamba/GDN attention and INT8 embedding
- INT8 custom allreduce/allgather optimized for XGMI interlinks
- Many new fused INT8 kernels
Accuracy vetting is unusually rigorous: KLD measured not just per token but per GEMM, attention block, KV lookup, and layer, with diagnostic scripts left in the branch for verification. The author advises checking whether your card's INT8 TOPS exceeds its FP8 FLOPS — if so, this project applies; hardware-agnostic Triton fallbacks benefit MI50/MI210 and older cards too. Everything is open-sourced on GitHub and Hugging Face.
More from Infra
- ASML CFO says US Match Act restricting DUV sales to China would backfire — pstAsiatech · 2026-10-15
- Bloomberg breakdown: a 1GW Texas AI data center may need nearly $3B for overruns alone — rohanpaul_ai · 2026-10-15
- 7 System Design Algorithms to Know: Merkle Trees to CAP/PACELC — goyalshaliniuk · 2026-10-15
- MiniCPM-V 4.7 runs a 35B MoE multimodal model on phones with just 3GB memory — nevrekaraishwa2 · 2026-10-15
- Sipeed MaixStation product page goes live for edge AI dev — xlltt · 2026-10-15
- Nvidia Is Now Worth More Than the Entire Indian Stock Market — theteknosaur · 2026-10-15