int8-vllm fork rewrites serving stack, hits 1000+ tok/s on 4x MI100
1ncehost · reddit · 2026-11-05
A Reddit maintainer released int8-vllm, a fork that rewrites most of the vllm serving stack for int8 inference, targeting older AMD cards (MI100/MI50/MI210/V620/Vega-RX7000) and int8-era NVIDIA cards (Volta/Ampere). Highlights: 4x MI100 or 2x MI210 runs Qwen3 27B above 1000 tok/s decode; fixed multiple vllm/rocm numerics bugs with quality near BF16 including int8 KV cache; fused NVFP4/IQ2xxs dequant plus a configurable int8 weight cache to run Nvidia Bolt checkpoints; AMD Infinity Storage direct-to-GPU loading cut Bolt load time from 7 min to 2. Stable in production for weeks.
More from Infra
- Microsoft rejects multibillion-dollar AI deal with UK's Nscale, FT reports — nordicinst · 2026-11-05
- Microsoft Details Running Coding AI Locally on 128GB RTX Spark Windows PCs Without Token Fees — lee_stott · 2026-11-05
- On 64GB combined memory, is there anything better than Qwen 3.8 27B yet? — xdcfret1 · 2026-11-05
- Baidu open-sources ERNIE 5.5 Flash Base: 280B MoE with 14B active params and 1M context — Baidu_Inc · 2026-11-05
- Revolut self-hosts GLM on 56 B200s, cutting coding token cost to ~$0.30/M — alex_verem · 2026-11-05
- Dev: skip any GPU under 256GB for local AI, prices are about to fall — AIFlow_ML · 2026-11-05