First INT4 model lands in int8-vllm: 4x used MI100 hits 222 tok/s decode
1ncehost · reddit · 2026-11-22
- int8-vllm added optimized support for Hy4 Flash, the project's first INT4 model: at INT4 it fits entirely in 128 GB of VRAM with 1.3M tokens of shared context left over.
- Benchmarks on 4x MI100 at C8: 222.4 tok/s tg128, 1655 tok/s pp512, 96.7 tok/s tg256 with 2048 context at concurrency 8. A user reports 1000+ tok/s decode for Qwen3.8 27B on 2x MI210s.
- The author found a 15% GEMM optimization by deferring epilogue work into later fused kernels so MI100's vector ALUs and matrix cores run concurrently; W4A8 GEMM support landed in a companion AITER fork. The INT4 checkpoint was built with NeUQI calibration and is on Hugging Face.
- The project targets AMD's older cards with docker compose files. A used MI100 costs $1,300 on eBay: vs similarly priced new cards expect roughly half prompt-processing speed but double token-generation speed.
More from Infra
- Decision models now take 26% of Vercel gateway requests; new teams pick Anthropic 2:1 — markjeffrey · 2026-11-22
- B300 GPUs clearing at $18/hour and sold out as a compute credit crunch looms — AccBalanced · 2026-11-22
- AMD owns the SC26 floor: Azure HDv2 on 500-core Venice EPYC, Helios open fabric — AccBalanced · 2026-11-22
- SC26: Google pitches TPUs to GPU-default devs, Microsoft shows off HBv5 — AccBalanced · 2026-11-22
- Gary Marcus mocks the biggest data center ever, tied to OpenAI, SB Energy and Nvidia — GaryMarcus · 2026-11-22
- AI stock selloff is a leverage squeeze: neoclouds down ~63%, CoreWeave CDS spikes — tengyanAI · 2026-11-22