Mixed-Precision Quantization for vLLM: Qwen3.6 27B Models at 2-8bit

victormustar · x · 2026-08-05

Developer bnjmnmarie successfully implemented mixed-precision quantization (2, 3, 4, 8-bit) for vLLM, combining AutoRound, LLM Compressor, a custom repacker, and vLLM 0.26's Humming Kernel. Released multiple quantized Qwen3.6 27B models (3.0/3.5/3.8/4.3-bit). The 3.8-bit and 4.3-bit versions work well; 3.5-bit and 3.0-bit are still being tested. This work aims to bring llama.cpp-like layer-wise quantization flexibility to vLLM, enabling smaller models. Full evaluation results will be published on the blog next week.

Original post →

Mixed-Precision Quantization for vLLM: Qwen3.6 27B Models at 2-8bit

More from coding & agent