vLLM and Ant Group open-source Humming: quantized inference kernels with up to 5x speedups
vllm_project · x · 2026-10-21
The vLLM project and Ant Group are open-sourcing Humming, a general-purpose high-performance kernel library for quantized inference, tackling the combinatorial problem of "every format × every GPU × every workload." Benchmarks show up to 5x kernel speedups and +14–43% end-to-end MoE throughput.
Key points:
- Broad device support: NVIDIA GPU architectures from Turing onward, plus CUDA-compatible devices like T-Head's Zhenwu 810E.
- Unified quantization architecture: supports diverse weight/activation format combinations for dense and MoE workloads — multiple integer/float types, bit widths, and scaling schemes (WNA16, W8A8, W4A8, W4A4, INT4 vs NVFP4, etc.).
- Performance: significantly outperforms existing vLLM Marlin, FlashInfer, and DeepGEMM implementations across representative workloads.
- Context: quantization cuts memory and data movement — critical for MoE models like DeepSeek-V3 (671B parameters, only 37B active per token, yet all weights must be stored).
Repo: github.com/vllm-project/humming
More from Infra
- Google: Gemini 4 Argon processed ~25 trillion tokens in first 72 hours — OfficialLoganK · 2026-10-21
- Cactus runs local speech-to-actions on a $10 ESP32 with sub-30MB models — alexcovo_eth · 2026-10-21
- FLUX 3 Klein on RTX 3090: int8 Doubles Speed, but FLUX.2 klein 4B Wins on Quality — strata2signal · 2026-10-21
- Oxide raises $445M Series D just months after its $200M Series C — Sethwinterroth · 2026-10-21
- Musk: xAI to bring 200K Nvidia GB200s online monthly, plus 10GW of compute by end-2027 — beffjezos · 2026-10-21
- Musk's take on space datacenters: run chips hot, since vacuum can't cool them anyway — sudoraohacker · 2026-10-21