Novita open-sources Chord W4A16 MoE kernels, up to 2.15x faster Kimi K2.x inference on B300
vllm_project · x · 2026-09-16
Novita AI and the vLLM team have open-sourced Chord, a high-performance W4A16 MoE CUDA operator (BF16 activations, INT4 weights, group-32 scales) built for Kimi K2.x serving shapes.
The core idea: no single MoE kernel fits every request — routed tokens per expert vary by orders of magnitude between prefill and decode, so Chord ships both indexed and grouped paths. The indexed path exposes a Humming-compatible import root, enabled via --quantization humming on compatible vLLM revisions; grouped integration is still WIP.
Measured per-layer gains vs public Humming:
- 1.11–1.20x on H200 EP8 prefill; 1.17–1.33x on H200 TP8 single-instance serving
- 1.16–1.24x on H200 EP8 decode, with the down stage reaching 1.31x
- Up to 2.15x on B300 EP8 decode vs Humming's untuned default (no SM100/SM103 tuning table ships with public Humming)
- Grouped paths: 1.00–1.35x across H200 EP8/EP16/EP32 scenarios
Kernels and full benchmark tables are open-sourced; details in the vLLM blog.
More from Infra
- German data center boom may triple Hessen power demand as DFKI works on energy-efficient AI — FlorianGallwitz · 2026-09-16
- VCs say the GPU shortage is really a capital access problem: 30% down payment prints money — sarahdrinkwater · 2026-09-16
- Developer builds Linux subsystem for Haiku with OCI runtime and container support — unixterminal · 2026-09-16
- Anthropic signs lease at Queensland data center park costing ~$30B, online 2027 — DigitalColmer · 2026-09-16
- Transformers models now run natively in vLLM with no port required — pcuenq · 2026-09-16
- TSMC builds 20 fabs yet can't meet AI demand as labor shortage slows expansion — emmanuelvivier · 2026-09-16