llama.cpp PR adds few-row CUDA MMQ kernel, speeding speculative decoding up to 35%
pmttyji · reddit · 2026-11-12
llama.cpp PR #31533 introduces a CUDA MMQ kernel for quantized weights with 2-24 columns, targeting speculative decoding (7-token drafts = 8 columns) and small batches where neither MMVQ nor batch-oriented MMQ is efficient.
Technical highlights
- Each weight is read and decoded once into int8 MMA fragments, multiplied against all columns on tensor cores
- Supports Q80, Q3K, Q4K, Q5K, Q6K, IQ3S, IQ4NL, IQ4XS on GPUs from Ampere up to DGX Spark
- Includes full reproducible commands: llama-server flags and speedbench.py script, tested with Qwen3.8-27B UD-Q4KM + DFlash2 draft over 80 conversations, 95% CI via paired bootstrap
Measured gains (per-step / throughput)
- RTX 3090 Ti: 1.344x, throughput +34.9% (57.6 → 77.7 tok/s)
- RTX 5090: 1.180x, +20.6% (141.5 → 170.7 tok/s)
- RTX PRO 6000 Max-Q: 1.183x, +16.7%
- DGX Spark: 1.106x, +10.1%
Faster in 80/80 conversations on every GPU tested; author plans to extend to more quant types.
More from coding & agent
- Emil Kowalski ships /learn-taste skill that turns your design corrections into rules — yuwen_lu_ · 2026-11-13
- Self-hosting a Git forge and Actions: why did I wait so long? — qtnx_ · 2026-11-13
- Intent adds Product Manager agent, Cursor provider, and GitLab support — LukeW · 2026-11-13
- "Why use LangChain?" — dev openly asks for reasons, willing to update — curious_vii · 2026-11-13
- Prefixwise: intentionally broke prompt cache 1,772 times to validate root-cause tool — No-Individual159 · 2026-11-13
- InstaBox gives AI agents their own cloud computers, 1M+ boxes in a month — JaynitMakwana · 2026-11-13