NVFP4 MoE experts force CPU fallback in llama.cpp; requantizing boosts Metal speed 5.8x
teortaxesTex · x · 2026-11-21
Unsloth's Inkling UD-Q8KXL quant is really BF16 dense + NVFP4 experts. llama.cpp's Metal backend lacks NVFP4 kernels, so all 40 expert tensors fell back to CPU — 90 graph splits, 1619 backend transitions, 5 t/s. The author traced the root cause, patched llama-quantize (commit f43e1feb0) to ignore overrides in COPY mode, and requantized only the expert tensors to Q4K/Q50 (18GB output). Result: splits 90→2, CPU nodes 11106→1, pp256 36.6→213.7 t/s (5.8×), tg64 5.86→19.6 t/s (3.4×), chat generation 19-20 t/s, arithmetic sanity check passed.
More from Infra
- 2M canary tokens expose stolen prompts and compromised AI providers — AccBalanced · 2026-11-21
- Unsloth release runs 26.4B Inkling 2 Mini on 16GB RAM, cuts MCP token use 87% — danielhanchen · 2026-11-21
- Seroter Daily Reading List #914: harnesses are cheap to swap, model behavior isn't — rseroter · 2026-11-21
- Inkling 2 Mini tops intelligence-per-joule test: 82.3% accuracy at just 1.17 kJ per query — simonguozirui · 2026-11-21
- Compute overbuild skeptic recants: efficiency gains cut AI compute cost ~4x per year — teortaxesTex · 2026-11-21
- AscendGym paper draws attention as Huawei compute platform research picks up — willknight · 2026-11-21