NVFP4 MoE experts force CPU fallback in llama.cpp; requantizing boosts Metal speed 5.8x

teortaxesTex · x · 2026-11-21

Unsloth's Inkling UD-Q8KXL quant is really BF16 dense + NVFP4 experts. llama.cpp's Metal backend lacks NVFP4 kernels, so all 40 expert tensors fell back to CPU — 90 graph splits, 1619 backend transitions, 5 t/s. The author traced the root cause, patched llama-quantize (commit f43e1feb0) to ignore overrides in COPY mode, and requantized only the expert tensors to Q4K/Q50 (18GB output). Result: splits 90→2, CPU nodes 11106→1, pp256 36.6→213.7 t/s (5.8×), tg64 5.86→19.6 t/s (3.4×), chat generation 19-20 t/s, arithmetic sanity check passed.

Related event: Quantized Inkling Runs on Old Laptops, Metal Inference 5.8x Faster After Requantization(2 posts)→

Original post →

More from Infra

Infra channel →