llama.cpp Fixes MTP Crashes on Legacy GPUs
TheSpicyBoi123 · reddit · 2026-07-15
The author patched the CUDA backend of llama.cpp, fixing a BF16 selection error that could trigger crashes during MTP (multi token prediction) on pre-Ampere GPUs.
The Issue
On older architecture cards (like Kepler / Maxwell / Pascal / Turing) that lack BF16 support, the original implementation might incorrectly trigger the BF16 path, causing a cuBLAS crash. This was reproduced on a Kepler card.
The Fix
- GPUs supporting BF16: Continue using BF16
- No BF16 support, but fast FP16: Fallback to FP16
- Older GPUs (e.g., Kepler): Fallback to FP32
Results
Tested on a Tesla K40c:
- stock llama.cpp: 22 tok/s
- MTP disabled: 17.5 tok/s
- MTP enabled (2 tokens): 25 tok/s
This means legacy GPUs can now run MTP more stably without sacrificing speed gains.
More from Infra
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22
- Gavin Baker says Nvidia’s $630B figure would be system revenue, not all Nvidia’s — GavinSBaker · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22
- Report says Nvidia could build 1,000 Vera Rubin racks a day, implying $630B quarterly at system level — GavinSBaker · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22