Reddit: why I'm giving up on spec decoding in llama.cpp Vulkan — slower and memory-hungry
ParaboloidalCrest · reddit · 2026-11-19
A Reddit user details why speculative decoding in llama.cpp (Vulkan) feels broken: sometimes slower than plain decoding, no deterministic min/max parameter recipes, slower prompt processing, zero benefit at 128k+ context, and memory costs that could fund an extra 128k of context. Based on experience with Qwen 27B.
More from Infra
- Building agents is easy, scaling them for thousands of customers is where the real work hides — Franc0Fernand0 · 2026-11-19
- Why Big Consulting Firms Went Silent After Microsoft's Major Outage — DavidLinthicum · 2026-11-19
- AMD XDNA 2 NPUs become first supported by UnslothAI Desktop, running 100+ agentic apps locally — danielhanchen · 2026-11-19
- Verba-TTS-Indic-Zero: a 7.6MB, 12-language TTS claiming 2000x cheaper than ElevenLabs — alexcovo_eth · 2026-11-19
- Kong launches Volcano, a 'self-healing' cloud for agents after handling 500T AI tokens a year — EXM7777 · 2026-11-19
- Pete Warden on why AI wants to run local: open-weight models challenge cloud frontier labs — bigdata · 2026-11-19