Training-free page-sparse decode in llama.cpp: 1.72× faster 128k decoding on an RTX 5090
topppggg · reddit · 2026-11-16
A developer implemented training-free page-sparse decode in a llama.cpp fork: the KV cache is split into 64-token pages, and each KV head reads only the first page, newest tokens, and most-likely-relevant pages — about 21% of the cache per decode step at 128k. Selection follows Quest with MOIRA-style statistics; the contribution is the llama.cpp/CUDA implementation and end-to-end testing.
Results at 128k (RULER: 13 tasks × 20 prompts):
- Qwen3-30B-A3B Q4KM on RTX 5090: 46.1→79.2 tok/s, 1.72×, RULER −1.0, passes;
- Same model Q80 on modded 4090 48GB: 1.41×, −0.6 passes;
- Qwen3-8B-YaRN on 3090: 1.68×; Llama-3.1-8B: 1.64×, both pass;
- Llama-3.2-3B: 1.38× but fails at −4.5 (11.5 points lower on multi-needle tasks); not recommended for multi-fact retrieval.
Limits: prefill unchanged (only 4–13% faster end-to-end); no VRAM savings (full cache stays resident plus 3–6% page stats); CUDA-only, batch size 1; slower below 32k, off by default; 1.86× at 256k on UltraLong-8B but with incomplete quality checks.
The author pre-registered gates and reports failures honestly: the first version lost 7.4 RULER points and the first CUDA build hit only 1.05–1.18× until two ggml kernels were fixed (one proposed upstream as llama.cpp #31855). Try it on the sparse-decode branch of topppg/llama.cpp with LLAMASPARSEDECODE=27648, prompts over 32k, model and cache fully on GPU.
More from Infra
- Oxide's second-gen Cloud Computer ships with Micron memory and storage; Micron invests — Sethwinterroth · 2026-11-16
- AMD MI455 beats Nvidia Rubin on raw specs, claims viral chip debate — zephyr_z9 · 2026-11-16
- llama.cpp offloads CI caches and runners to Hugging Face to survive 35 PRs/day — ggerganov · 2026-11-16
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- ZGX Fury stress-tested: 351 tok/s on Qwen3.8-Flash-Next, offline agents and local RAG demos — Sam Witteveen · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16