Training-free page-sparse decode in llama.cpp: 1.72× faster 128k decoding on an RTX 5090

topppggg · reddit · 2026-11-16

A developer implemented training-free page-sparse decode in a llama.cpp fork: the KV cache is split into 64-token pages, and each KV head reads only the first page, newest tokens, and most-likely-relevant pages — about 21% of the cache per decode step at 128k. Selection follows Quest with MOIRA-style statistics; the contribution is the llama.cpp/CUDA implementation and end-to-end testing.

Results at 128k (RULER: 13 tasks × 20 prompts):

Limits: prefill unchanged (only 4–13% faster end-to-end); no VRAM savings (full cache stays resident plus 3–6% page stats); CUDA-only, batch size 1; slower below 32k, off by default; 1.86× at 256k on UltraLong-8B but with incomplete quality checks.

The author pre-registered gates and reports failures honestly: the first version lost 7.4 RULER points and the first CUDA build hit only 1.05–1.18× until two ggml kernels were fixed (one proposed upstream as llama.cpp #31855). Try it on the sparse-decode branch of topppg/llama.cpp with LLAMASPARSEDECODE=27648, prompts over 32k, model and cache fully on GPU.

Original post →

More from Infra

Infra channel →