llama.cpp PR rotates K/V read order to boost L2 cache hits, big long-context speedup

jacek2023 · reddit · 2026-11-17

NVIDIA engineer gaugarg-nv opened llama.cpp PR #32407, 'Rotate K/V read order for better L2 cache hit rates,' reordering K/V reads during long-context prompt processing to improve L2 cache hit rates, reportedly yielding significant prompt-processing speedups. The change touches a single file (+64/-16) and has 2 reviews so far — a notable low-level win for local inference.

Original post →

More from Infra

Infra channel →