llama.cpp PR rotates K/V read order to boost L2 cache hits, big long-context speedup
jacek2023 · reddit · 2026-11-17
NVIDIA engineer gaugarg-nv opened llama.cpp PR #32407, 'Rotate K/V read order for better L2 cache hit rates,' reordering K/V reads during long-context prompt processing to improve L2 cache hit rates, reportedly yielding significant prompt-processing speedups. The change touches a single file (+64/-16) and has 2 reviews so far — a notable low-level win for local inference.
More from Infra
- Megaspeed Got $800M in Nvidia Chips After No-Sell Listing, Faces Criminal Probe — GaryMarcus · 2026-11-17
- A 4.5M-parameter TTS model runs fully offline on an ESP32-S3 — ngxson · 2026-11-17
- Over 45 Companies Including Major Names Hit by Simultaneous Outages — Serasul · 2026-11-17
- Indie dev's cloud bill for Pearl Chat is so steep he jokes about a poultry farm — weskambale · 2026-11-17
- Open-Source Tokspur Engine Claims 4x Faster Mac Inference Than Ollama.cpp — SimonSanDigital · 2026-11-17
- ExLlamaV3 MoE offloading on 9070XT doubles prefill to 2606 t/s for Qwen3.6 35B — willeyh · 2026-11-17