CachyLLama fork cuts repeated prompt processing in long local-agent sessions
UsualResult · reddit · 2026-07-25
CachyLLama adds persistent KV caching for long local-agent sessions
CachyLLama is a fork of llama.cpp that focuses on one practical bottleneck: repeatedly reprocessing the same prompt context in long agent sessions.
What it does
- Adds an SSD-backed persistent KV cache.
- Adds a system-prompt cache.
- If the beginning of a request matches previously processed context, it can restore that state and only evaluate the changed tail.
- Checkpoints can survive a server restart.
Reported impact
On the project’s own 7840U/780M benchmarks, warm runs are dramatically faster than cold ones:
- 1,243-token prompt: 9.3s cold vs 0.41s warm
- 5,409-token prompt: 43.3s cold vs 0.57s warm
- 15,700-token prompt: 143.1s cold vs 0.99s warm
Important caveat
This does not speed up generation itself. It reduces repeated prompt-evaluation work, which is exactly what hurts long agent workflows on slower hardware.
More from coding & agent
- Warp's six non-engineering teams all run on Linear and Claude Code — mon__lim · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- Anthropic researcher: 99% of engineers now run swarms of 300+ self-improving agents — AlishaOutridge · 2026-09-11
- Gergely Orosz: Shipping 10x PRs With AI Agents, Sites Fill With Small Regressions — ducha_aiki · 2026-09-11
- Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11