CachyLLama fork cuts repeated prompt processing in long local-agent sessions
UsualResult · reddit · 2026-07-25
CachyLLama adds persistent KV caching for long local-agent sessions
CachyLLama is a fork of llama.cpp that focuses on one practical bottleneck: repeatedly reprocessing the same prompt context in long agent sessions.
What it does
- Adds an SSD-backed persistent KV cache.
- Adds a system-prompt cache.
- If the beginning of a request matches previously processed context, it can restore that state and only evaluate the changed tail.
- Checkpoints can survive a server restart.
Reported impact
On the project’s own 7840U/780M benchmarks, warm runs are dramatically faster than cold ones:
- 1,243-token prompt: 9.3s cold vs 0.41s warm
- 5,409-token prompt: 43.3s cold vs 0.57s warm
- 15,700-token prompt: 143.1s cold vs 0.99s warm
Important caveat
This does not speed up generation itself. It reduces repeated prompt-evaluation work, which is exactly what hurts long agent workflows on slower hardware.
More from coding & agent
- Rethinking dependencies in the agent era: drop libs that save 20 lines of code — gethackteam · 2026-10-26
- RustGIS hits 2,000 GitHub stars in two weeks with agent-first architecture — giswqs · 2026-10-26
- Vercel Ships 12 New Products at Its 2026 Conference as It Races Rivals for the Agent Entry Point — FinanceYF5 · 2026-10-26
- Build an AI agent from scratch in under 50 lines of pure Python: ReAct loop tutorial — adnan_hashmi · 2026-10-26
- Dense instance segmentation via Muse Wonder + SAM 3.1 on Meta's Model API — nikhilaravi · 2026-10-26
- Security tip: use AI agents to build a NIST CSF knowledge base for incident response — prasanna_says · 2026-10-26