CachyLLama fork cuts repeated prompt processing in long local-agent sessions
UsualResult · reddit · 2026-07-25
CachyLLama adds persistent KV caching for long local-agent sessions
CachyLLama is a fork of llama.cpp that focuses on one practical bottleneck: repeatedly reprocessing the same prompt context in long agent sessions.
What it does
- Adds an SSD-backed persistent KV cache.
- Adds a system-prompt cache.
- If the beginning of a request matches previously processed context, it can restore that state and only evaluate the changed tail.
- Checkpoints can survive a server restart.
Reported impact
On the project’s own 7840U/780M benchmarks, warm runs are dramatically faster than cold ones:
- 1,243-token prompt: 9.3s cold vs 0.41s warm
- 5,409-token prompt: 43.3s cold vs 0.57s warm
- 15,700-token prompt: 143.1s cold vs 0.99s warm
Important caveat
This does not speed up generation itself. It reduces repeated prompt-evaluation work, which is exactly what hurts long agent workflows on slower hardware.
More from coding & agent
- Codex users report tighter weekly limits and shrinking reset quotas — breath_mirror · 2026-07-25
- Practical ChatGPT Voice and Codex Workflow Tips for Coding — jxnlco · 2026-07-25
- Repost says Codex no longer needs the /goal command — MickeySteamboat · 2026-07-25
- Claude 3.5 Sonnet Builds Interactive 3D V8 Engine from a Single Prompt — chrisfirst · 2026-07-25
- Free open-source `/explain-this` skill targets agent workflow defaults — ivanhzhao · 2026-07-25
- SaaStr says fewer visible agents delivered 4x output after a stack consolidation — HaktanSuren · 2026-07-25