CachyLLama fork cuts repeated prompt processing in long local-agent sessions
UsualResult · reddit · 2026-07-25
CachyLLama adds persistent KV caching for long local-agent sessions
CachyLLama is a fork of llama.cpp that focuses on one practical bottleneck: repeatedly reprocessing the same prompt context in long agent sessions.
What it does
- Adds an SSD-backed persistent KV cache.
- Adds a system-prompt cache.
- If the beginning of a request matches previously processed context, it can restore that state and only evaluate the changed tail.
- Checkpoints can survive a server restart.
Reported impact
On the project’s own 7840U/780M benchmarks, warm runs are dramatically faster than cold ones:
- 1,243-token prompt: 9.3s cold vs 0.41s warm
- 5,409-token prompt: 43.3s cold vs 0.57s warm
- 15,700-token prompt: 143.1s cold vs 0.99s warm
Important caveat
This does not speed up generation itself. It reduces repeated prompt-evaluation work, which is exactly what hurts long agent workflows on slower hardware.
More from coding & agent
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11