CachyLLama caches KV state on SSD and cuts 15,700-token prompts to under 1s
UsedMorning9886 · reddit · 2026-07-25
CachyLLama caches KV state on SSD to speed up local agent workflows
CachyLLama is a llama.cpp fork built for local agentic coding setups such as Aider and Claude Code, where prompt evaluation can dominate runtime.
What it changes
- Persistent on-disk KV cache: conversation checkpoints are stored on SSD and survive restarts or power loss.
- Global system-prompt cache: repeated static prefixes, tool schemas, and long history blocks can be skipped on subsequent requests.
- Hybrid architecture support: it tracks recurrent state as well as attention state for models such as Qwen 3.5/3.6, Gemma 4, GLM-4.7, and DeepSeek-V3.
- Multi-tier storage: hot states stay in RAM, idle sessions move to disk, and kernel readahead is used to overlap I/O with compute.
Benchmarks on AMD Ryzen 7840U / 780M
- 1,243-token prompt: cold start 9.3s, cached 0.41s
- 15,700-token prompt: cold start 143.1s, cached 0.99s
The project does not speed up token generation itself; it mainly removes repeated prompt-processing overhead, which is exactly the bottleneck many local agent harnesses hit.
Related event: CachyLLama Persists KV Cache to Cut Local Agent Overhead(2 posts)→
More from coding & agent
- Swiwal AI coding agent lands in the official OpenBSD ports tree — jedisct1 · 2026-07-25
- Headroom Context Compression: Viral "95% Token Savings" Claim Drops to 20% for Coding Agents — alex_verem · 2026-07-25
- Bindu Reddy says vibe-coded apps with self-improving AI agents can cut costs 10x — bindureddy · 2026-07-25
- llama.cpp native MTP brings 1.4x–2.2x speedups on dense models, little on MoE — UsedMorning9886 · 2026-07-25
- Claude-built search page maps 390 NBER talks to the exact YouTube start time — joshgans · 2026-07-25
- Geek shares hardcore Birkin bag sourcing operation using AI agents — teodorio · 2026-07-25