RAG vs CAG: KV Cache Cuts LLM Costs by 90%
blaizedsouza · x · 2026-08-21
The post compares Retrieval-Augmented Generation (RAG) with Cache-Augmented Generation (CAG), noting that RAG's constant vector DB hits are expensive and slow. CAG stores static info in the model's KV memory to reduce redundancy. The optimal approach combines both: caching static data (policies, docs) in KV memory while fetching dynamic data via retrieval. This method cuts input token costs by 90% and speeds up inference up to 14x. Links to Stanford research and the LMCache open-source project are included.
More from Infra
- DecagonAI achieves sub-30ms latency for real-time TTS — dhruv2038 · 2026-08-21
- US union warns: New England data center bans could kill thousands of jobs — Polymarket · 2026-08-21
- Waymo Cuts Hardware to $20k, Tesla Aims for Cybercab COGS Under $20k — JOBhakdi · 2026-08-21
- Cursor's Git Storage System: S3 as Source of Truth, Local Disk as Cache — xennygrimmato_ · 2026-08-21
- Unsloth Desktop Update: Auto Compaction and LAN Remote Access — danielhanchen · 2026-08-21
- AMD ROCm 10.1 fixes major issues: LLaMA.cpp runs flawlessly on RDNA2 — smellof · 2026-08-21