RAG vs CAG: KV Cache Cuts LLM Costs by 90%
blaizedsouza · x · 2026-08-21
The post compares Retrieval-Augmented Generation (RAG) with Cache-Augmented Generation (CAG), noting that RAG's constant vector DB hits are expensive and slow. CAG stores static info in the model's KV memory to reduce redundancy. The optimal approach combines both: caching static data (policies, docs) in KV memory while fetching dynamic data via retrieval. This method cuts input token costs by 90% and speeds up inference up to 14x. Links to Stanford research and the LMCache open-source project are included.
More from Infra
- Haight Street Compute Market: a one-day trading floor for GPU deals in SF — ctjlewis · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- LMA brings day-0 model support to TPU for the first time, spanning NVIDIA, AMD, Intel via SGLang — ying11231 · 2026-10-16
- Semtech's analyst-day slide cleanly explains the four signal-integrity impairments — BenBajarin · 2026-10-16
- Fortytwo Engine boosts Qwen3.8-27B accuracy 7% and cuts tokens 25% via Swarm Inference — rohanpaul_ai · 2026-10-16
- Berkeley scholars: concentrated chip supply chain makes a hardwired frontier AI pause feasible — wfithian · 2026-10-16