SemPIC: Cutting RAG Inference Costs by Pre-compiling Documents into KV Caches
TheTuringPost · x · 2026-08-06
SemPIC introduces a novel approach to reduce RAG inference costs by treating reusable documents as compiled inference artifacts—specifically, position-independent KV caches that can be reused across requests.
- Mechanism: A LoRA-enabled "Writer" generates position-independent, per-layer KV caches offline. The unchanged base model ("Reader") can then read and combine these caches without recomputing the original document tokens.
- Problem Solved: Traditional raw-text RAG repeatedly processes the same documents. While prefix caching exists, KV caches are generally not portable across changing contexts. SemPIC overcomes this limitation.
- Performance: Across Llama-3.1-8B and two Qwen3 models on four retrieval tasks, SemPIC achieved a micro-F1 of 0.60, closely approaching full recomputation (0.62) and significantly outperforming previous caching methods (0.53).
- Limitation: Each model and domain still requires its own adapter training.
Related event: SemPIC Pre-compiles Document KV Caches to Cut RAG Costs(2 posts)→
More from Infra
- Musk: AI Compute Growing 10x Every Six Months, Traditional Data Centers Will Transition — r0ck3t23 · 2026-08-06
- Minimax H3 VRAM Paradox: Higher Resolution Actually Uses Less Memory? — gerentedesuruba · 2026-08-06
- SpaceX Ramps Up Tesla Megapack Purchases to Power AI Data Centers — Scobleizer · 2026-08-06
- Running MiniMax H3 on RTX 5090: 12 Mins for an 8-Sec 1080P Video — jefharris · 2026-08-06
- Uber CTO: AI Usage Quadrupled While Token Costs Dropped — Scobleizer · 2026-08-06
- tinygrad Team Building GPU-native RTOS and Shader Compiler — mohbibi_ · 2026-08-06