KV-cache Grafting: Freezing Small Models Makes Them Stronger
Corbenci · hf · 2026-07-17
## Making Frozen Small Models Cheaper and Stronger with KV-cache Grafting This work proposes a method to boost small model capabilities and reduce inference costs without changing weights: storing verified knowledge as **byte-exact** KV states, which are later grafted back into new inference contexts. ### Core Features - Recovery is **bit-level identical**: Under fixed deterministic configurations, the logits after grafting are byte-for-byte identical to recomputed results - The paper verified byte-exactness on **12B** and **31B** model scales across two GPU targets - Provided committed input/output hashes for easy verification ### Results - On **AIME 2025**, frozen Gemma-4-12B improved from **80.0%** to **93.3%** - On recurring cases, 8 problems originally unsolvable within a 401,026 token budget were solved using cached verified answers within **61 decode tokens** - The paper claims this equates to **6,574x fewer tokens** and roughly **8,700x lower energy consumption** - Usable context length expanded from **32,768** to **2,854,766 tokens** without increasing extra accelerator memory The authors describe this as a "verified-knowledge flywheel": knowledge is verified once and reused multiple times.
Related event: KV-Cache Grafting Boosts Frozen Small Models(2 posts)→
More from Infra
- OxDeAI opensource protocol moves AI-agent policy checks before execution — docybo · 2026-07-21
- LLM cost per request tripled because retries and prompt bloat, not traffic — Lance_Saul_85 · 2026-07-21
- NVIDIA CMP 170HX sellers on Alibaba allegedly doubled prices after jailbreak news — Hannibalj2ca · 2026-07-21
- AI performance is increasingly limited by materials science, not just compute — nordicinst · 2026-07-21
- A GLM-5.2 inference debate asks how 750B parameters can exceed 1 token per second — francoisfleuret · 2026-07-21
- Why vector databases slow AI agents down after constant writes — PrajwalTomar_ · 2026-07-21