CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG
Gyuwan Kim, Cheoneum Park, Tao Yang
cs.CL, cs.AI, cs.IR, cs.LG
2026-08-08
CoinRAG reuses nugget KV caches sliced from precomputed chunk caches, beating TurboRAG 5.3% F1 on LongBench multi-hop QA with shorter context at sub-100ms first-token latency.
Retrieval-augmented generation (RAG) feeds an LLM external documents at query time, but the retrieved context often runs to thousands of tokens, and simply encoding that context in the prefill stage dominates both latency and compute. For interactive Q&A, time-to-first-token (TTFT) past about 100 ms starts to feel laggy.
The standard fix for slow prefill is chunk-level KV cache reuse: precompute each document chunk's key-value cache offline, then splice those caches together at query time to skip re-encoding. TurboRAG, CacheBlend, and KVLink all follow this route. The trouble is granularity. A 512-token chunk often contains only a short span relevant to the current question; the rest is redundancy and noise that slows decoding and hurts accuracy.
CoinRAG wants both: the low latency of cache reuse, and relevance filtering finer than a chunk.
The core mechanism: instead of reusing whole chunks, slice "information nuggets" (short, fact-bearing text spans, about a dozen words each) out of the chunks and reuse their KV caches. The steps:
Two design choices deserve explanation because they are what make the method work.
First, nuggets are sliced from the chunk cache rather than encoded in isolation. Some prior work encodes nuggets as standalone text, which loses their position-within-context information. CoinRAG's sliced KV retains the contextual grounding from the original full-chunk pass, which is why it is called "contextualized." In ablation, switching to isolated encoding drops F1 by 6.3, 4.9, and 3.9 points on the three datasets. This is what lets fine-grained filtering proceed without accuracy loss.
Second, retrieval is two-stage rather than searching nuggets directly over the whole corpus. Direct nugget retrieval has too many candidates and too much noise; chunk-first, nugget-second confines the search to query-relevant chunks, for a 9.6% to 17.5% relative F1 gain. Position alignment via RoPE rotation adds another 3.0% to 8.5% F1 under the tight 75 ms budget.
Evaluation is on three LongBench multi-hop QA datasets (HotpotQA, 2WikiMQA, MuSiQue), with Qwen2-7B-Instruct as the backbone and BGE-M3 as the retriever. Baselines are standard RAG, TurboRAG (chunk-level reuse), CacheBlend (selective recomputation), and KVLink (cross-chunk attention links).
Under a P99 TTFT budget of 100 ms:
| Method | HotpotQA | 2WikiMQA | MuSiQue | Avg F1 | Avg TTFT (ms) | Avg length |
| Standard RAG | 40.1 | 28.5 | 21.0 | 29.9 | 74 | 723 |
| TurboRAG | 49.1 | 42.2 | 27.4 | 39.6 | 75 | 855 |
| KVLink | 49.0 | 38.7 | 26.8 | 38.2 | 65 | 1117 |
| CoinRAG | 51.4 | 42.4 | 31.4 | 41.7 | 65 | 465 |
CoinRAG averages 41.7 F1, 5.3% above the next-best TurboRAG (39.6), at lower latency (65 vs 75 ms) and with a shorter assembled context (465 vs 855 tokens, 1.84x shorter). It traces a new Pareto frontier on the accuracy-versus-latency plane.
With the latency limit lifted, the gap holds: CoinRAG 42.7 vs TurboRAG 40.6 (5.2%), at 6.8x shorter average context (580 vs 3955 tokens). But once the budget relaxes to about 160 ms, standard RAG and TurboRAG catch up, and KVLink matches on HotpotQA. CoinRAG's advantage concentrates in the tight-budget regime and converges as the budget widens.
For teams running interactive RAG under a sub-100 ms first-token target, CoinRAG offers the best accuracy-versus-cost trade-off in that regime. A 1.84x to 6.8x shorter context means not only faster prefill but lower KV memory and cheaper decoding, which translates directly into serving cost under concurrency.
The more transferable idea is the trick of slicing contextualized nuggets from a precomputed chunk cache. It shows that chunk-finer relevance can be had without paying for re-encoding and without losing contextual grounding. The mechanism is not specific to multi-hop QA; any setting that can afford to pre-encode full documents but wants to use only fragments online could borrow it.
The paper's own caveats, the notable ones:
A gap the paper does not address: all evaluation is English multi-hop QA on a single 7B backbone. Whether the method generalizes to Chinese, to larger models, or to non-QA generation (summarization, code completion) is left open.