SemPIC: Learning Semantic Position-Independent KV Caches
Hui Xie, Peng Xiao, Yutong Deng\textsuperscript, Shuoran Dou, Jian Yang, Jinyang Guo
cs.AI
2026-07-30
SemPIC compiles each document's KV cache offline via a LoRA Writer for a frozen decoder, lifting micro-F1 0.53 to 0.60, near 0.62 full recompute, with no online recomputation.
Retrieval-augmented generation and agentic workloads keep reusing the same documents, only the instructions, conversation history, and document ordering change each turn. Prefix caching only hits when content sits behind an identical prefix; change the query or reorder the docs and it misses. Position-independent caching (PIC) takes the other fork: pre-compute each document's KV cache on its own, then stitch them together at request time. The problem is that a document compiled in isolation never saw the context that will precede it in a real request. RoPE re-rotation fixes positional phase but cannot recreate attention to context that did not exist at compile time. Under a standard causal decoder, stitched caches are position-aligned yet contextually incomplete, and quality collapses.
How bad is stitching alone? The paper's No Recompute baseline averages 0.17 micro-F1 across three models and four tasks, against 0.62 for full recomputation. Contextual incompleteness is not a rounding error; it breaks cache reuse outright.
Existing fixes split into two camps. Online selective recomputation (CacheBlend, EPIC, Cache-Craft, CacheClip, ProphetKV, QCFuse) puts part of the document compute back on the request path to repair the cache. Offline adaptation either touches only boundaries (KV Packet adds Header/Trailer tokens around each document) or bolts on an external encoder and a modified Reader interface (COMB). SemPIC's authors first run a diagnostic: KV Packet, a boundary-conditioned baseline, drives attention deviation near reusable-block boundaries very low (pre-region error 0.016 to 0.081), but interior deviation stays high (0.664 to 1.257, below 1 in only 8 of 12 settings). Boundary patching alone is not enough; the document representation itself has to adapt.
The core is Writer-Reader separation. The Writer is the base model plus a LoRA adapter that processes each document independently and produces native per-layer Keys and Values; it exists only during offline cache construction. The Reader is the frozen base model with LoRA disabled, consuming the compiled caches as-is. What gets served is the Reader, with its interface untouched.
The engineering detail that makes this work: disabling the LoRA adapter does not detach the KV tensors. So every layer's K and V plays two roles at once, the reusable representation the frozen Reader consumes, and the differentiable interface that carries loss back to the Writer. That is how 'Reader unchanged' and 'Writer still trainable' hold simultaneously.
Training is behavioral distillation. The teacher is the base model's output distribution given full context; the student is the distribution produced when the frozen Reader reads the Writer-compiled cache, compared via per-position per-head KL divergence at temperature 1. In one line: teach the Writer to produce a cache that makes a frozen Reader, which never saw the real context, behave as if it had. Configuration is LoRA rank 8, scale 16, learning rate 5e-4 with linear decay, 5 epochs, bfloat16, one Writer per model-domain pair.
Cache-hit serving is deliberately ordinary. A compiled SemPIC cache is just a standard model cache: on a hit, assign final logical positions, re-rotate the RoPE keys, run only the request-side inline tokens through the pretrained decoder, interleave compiled and inline caches under the causal mask, and decode autoregressively. The request path has neither a document-side forward pass nor any recomputation, which is the main selling point over the online selective-recomputation camp.
To make long-document Writer training fit in memory, the paper introduces KV Gradient Checkpointing: checkpoint around each Writer layer, keep only the hidden state and that layer's K/V, discard the rest, and recompute on backward. One subtlety: the checkpoint must restore the LoRA-enabled context, otherwise backward would recompute the layer with LoRA disabled and the Writer would drift. A Joint variant co-trains the SemPIC document-level LoRA with KV Packet's boundary embeddings, to test whether document-wide and boundary adaptation are complementary.
The main table is corpus micro-F1 across three models (Llama-3.1-8B, Qwen3-4B, Qwen3-8B) and four tasks (synthetic Biographies, HotpotQA, MuSiQue, NIAH).
| Method | Overall average |
| Full Recompute (quality ceiling) | 0.62 |
| No Recompute (stitch caches directly) | 0.17 |
| KV Packet | 0.53 |
| SemPIC | 0.60 |
| Joint (SemPIC + Packet boundaries) | 0.61 |
SemPIC beats KV Packet in 10 of 12 settings and matches or exceeds it in all 8 Qwen3 settings. Per model, the real separation is on Qwen3: 4B rises from Packet's 0.49 to 0.59 (+0.10), 8B from 0.45 to 0.58 (+0.13); on Llama the two tie at 0.64. The one clear exception is Llama MuSiQue, where SemPIC alone reaches only 0.28 versus KV Packet's 0.37, though Joint recovers to 0.38.
The attention diagnostic moves the same way: SemPIC lowers interior attention deviation Rint from KV Packet's 0.664 to 1.257 down to 0.564 to 0.985, dropping in all 12 settings. On cross-domain transfer (Qwen3-8B), mixture-domain training gives KV Packet 0.51 and both SemPIC and Joint 0.57; across 15 training sources SemPIC beats the corresponding KV Packet average in 14. On memory, a single A100 80GB with gradient checkpointing brings Qwen3-8B peak from 23.8 to 18.0 GiB at batch 1 and 75.8 to 33.2 GiB at batch 8, turning HotpotQA, MuSiQue, and NIAH from out-of-memory into trainable (26.4, 33.1, 43.4 GiB). The cost is slower training at small batch (Biography 34 to 115 minutes); at batch 8 the overhead is negligible.
For RAG and agent serving that reuses a fixed document corpus (knowledge-base QA, document retrieval, agent memory), pre-compiling document KV caches once and reusing them across queries eliminates re-prefilling documents on every request, cutting time-to-first-token. Existing offline PIC either patches only boundaries or requires a modified Reader and an external encoder. SemPIC is the first to show you can train the document-side representation itself while leaving the inference model's parameters untouched. The deployment appeal is exactly that the Reader is unchanged: production runs the original model, only the cache is pre-compiled, with no framework swap and no altered decode path.
It complements rather than replaces online selective recomputation. Offline methods pay the cost up front at cache-build time; online methods return document compute to the request path. The paper is explicit that no method dominates across all tasks, and which camp wins depends on whether a deployment permits online recomputation. Honestly, on Llama it only ties KV Packet; the real gains are on Qwen3, and each domain needs its own trained Writer, so this is not yet plug-and-play.
The authors themselves note that the link between lower attention deviation and higher F1 is only correlational; the evidence is 'descriptive', not a proven cause. The block-local artifact also persists: after SemPIC, the first token's attention-density ratio T1 still reaches 132.5 to 3248.3, versus 0.61 to 23.49 for offsets 2 to 8. The attention sink concentrates at block start, survives adaptation, and merely coexists with better task behavior.
SemPIC alone underperforms KV Packet on Llama MuSiQue (0.28 vs 0.37) and needs the Joint variant to recover. On memory, gradient checkpointing is slow at small batch (about 3x training time), and the paper states plainly that this is a memory-time tradeoff, not a throughput claim, and that the cache-hit serving path is unaffected. Each model-domain pair needs its own trained Writer; a single universal adapter is not provided, and while mixture training transfers well (0.57), cross-domain generalization remains open. The paper also concedes that the reported tradeoff curves do not cover untested recomputation budgets and give no end-to-end cache-miss latency.