PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction
Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim
cs.LG
2026-09-28
PReCache precomputes per-agent low-rank KV on first pass, no retraining. PreLRShared hits 3.1× TTFT and 2.3× throughput vs no sharing; ReBaseShared drops 1.1 points on average.
Multi-LoRA agents share one backbone and hang a small adapter on each role. The context is still shared: the user request, retrieval hits, tool observations, and the previous role's output pile into a growing trajectory. Every time a role takes a turn, it prefills that whole trajectory again and builds its own KV cache. On long-horizon jobs that repeated work is real compute and real memory.
Reusing the previous role's cache wholesale (FullShared) drops accuracy. The cache the current role reads was produced under someone else's adapter, so its own LoRA behavior gets washed out. Existing patches each get stuck. Selective recomputation (DroidSpeak, CacheBlend, RelayCaching) reruns a subset of layers or tokens; at the 30% budget used here, error remains, and raising the budget gives the savings back. Methods that need extra training or a special architecture cannot be dropped onto LoRAs you already have. The same group's earlier LRAgent splits KV into a shared base cache plus a compact per-agent low-rank cache, which holds accuracy, but the current agent still has to run the full backbone over the accumulated context to build its LR cache. Most of the repeated prefill stays.
PReCache, from Seoul National University, is training-free. Both designs share a base cache computed with the pretrained weights and precompute a skinny low-rank (LR) cache per agent. Default LoRA sits on query and value at rank 8; the key projection has no LR term and is shared in full.
PreLRShared does the extra work the first time a context segment is processed. While agent i runs a new chunk, every agent's LoRA down-projection Ak is applied to those hidden states, producing N rank-r LR caches in one pass. Agent j later reuses the shared base cache and its own precomputed LR cache, and the backbone only sees newly added tokens. Full-width prefill over the old trajectory becomes a rank-r matmul.
The shared base cache is still built from agent i's adapter-conditioned hidden states, so it carries the previous role. ReBaseShared turns adapters off, reruns the shared context, and rebuilds the base cache from those neutral hidden states. The appendix gives a geometric condition: if the previous agent's offset from the neutral state is large enough relative to the current agent's (ρ ≥ 2 cos θ), the neutral base cache is closer to what the current agent would have built. Layer-wise measurements on HotpotQA trajectories stay above 1 on average for both LLaMA-3.1-8B and Ministral-8B.
Reconstruction costs extra, so there are two schedules. Lazy prefill (LP) is for single-stream and edge: the current agent generates as usual, then one contiguous adapter-free prefill runs after the turn, kept out of the next agent's TTFT. Double batching (DB) is for concurrent serving: the adapted path and the neutral path share the same batch so a post-turn prefill does not stall the queue. Both schedules produce equivalent cache state; they differ only in when reconstruction runs.
Accuracy uses AutoAct's three roles (plan / action / reflect) on LLaMA-3.1-8B-Instruct and Ministral-8B-Instruct, HotpotQA and ScienceQA at three difficulty bands. Selective-recomputation baselines use a 30% recompute ratio.
| Method | LLaMA HotpotQA | LLaMA ScienceQA | Ministral HotpotQA | Ministral ScienceQA |
| NonShared | 37.52 | 69.19 | 36.72 | 69.07 |
| FullShared | 34.03 (-3.48) | 65.62 (-3.57) | 33.37 (-3.35) | 63.39 (-5.68) |
| BaseShared | 36.97 (-0.55) | 68.64 (-0.56) | 35.43 (-1.28) | 66.74 (-2.33) |
| PreLRShared | 36.32 (-1.20) | 68.22 (-0.97) | 34.77 (-1.95) | 65.78 (-3.29) |
| ReBaseShared | 36.95 (-0.57) | 68.77 (-0.42) | 36.08 (-0.62) | 66.47 (-2.60) |
Averaged over the four settings, ReBaseShared sits 1.1 points below NonShared, in the same band as BaseShared, without most of BaseShared's repeated prefill. PreLRShared drops 1.0 to 3.3 points and still matches or beats selective recomputation. The clear miss is Ministral ScienceQA Easy: ReBaseShared falls from 70.96 to 64.71, worse than PreLRShared's 68.04. Standard deviations across 20 full reruns stay under 0.60 points.
Single-stream efficiency uses controlled traces from 1.9k to 66.4k tokens on one A6000. At 66.4k on LLaMA, NonShared TTFT is 73.18 s and throughput 775.6 tok/s; PreLRShared is 27.08 s and 1685.4 tok/s; FullShared is 23.15 s and 1841.7 tok/s. The paper's headline is up to 3.1× TTFT and 2.3× per-request throughput versus no sharing, with PreLRShared TTFT within 17% of FullShared at the longest trace. ReBaseSharedLP reaches up to 3.0× TTFT and 1.6× throughput. BaseShared stays close to NonShared and DroidSpeak because it still runs the backbone over accumulated context.
On vLLM with one A100, a fixed 17.3k-token trajectory, and 0.5 to 16 QPS, unsaturated PreLRShared and ReBaseSharedDB throughput is about 1.6× and 1.3× NonShared. Peak memory at 66.4k: PreLRShared 27.33 GB, ReBaseShared 27.34 GB, within 2% of FullShared's 27.13 GB and about 23% below DroidSpeak's 35.50 GB. Scaling agents from 4 to 8 at 64k tokens, PreLRShared TTFT moves from 26.36 s to 27.17 s (+3%); NonShared moves from 68.36 s to 90.05 s (+32%).
This is a serving patch for multi-LoRA agents whose adapters already exist and do not share a down-projection. No retraining, no architecture lock-in. Against LRAgent's BaseShared, the saving is the full-length backbone pass that was only there to build the LR cache. Against DroidSpeak-style methods, the saving is full-width KV recompute. LP for the edge, DB for a server. The split is clean.
The tradeoff is equally clean. Accuracy-first means ReBaseShared, and throughput falls from 2.3× to 1.6×. Speed-first means PreLRShared, and the accuracy gap is about one extra point. Default is QV at r=8. Under parameter-matched QKVO (r=4), the key-side LR term has to be expanded before RoPE; at 66.4k, PreLRShared TTFT is 32.55 s, still faster than every baseline except FullShared (24.62 s), with a thinner margin.
Incremental, for a specific setting: prefill-heavy, long shared trajectories, multiple roles. Long-decode workflows where roles barely share context will gain much less. That setting is not measured.
There is no standalone limitations section. The appendix does say ReBaseShared's recovery versus PreLRShared is uneven across difficulty groups, and that the main eval covers only HotpotQA and ScienceQA, where role-specific training trajectories exist.
A few numbers the paper does not dwell on. Neutral reconstruction is worse on Ministral ScienceQA Easy, so the claim that the neutral base is closer to the current agent holds on average, not in every split. The efficiency tables replay controlled traces with matched schedules and token counts; under free generation, cache sharing changes later decisions and trajectory length. Appendix B.1's LLaMA HotpotQA traces average 1200 to 1700 tokens, far below the 66.4k stress test. Two 8B models, one AutoAct three-role workflow, no software-engineering agents, no larger models. The implementation depends on the same group's Flash-LoRA-Attention. Prefix-correction methods collapse to FullShared when prefixes are matched, so they have no separate numbers in the comparison table.