KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systems
Hyesung Jeon, Hyeongju Ha, Seoyoung Lee, Beomseok Kang, Jae-Joon Kim
cs.LG
2026-09-28
KVCMAS uses rank-32 anchors and chained workflow correction, matching prior sharing accuracy with 2.0× TTFT vs no sharing at 32K/8 QPS and 3.7× less memory than KVComm.
Prompt-specialized multi-agent systems share one weight set and split roles with different prefixes. New roles need no training, which makes serving cheap. It also breaks KV reuse. The same user query, retrieval, tool return, or upstream reply sits in a shared segment, but each agent's system prompt and glue tokens change the cache for that segment. Every agent prefills the growing overlap again.
Copying the previous agent's cache without correction collapses accuracy: MMLU falls from 67.54 without sharing to 36.45 with naive reuse. Selective recomputation rebuilds only some layers or tokens (DroidSpeak, CacheBlend, RelayCaching). Under the paper's budget, at least half the reuse error stays outside that subset; on MathVista and Video-MME, deviation and attention selection clear about 20% of the error, close to random. Cleaning the rest means running the model again. Delta correction estimates the cache difference from a reference and adds it. GraphFlow-style corrections are tied to previously seen context relations, so new requests fail (MMLU 36.38, near naive reuse). KVComm keeps an online anchor pool that can track changing context, but it stores full-dimensional base caches and per-agent deltas, and it OOMs at 8K and 32K shared tokens. Those delta methods also prefill a separate context-free reference outside the workflow. An approximate first-agent cache then enters the shared text later agents consume.
KVCMAS, from Seoul National University ECE, does two things.
Cross-agent cache deltas live in a low-dimensional subspace. Effective rank, the smallest rank that keeps 90% of singular-value energy, stays below 32 for those deltas, far under the full feature width. Base caches are similarly low-rank, a bit higher. Each anchor stores truncated-SVD factors of its base cache and of each consumer delta: L×r and r×D. Attention still sees a full-dimensional KV tensor; matching and correction use the reconstructed low-rank pieces. Anchor memory drops from O(V N L D) to O(V N r(L+D)). For Llama-3.1-8B in BF16, a 4K-token pool with 20 anchors and 4 consuming agents is about 50 GiB if stored dense.
The second move is chained correction. The first agent densely prefills and keeps an exact cache. On each later edge, the source agent's already materialized shared-segment cache is the reference. There is no extra context-free prefill. The target cache is the position-aligned source cache plus a similarity-weighted mix of low-rank deltas. If the match is unreliable, the target agent densely prefills and inserts a new anchor. Figure 3 shows lower aggregate reconstruction error than a non-chained context-free reference, both across layers and along A0→A1→A2→A3.
On a hop from agent i to j:
Default rank is 32. Agent-specific prefixes are never shared; only shared segments are corrected.
Evaluation reuses KVComm's multi-agent stack (AgentPrune plus GPTSwarm): three task agents plus a reflection agent. Llama-3.1-8B on MMLU and GSM8K, Qwen2.5-Coder-7B on HumanEval, LLaVA-OneVision-7B on MathVista and Video-MME. Target ρ is about 0.8 for LLMs and 0.6 for VLMs. Accuracy is a three-run mean. Latency is on an A100 80 GB at 1 QPS.
| Method | MMLU | GSM8K | HumanEval | MathVista | Video-MME |
| NonShared | 67.54 | 85.60 | 86.96 | 55.90 | 53.00 |
| FullShared | 36.45 | 29.42 | 80.75 | 50.20 | 37.25 |
| RelayCaching | 60.68 | 77.94 | 83.23 | 50.50 | 44.92 |
| KVComm | 62.57 | 80.14 | 85.09 | 53.50 | 51.56 |
| KVCMAS | 65.45 | 79.53 | 84.47 | 53.80 | 52.36 |
KVCMAS leads sharing methods on three of five tasks and trails KVComm by under 0.7 points on GSM8K and HumanEval. The remaining gap to NonShared is real, 6.07 points on GSM8K. Peak memory on GSM8K is 16.10 GB versus KVComm's 59.18 GB, about 3.7×; on MMLU, 16.06 versus 52.40.
Concurrent serving uses a fixed four-agent trace on vLLM, A100. At 32K shared tokens, 8 QPS, and delta ρ=0.8, median TTFT is 61.95 s versus NonShared 125.89 s (2.0×) and 27% below GraphFlow, the next fastest sharing method. KVComm OOMs at 8K and 32K. Holding the rest of KVCMAS fixed, dropping chaining for a context-free reference raises p50 and p90 TTFT by 34% at that load, and cuts MMLU from 65.45 to 59.01 and Video-MME from 52.36 to 50.83. With 6K accumulated generated tokens at 8 QPS, speedups versus NonShared are 2.3× p50 and 2.4× p90.
On an A6000 48 GB, single-stream, 32K, KVComm's anchors are estimated at 435.6 GB; KVCMAS sits near 29 GB. Raising rank from 16 to 32 still helps accuracy; (128, 128) OOMs. Growing the pool from 10 to 20 anchors moves MMLU 65.45→65.80 and Video-MME 52.36→53.15. Scaling agents (2/4/8) or rounds (2/4/8) amortizes the first dense prefill; KVCMAS keeps the lowest TTFT among sharing methods in those sweeps.
This is for serving many prompt-specialized agents on one weight set, not heterogeneous models or LoRA variants. The win is avoided repeated prefill when shared context is long, QPS is high, or more agents and rounds reuse the same tokens. The implementation sits on vLLM, with code released.
Versus KVComm, the savings are anchor memory and the extra reference prefill. Versus selective recomputation, the savings are model execution on selected tokens or layers. Accuracy does not return to dense prefill. It is a 7B to 8B incremental systems result, useful if KV sharing is already on the table and full-dimensional anchors do not fit.
There is no standalone limitations section. The boundaries sit in the setup.
Accuracy stays below NonShared on every task; GSM8K is the largest hole. On Video-MME, one of three reuse-eligible hops has a persistently large cache deviation and always falls back, so ρ saturates near 0.65 no matter how loose τ is. τ is tuned per workload (0.29 on MMLU, 0.60 on Video-MME).
Only 7B to 8B shared-weight, prompt-specialized agents were tested. Heterogeneous models, adapters, and larger scales are untested. Accuracy trajectories average about 2K–7K tokens; the 32K numbers come from controlled serving traces, not the same accuracy runs. Anchor SVD runs after the hop emits its first token, so it is excluded from TTFT by definition and only shows up in end-to-end latency and throughput. The GraphFlow baseline drops that paper's graph scheduler and keeps only preconstructed corrections, which is unfair as a full-system comparison and adequate for the claim that static corrections miss dynamic context.