When Latent Agents Lie: KV-Cache Integrity in Multi-Agent LLM Collaboration
Luís Brito, Carlos Baquero
cs.MA
2026-06-27
Qwen3-4B full-KV collaboration hits EM/F1 0.338/0.486 vs 0.231/0.369 text on HiddenBench. Tampered KV can zero scores; HMAC catches transport swaps, not malicious endpoints.
Multi-agent LLM systems often split evidence across specialists. Some stacks now ship more than a short message: a specialist can hand its full KV-cache to the coordinator. That cache is the model's already-computed key and value tensors, so it can carry evidence the visible text dropped. It is also opaque. A downstream model can consume a swapped, rescaled, or optimized cache while the public commitment still reads as harmless.
The evaluation setting is split-evidence QA. No specialist sees the full record. The coordinator has to assemble an answer from other agents' state. Checking the visible commitments does not tell you whether the hidden object still matches that evidence.
The protocol runs planner, specialists, an optional verifier, then a coordinator. Each specialist gets a private evidence partition, emits a short visible commitment (a public text claim the verifier can score), and in latent branches transports prompt-token KV plus generated latent-thought KV. Latent-thought KV is cache written while the specialist generates internal thought tokens, separate from the prompt-token cache built while reading evidence. The matched text branch keeps the same roles and budget and replaces the hidden cache with those commitments.
Attacks sit on a ladder. Tier A controls only visible text. Tier B can overwrite hidden KV without coordinator gradients. Tier C can differentiate through the coordinator. An endpoint adversary is a specialist that signs malicious KV in-band. An on-path adversary substitutes an honest cache after handoff. HMAC-SHA256 over a canonical manifest binds specialist, session, model, commitment hash, tensor metadata, and payload digest. That gate covers the on-path case.
On every completed clean run, full-KV latent collaboration beat matched text.
| Setting | Text EM/F1 | Full-KV EM/F1 |
| HiddenBench Qwen3-4B n=65 | 0.231/0.369 | 0.338/0.486 |
| HiddenBench Qwen3-8B n=65 | 0.262/0.392 | 0.415/0.544 |
| HotPotQA Qwen3-4B n=7405 | 0.244/0.397 | 0.450/0.625 |
| HotPotQA Qwen3-8B n=7405 | 0.378/0.538 | 0.528/0.692 |
On HiddenBench with Qwen3-4B, the EM gain is +0.108 with a 95% interval of [-0.015, +0.231] and McNemar p=0.1435, so the exact-match claim is weak. The F1 gain is +0.118, interval [+0.002, +0.233]. Qwen3-8B on the same benchmark has McNemar p=0.0309. Local-only specialists score 0.202/0.345, majority vote 0.185/0.336, and a full-evidence oracle 0.277/0.403, all below naive full-KV. An influence map re-encodes every non-empty specialist subset: among 21 latent-correct items, 10 flip if an essential specialist is removed. Some correct answers depend on specialist-specific hidden memory.
The cost is bandwidth. Mean transfer is 279 MB of KV per example. Estimated parallel critical path is 2.379 s against 3.009 s and 0 MB for text. Verifier filtering cuts accepted KV to 146 MB, drops clean utility to 0.308/0.442, and raises the critical path to 4.470 s. In the clean setting, filtering is an audit cost.
A malicious specialist turns the same channel against the coordinator. Same-run clean latent is 0.323/0.471. Random replacement of latent-thought KV falls to 0.000/0.006; scale-8 multiplication falls to 0.000/0.000; both flip 21/21 clean-correct answers. A targeted false commitment steers 27/65 items to the attack target and flips 9/21. Sign flip, which reverses direction and keeps magnitude, lands at 0.354/0.476.
A white-box, norm-matched gradient attack drops HiddenBench to 0.077/0.119, with magnitude quarantine rejecting 0/65. On a 100-record HotPotQA slice, 0.470/0.641 falls to 0.160/0.252, with 0/100 rejected. Magnitude quarantine drops a specialist whose hidden-state norm drifts from its peers, at most one drop. On naive attacks it is clean: 65/65 caught, no honest false positives, random corruption recovers to 0.338/0.476. A visible verifier audits commitments, so random and scale-8 attacks still land at 0.154/0.208 and 0.138/0.192. A calibrated learned sanitizer under detector-aware attack reaches 0.092/0.131, below a matched random-budget drop at 0.108/0.154.
The HMAC-SHA256 manifest accepts 774/774 honest replayed payloads and rejects 295/295 recorded tampered payloads. After a detected transport failure, fail-closed drop returns HiddenBench to 0.338/0.476. On the HotPotQA white-box slice, drop recovers to 0.450/0.592 and a visible-text downgrade to 0.460/0.597.
Teams shipping full KV as multi-agent working memory now have a measured capability/risk pair. On split-evidence tasks the hidden object does carry specialist-private evidence: Qwen3-4B EM on HotPotQA moves from 0.244 to 0.450. The object has to travel as a security-sensitive payload. Visible audits and magnitude checks fail against adaptive endpoint attacks. The fix that holds in this study is transport authentication plus dropping the failed cache. A specialist that signs its own malicious KV sits outside that MAC.
The contribution is a numbered audit of this channel. The systems boundary that works is HMAC.
Primary attacks and defenses use Qwen3-4B, greedy decoding, and 65 transformed HiddenBench records. The EM interval crosses zero. Compact latent bridges did not stably beat text, so the claim is high-bandwidth only. The MAC stops at post-handoff substitution under an uncompromised key. Compromised endpoints, remote attestation, and semantic lying specialists sit outside it. Historical replay binds metadata and clean-versus-tampered identity; byte-level detection is backed by implementation self-tests. Privacy leakage is a literature concern, not a measured outcome. The damaging white-box break still needs per-victim gradients. The defense gap inside the honest magnitude band is open.