Meta's KVUse stores user profiles as reusable KV caches to speed up LLM recommenders
_reachsumit · x · 2026-11-18
Meta publishes the KVUse paper on arXiv, tackling serving efficiency in memory-augmented LLM recommenders.
- Problem: user profiles stored as text must be spliced into every ranking call and re-prefilled; ranking calls are prefill-heavy and repeatedly invoked, and prefix caching can't help because profiles are preceded by different text each call.
- Method: store profiles directly as reusable KV caches. Lightweight LoRA writers trained by self-distillation produce the reusable KV while decoding the profile text, so the LLM reads them at new positions without recomputation; a custom KV scheduler prefetches the next batch's KV behind GPU compute.
- Results: higher throughput with no ranking-quality loss on four InstructRec datasets using Llama-3.1-8B-Instruct.
More from Infra
- Google donates Agent Substrate to CNCF: 500ms restores, 10x density — krishnan · 2026-11-18
- Report: Nvidia brokered Alibaba-Megaspeed deal, Alice Huang ties run deeper than known — GaryMarcus · 2026-11-18
- n8n's open-source agent Nod: secure remote access via private reverse proxy — victorialslocum · 2026-11-18
- pci_eerie: Open-Source PCIe Soft Stack for Xilinx 7-Series FPGAs as a Drop-in PCIE_2_1 Replacement — IgorCarron · 2026-11-18
- Hugging Face Nears 1 Exabyte of Monthly Data Distribution as Open Model Downloads Explode — andimarafioti · 2026-11-18
- NVIDIA, AMD, Intel, Google and Meta join SGLang Consortium as v1.0 ships — heyshrutimishra · 2026-11-18