Amazon Paper: Match KV-Cache Policy to Prevent Long-Context Failures
rohanpaul_ai · x · 2026-08-31
A new Amazon paper argues that KV-cache policy isn't just an inference optimization; it should dictate the training regime. If an LLM forgets context at inference, it should be trained to forget that way too.
- The Mismatch: Sparse attention allows fixed-size KV caches for long contexts, but models are often fine-tuned with full attention, causing failures when memory is missing at inference.
- Findings: In 128k-token tests, models trained with full attention often produced long, nonsensical answers under sparse inference, while those fine-tuned with the matching cache policy behaved correctly.
- Method: The paper makes policy-matched training practical for arbitrary cache policies, including computing gradients for a 4B model on a 40GB A100.
Related event: Amazon Paper: Training Should Match KV-Cache Inference Strategies(2 posts)→
More from Infra
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16