Amazon Paper: Match KV-Cache Policy to Prevent Long-Context Failures
rohanpaul_ai · x · 2026-08-31
A new Amazon paper argues that KV-cache policy isn't just an inference optimization; it should dictate the training regime. If an LLM forgets context at inference, it should be trained to forget that way too.
- The Mismatch: Sparse attention allows fixed-size KV caches for long contexts, but models are often fine-tuned with full attention, causing failures when memory is missing at inference.
- Findings: In 128k-token tests, models trained with full attention often produced long, nonsensical answers under sparse inference, while those fine-tuned with the matching cache policy behaved correctly.
- Method: The paper makes policy-matched training practical for arbitrary cache policies, including computing gradients for a 4B model on a 40GB A100.
Related event: Amazon Paper: Training Should Match KV-Cache Inference Strategies(2 posts)→
More from Infra
- TensorSharp vs llama.cpp: Qwen 3.8 Flash Next Benchmarks — fuzhongkai · 2026-09-01
- Why did increasing context size increase speed in Llama.cpp? — satnl · 2026-09-01
- mlx-signal-processing brings 10-200x faster signal ops to Apple Silicon — TheMoonMidas · 2026-09-01
- AI inference demand surges again, supply brutally outpaced by token growth — Baconbrix · 2026-09-01
- Warp founder predicts cloud-based collaborative factories for all companies within a year — charlieholtz · 2026-09-01
- JPM: 1GW of AI Infrastructure Costs $40-45B, Frontier Labs Make ~$30B per GW — zephyr_z9 · 2026-09-01