KV Cache Blending Boosts Prefill Speed 3x in Tests
maddie-lovelace · reddit · 2026-08-22
An engineer reports successful testing of KV cache blending for LLM inference. By splitting long prompts into chunks, generating caches in isolation, and concatenating them with overlap, the model maintains full needle-in-a-haystack retrieval and synthesis capabilities. The method boosted prefill speed by 3x on Ling3-tiny, hitting 1.3k tps at 256k tokens, matching Qwen 38-27B performance on a 5090 GPU.
More from Infra
- Copper to optical: optical backplanes seen as necessary as racks move to optics — BenBajarin · 2026-10-16
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-16
- Chinese labs run on ~1/10 US compute; ByteDance pays 2-4x to hoover up SEA supply — robleclerc · 2026-10-16
- Zymtrace raises $12.2M and hires ML engineers to build agents that fix GPU bottlenecks — cfregly · 2026-10-16