AWS's distributed KV caching agent skill cuts self-hosted LLM cost 40%, TTFT 80%

AWS ML Blog · rss · 2026-11-21

AWS released a distributed KV caching skill in its Agent Toolkit, built on Amazon ElastiCache, claiming up to 40% lower inference cost and 80% lower time-to-first-token for self-hosted LLMs.

The problem: self-hosted fleets keep isolated per-server caches, so returning multi-turn sessions force full prefill recompute of long histories.

How it works: the skill guides coding agents (Kiro, Claude Code, OpenAI Codex) through four flows — diagnosing cold-prefill latency, estimating savings with the free KVTrace tool from your real traffic (only recommending caches that pay for themselves), wiring LMCache into existing vLLM/SGLang fleets with unified KV hashing and data types, and pre-production validation with human approval for high-impact operations.

Requirements: EFA-capable GPU instances (e.g. p5), LMCache 0.5.8rc1+, vLLM 0.11.1+ or SGLang 0.5.13+, ElastiCache for Valkey 9.1.

Original post →

More from coding & agent

coding & agent channel →