Prefixwise: Open-Source Tool Names the Cause of Every Prompt-Cache Miss

No-Individual159 · reddit · 2026-11-18

A new paper and MIT-licensed tool, Prefixwise, tackles agent workloads' prompt-cache pain: one changed byte near the prefix (reordered tools, a timestamp, an edited message) re-bills the whole prompt, and provider usage data says how many tokens missed but not why. Prefixwise runs as a proxy / SDK wrap / LiteLLM callback / vLLM middleware, splits requests into provider-ordered blocks, keeps three fingerprints per block to distinguish reordering/timestamp ticks from real edits, names the first changed block as the cause, and joins vLLM/SGLang KV-cache events to confirm evictions vs routing misses. It prices each miss and applies live fixes (stable tool order, automatic breakpoints). Evaluation: 1,772 injected changes into real Claude Code trajectories and vLLM/SGLang conversations — all detected, correctly attributed, zero false alarms on 3,622 append-only turns; agreed with Anthropic's diagnostics on 32/32 misses; fixes recovered 97.5–99.9% of predicted savings; 3 ms per request. Limits: TTL expiry and routing misses on hosted APIs inferred by elimination; attribution per block, not per token.

Original post →

More from coding & agent

coding & agent channel →