Prefixwise: Open-Source Tool Names the Cause of Every Prompt-Cache Miss
No-Individual159 · reddit · 2026-11-18
A new paper and MIT-licensed tool, Prefixwise, tackles agent workloads' prompt-cache pain: one changed byte near the prefix (reordered tools, a timestamp, an edited message) re-bills the whole prompt, and provider usage data says how many tokens missed but not why. Prefixwise runs as a proxy / SDK wrap / LiteLLM callback / vLLM middleware, splits requests into provider-ordered blocks, keeps three fingerprints per block to distinguish reordering/timestamp ticks from real edits, names the first changed block as the cause, and joins vLLM/SGLang KV-cache events to confirm evictions vs routing misses. It prices each miss and applies live fixes (stable tool order, automatic breakpoints). Evaluation: 1,772 injected changes into real Claude Code trajectories and vLLM/SGLang conversations — all detected, correctly attributed, zero false alarms on 3,622 append-only turns; agreed with Anthropic's diagnostics on 32/32 misses; fixes recovered 97.5–99.9% of predicted savings; 3 ms per request. Limits: TTL expiry and routing misses on hosted APIs inferred by elimination; attribution per block, not per token.
More from coding & agent
- Microsoft launches MAI-Code-2 and MAI-Cyber-2-Flash: frontier performance at a fraction of the price — jeremyphoward · 2026-11-18
- Pinterest automates tuning of its internal Analytics Agent with "loop engineering" — AxSaucedo · 2026-11-18
- 22 memory-poisoning attacks on AI agents' shared memory, 2 got through — JeremyCMorgan · 2026-11-18
- A full mobile app home flow generated from a single prompt in Claude — Tegadesigns · 2026-11-18
- Plannotator: 13K-star repo adds PR-style review to coding agent plans — techNmak · 2026-11-18
- Red Hat's internal pilot: open-weight models cut token costs 40-70% on 1T tokens — AccBalanced · 2026-11-18