llama.cpp fork dedups repeated prompts losslessly, cutting 108k to 71k tokens in agent loops
Odd_Cauliflower_8004 · reddit · 2026-09-26
A developer published a llama.cpp fork targeting a common agent-loop cost: the same large content resent over and over — a file re-read ten turns later, or a tool returning identical output — every copy sitting in context and getting prefilled.
- How it works: a pass in llama-server's chat parser replaces a later byte-identical message (same role, above a size threshold) with a one-line reference pointing to the earlier copy; the first occurrence always stays in full
- Off by default — with it off, the rendered prompt is byte-identical to upstream; stateless and deterministic so the prompt cache keeps hitting
- Configurable via --message-dedup, --message-dedup-min-bytes, --message-dedup-roles, or per-request messagededup; response timings report dedupn, dedupbytessaved, deduptokenssavedest
- Ships with an eval suite of 15 synthetic agentic scenarios (dedup on/off, two runs per arm): every scenario passing with dedup off also passes with it on; heavy scenarios shrink sharply, e.g. 18,092→6,820 tokens and 108,197→71,697
Limits: catches only exact repeats, not near-duplicates; end-to-end wall-clock speedup not yet benchmarked.
More from coding & agent
- Sydney Codex Community Buildathon set for Nov 14 with Cerebras-backed ultrafast compute track — paw_lean · 2026-11-11
- shadcn's improve hits 10k stars: let top models plan, cheap models execute code fixes — shadcn · 2026-11-11
- OpenAI open-sources Codex Security CLI and SDK for AI-driven vulnerability scanning and fixing — openai · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11
- 'Adam can now do CAE': AI agent showcase in engineering simulation draws buzz — burhop · 2026-11-11