llama.cpp fork dedups repeated prompts losslessly, cutting 108k to 71k tokens in agent loops
Odd_Cauliflower_8004 · reddit · 2026-09-26
A developer published a llama.cpp fork targeting a common agent-loop cost: the same large content resent over and over — a file re-read ten turns later, or a tool returning identical output — every copy sitting in context and getting prefilled.
- How it works: a pass in llama-server's chat parser replaces a later byte-identical message (same role, above a size threshold) with a one-line reference pointing to the earlier copy; the first occurrence always stays in full
- Off by default — with it off, the rendered prompt is byte-identical to upstream; stateless and deterministic so the prompt cache keeps hitting
- Configurable via --message-dedup, --message-dedup-min-bytes, --message-dedup-roles, or per-request messagededup; response timings report dedupn, dedupbytessaved, deduptokenssavedest
- Ships with an eval suite of 15 synthetic agentic scenarios (dedup on/off, two runs per arm): every scenario passing with dedup off also passes with it on; heavy scenarios shrink sharply, e.g. 18,092→6,820 tokens and 108,197→71,697
Limits: catches only exact repeats, not near-duplicates; end-to-end wall-clock speedup not yet benchmarked.
More from coding & agent
- AI dev discourse has shifted from frameworks to model-version semantics, says engineer — vboykis · 2026-09-26
- Devs mock coding models that ignore better-auth and hand-roll custom auth — haydendevs · 2026-09-26
- Open-source Minara AI CLI runs autonomous agent workflows via MCP from your desktop — dr_cintas · 2026-09-26
- Codex merges Memory V2 prompts: 10KB summary cap, citations and selective retrieval — morqon · 2026-09-26
- AgentOdyssey, a Text-Game Engine for Test-Time Continual Learning Agents, Accepted at NeurIPS — DanielKhashabi · 2026-09-26
- Mario design rules for vertical agents: consistent inputs and progressively unlocked capabilities — dragon_khoi · 2026-09-26