Agent harness was silently breaking its own KV cache, wasting 43% of a local LLM run

Stunning-Sherbet1853 · reddit · 2026-10-24

A developer describes debugging a slow local agent (Ollama + Qwen3 27B on RTX 3090, driving real Chrome to fill Excel). After ruling out prompt ordering, mid-run thinking toggles (47.3s vs 1.9s first-token latency), a logging feature, and mid-run tool-list changes, they split model time from tool-wait time: 552s of a 1,274s run (43%) went to calls where earlier history had changed.

Root cause: the harness re-picked which old tool results to shrink on every call to stay under the context budget, so the cut point slid forward one message per call, invalidating the KV cache from that point every time.

Fix: shrink once, remember it, keep history append-only, re-shrink only on real overflow in one batch. Offline replay: rewrites 15→5, re-read text -61%. Real tests (3 runs each): prefill per run 327s→196s, total run 721s→541s (-25%) with identical output quality.

Advice: log prefill time per call and check whether your prompt prefix matches the last call. Next target: step count — reading 11 GitHub repos took 56 steps, 21 just opening pages; sites with APIs don't need a browser at all.

Original post →

Agent harness was silently breaking its own KV cache, wasting 43% of a local LLM run

More from coding & agent