What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics
Shuyu Liu
cs.AI
2026-08-17
In a bounded 24-turn agent environment, 5x sliding compression leaves GPT-5.5 completion statistically unchanged (80% to 85%) while retrieval calls triple (21.0 to 63.9, p=.002); retrieval rises in all six model-regime comparisons, a cost completion metrics never see.
Context compression is standard equipment for long-horizon agents: the trajectory grows too long, so early content is summarized or dropped by a sliding window. The standard way to evaluate a compression method is task completion. This paper shows that metric is incomplete: when dropped state is still needed during execution, the agent reacquires it through extra tool calls, the task still completes, and the interaction cost silently inflates. Completion never sees that bill.
Not a hypothetical. The author builds a reproducible measurement protocol and demonstrates "completion unchanged, retrieval tripled" as an experimental fact on three models.
The environment is IRBench, a deterministic project-planning task: 10 tasks, capacity-one resources, and hidden execution constraints (ordering rules, resource holds, fail-until-success counts) revealed only through interaction. A fixed 24-turn horizon; completion is the fraction of tasks finished when it ends. Of the four tools, queryresource, checkdependency, and gettaskinfo count as retrieval (re-acquiring external state) and execute as execution.
Two compression operators. Sliding drops early turns outright at a budget of 1/r of the trajectory (r from 1.7x to 10x). Extractive summary gets the same token budget but replaces dropped turns with a deterministic digest of observed state facts. Same budget, the only difference is whether dropped content remains recoverable in principle: a clean control.
Oracle interventions then inject specific dropped state back as a trailing message, splitting D-state (externally queryable: task graph, prerequisites, resource occupancy) from R-state (history-dependent: revealed constraints, failure counts). Three models (DeepSeek deepseek-v4-flash, Qwen qwen3.7-plus, GPT-5.5) run in two task regimes: high-IR, where constraints surface only during execution, and low-IR, where state is re-derivable from the public task graph.
Retrieval calls increase in all six model-regime comparisons and account for almost all added interaction; five of six stay significant after Holm correction. Execution calls barely move.
| Setting | Completion | Retrieval calls |
| GPT-5.5 high-IR, Full | 80% | 21.0 |
| GPT-5.5 high-IR, 5x sliding | 85% (p=1.0, ns) | 63.9 (p=.002) |
| DeepSeek high-IR, Full | 83% | 22.2 |
| DeepSeek high-IR, 5x sliding | 72% | 55.1 |
| DeepSeek high-IR, 10x sliding | 66% (p=.016) | 63.2 |
| Same, extractive summary 5x | 83% | 19.5 |
GPT-5.5 is the cleanest case: completion statistically unchanged while retrieval triples. DeepSeek's completion only drops significantly at 10x, but its retrieval is already up at 5x; retrieval is the earlier signal. The summary operator holds completion at 83% and pushes retrieval back down to 19.5 at the same budget, which locates the cost in losing execution-relevant state, not in compression per se.
The retention interventions are the more interesting half. Random fact selection matches an offline hindsight oracle (a gap of 0.1 calls), but replacing retained D-state with semantically irrelevant fabricated content raises retrieval by 57% (p<.001) with completion statistically unchanged. Retained content is behaviorally load-bearing; fine-grained selection among real facts has low marginal value. In ALFWorld, the same sliding compression produces no retrieval surge at all (completion 10.0% vs 8.9%), because relevant state there can be re-observed directly.
For anyone operating long-horizon agents, this changes an evaluation habit: A/B tests of compression cannot read completion alone; log tool-call counts. Retrieval responds at milder compression before completion does, making it the more sensitive regression signal, one that catches a bad compression configuration before completion collapses. The practical takeaways are direct: an operator that preserves a digest of recoverable state beats bare sliding windows at equal budget, and never pad digests with irrelevant content; padding does not just waste tokens, it induces defensive re-querying. The framing of completion as a non-identifying projection of interaction cost is a useful export to evaluation methodology generally.
The author lists eleven; the heavy ones: two bounded, synthetic-adjacent environments, so specific cost magnitudes may not transfer; three model families only, with cross-model comparisons descriptive rather than paired; the 24-turn horizon is a design choice, and different horizons could shift where cost converts into failure; tool-call count is a proxy, not wall-clock or dollar cost; the retrieval/execution split is tool-level, not semantic intent. The ALFWorld result also bounds the claim: the retrieval surge is environment-dependent, not an intrinsic consequence of shortening context. One of the six comparisons does not survive correction. Single-author paper; treat edge conclusions with care.