TokenPilot Cuts LLM Agent Inference Costs Up to 87% With Cache-Aware Context Management

青稞AI · wechat · 2026-10-13

ZJUNLP's LightRSI project ships an engineering update around TokenPilot, its cache-efficient context management system for long agent rollouts, with the paper accepted to EMNLP 2026.

Three optimization levers

Prefix stabilization: stabilizes dynamic fields like timestamps to preserve prefix-cache hits; macro cache hit rates rose from 38.7%→79.2% and 67.2%→83.1% on two benchmarks
Tool output compaction: trims structural noise and long logs before they enter context, with archiving and on-demand recovery of originals
History management: retains task-relevant history based on future value, removing stale content when safe

Engineering落地

Adapters for six harnesses: OpenClaw, Claude Code, Codex, pi, DeepSeek Harness, OpenCode; a local lightrsivisual dashboard shows cache audits and compaction records
ContextCleaner lets users review task-grouped context and explicitly choose what to clean; in-progress tasks, open issues, and system instructions are protected

Measured results (sequential tasks in one session): PinchBench inference cost dropped from $7.24 to $2.79 (61% savings) while score rose 79.2→81.3; Claw-Eval cost fell from $81.52 to $10.58 (87%) though score dipped 63.4→60.8. Savings depend on backend cache support and task overlap. Code and paper are open-sourced.

Original post →

TokenPilot Cuts LLM Agent Inference Costs Up to 87% With Cache-Aware Context Management

More from coding & agent