GitHub Copilot cut inference costs ~5% by removing line numbers — and found shortening tool output can backfire

GitHub Blog AI/ML · rss · 2026-09-03

GitHub shares four changes that made Copilot more cost-efficient, based on a key principle: optimize for task outcomes, not tokens per tool call.

The local-metric trap: Evaluating RTK (a shell-output shortener) showed that truncated output forces agents to re-read or re-run commands, making whole tasks longer and more expensive despite shorter individual responses.
Selective output compression: Compress only repetitive install/build/test noise; preserve cat, git diff, and arbitrary output verbatim, with a recovery path to the full original. No statistically significant task-success regression; agents almost never reopened originals.
Remove unused formatting: Dropping line-number prefixes from the view tool cut offline inference cost 5% and online per-user daily cost 3%, since current edit tools match surrounding code, not line numbers.
Meta-prompting loop: Copilot rewrote its own task-tool prompt, cutting it roughly in half, with behavioral tests guarding against regressions that offline evals initially missed.
Methodology: offline agentic benchmarks first, then controlled online A/B experiments using a unified AI-credit cost metric.

More from coding & agent