SigMap study across 20 repos: leaner agent context cut cost 7% but grew tokens 18%

Independent-Flow3408 · reddit · 2026-11-23

The SigMap maintainer ran paired held-out experiments across 20 repositories measuring retrieval setups end-to-end for coding agents. A short index with two MCP lookup tools cut API cost 7% (95% CI -13%-2%) but raised total tokens 18% with task time roughly flat. A prefetched file-ranking approach cut time 18% and tokens 26%, though the cost interval crossed zero and five repos got pricier; a Claude Code hook showed smaller effects than direct prompt injection. Experiments only test code location, not patch correctness, and no reliable offline predictor of savings was found. The author asks how others weigh success rate, dollars, and wall-clock time when benchmarking retrieval layers.

Original post →

More from coding & agent

coding & agent channel →