SigMap study across 20 repos: leaner agent context cut cost 7% but grew tokens 18%
Independent-Flow3408 · reddit · 2026-11-23
The SigMap maintainer ran paired held-out experiments across 20 repositories measuring retrieval setups end-to-end for coding agents. A short index with two MCP lookup tools cut API cost 7% (95% CI -13%-2%) but raised total tokens 18% with task time roughly flat. A prefetched file-ranking approach cut time 18% and tokens 26%, though the cost interval crossed zero and five repos got pricier; a Claude Code hook showed smaller effects than direct prompt injection. Experiments only test code location, not patch correctness, and no reliable offline predictor of savings was found. The author asks how others weigh success rate, dollars, and wall-clock time when benchmarking retrieval layers.
More from coding & agent
- Declarative Software Factory: making 'Done' explicit as agents do the work — hwchase17 · 2026-11-23
- Cutting MCP tools from 7 to 2: 7% cheaper API calls but 18% more tokens — Independent-Flow3408 · 2026-11-23
- huashu-design 3.5 ships: WeChat mini-program design support, two days after 3.4 — AlchainHust · 2026-11-23
- Dev Fixes 3 Agent Eval Holes: Blind Recount Hits 89% Agreement, Pooled 4% Miss Rate Hid a Bad Lane — armanidev_ · 2026-11-23
- Firecrawl vs context.dev: Developers weigh web data stacks for AI agents — lewishorsley96 · 2026-11-23
- Popfold: an open-source three.js engine built with parallel Claude Code sessions — cherichita · 2026-11-23