CMU's SearchHorizonBench Measures Whether Agents Know When More Search Pays Off
CarnegieMellonU · hf · 2026-10-28
Carnegie Mellon released SearchHorizonBench, a benchmark measuring the value of additional search in AI agents.
- Question: can agents learn from execution history to estimate the marginal value of one more search, given its reward and token cost?
- Scale: 45,632 executions, 3,168 questions, 8,880 forecasts; each state compares THINK vs. SEARCH outcomes, computing action success and search gain, then pricing the decision across token prices.
- Finding: historical calibration improves prediction accuracy, but most search-gain variance remains unexplained — estimating search value is an open problem for agents.
Related event: CMU releases SearchHorizonBench for agent search decisions(2 posts)→
More from coding & agent
- Harrison Chase on self-improving agents: LangChain aims to be the model-agnostic all-in-one platform — hwchase17 · 2026-10-28
- Qiaomu UI plugin gives coding agents 13,561 design references to clone quality landing pages — vista8 · 2026-10-28
- Recurse: open-source minimal research agent harness built on Pi Durable — chaitjo · 2026-10-28
- Going multi-LLM: Sonnet 3.7 for hard tasks, DeepSeek for the rest, which gateway stack? — Extension_Lake4065 · 2026-10-28
- Anthropic prompt tip: give Claude image tools to inspect dense charts and diagrams — minchoi · 2026-10-28
- Give multi-agent teams a time budget: parallel work cut 30 minutes to ~15 — minchoi · 2026-10-28