SearchHorizonBench: 45,632 runs to teach agents when another search is worth it
_reachsumit · x · 2026-10-28
SearchHorizonBench evaluates when an extra web search is worth its cost for AI agents. Paired continuations and repeated executions separate expected search gain from generation noise; it spans 3,168 questions and 45,632 executions. Historical calibration cuts Brier error by 16.2% on 1,632 held-out questions, yet existing estimators explain under 2% of the 21.4-point gain variance. At reference token prices, free-selection observations yield 2.49 utility points over the best constant action. Ships with reusable histories, frozen splits, baselines and scoring tools.
More from Research
- A Bayesian take on P vs NP: likelihood ratio 10:1 favors P=NP — jessi_cata · 2026-10-28
- Karpathy's GitHub repos as a free LLM learning path: micrograd to nanochat — lmoroney · 2026-10-28
- AI Science vs. Human Science: An Essay Inspired by Thurston's Math Philosophy — hugo_larochelle · 2026-10-28
- Top econometricians publish paper on how their LLMs wrote it — lihua_lei_stat · 2026-10-28
- Frontier models hit 73.3% on SpatialBench 10 months after release, near 75% ceiling — kenbwork · 2026-10-28
- Minimal PyTorch project implements transformer autograd by hand for learning custom backward passes — ifaposto · 2026-10-28