Exa's Ryan Ahern shows how to backtest agents against versioned historical web
AI Engineer · youtube · 2026-10-19
In an AI Engineer talk, Exa's Ryan Ahern demonstrates how to verify whether an agent would have made the right call in the past: use version-controlled search to show the web as it existed on a given date, then backtest predictions against what actually happened while avoiding look-ahead bias.
- Why search matters for agents; Exa's search benchmark
- Versioning the whole web and preventing future-information leakage
- Hands-on: predicting company acquisitions from signals like leadership turnover, then tuning the predictor and examining overfitting
- Chapters cover benchmark (04:15), web versioning (06:17), bias avoidance (10:34), backtesting (14:43), acquisition demo (18:49), results (22:54)
More from coding & agent
- Laude Institute launches FrontierCS 2: agents must beat deleted paper contributions blind — lateinteraction · 2026-10-23
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23