FutureSim team: Exa Snapshot fixes leaky backsearch, opens new agent evals

maksym_andr · x · 2026-11-17

The FutureSim team recounts 9 months of building reliable backsearch: Brave API proved leaky, so Shashwat Goel built a search tool atop CCNews to power their simulations. ExaAI's new Snapshot brought massive improvements — no leakage and much better coverage.

Reliable backsearch enables new evaluation types, like testing whether an agent could have reproduced past scientific discoveries.

A quoted thread adds that most models now improve significantly over test time — not just GPT — with even open-weight models positively calibrated; Muse Wonder shows the least improvement.

Original post →

More from Models

Models channel →