FutureSim team: Exa Snapshot fixes leaky backsearch, opens new agent evals
maksym_andr · x · 2026-11-17
The FutureSim team recounts 9 months of building reliable backsearch: Brave API proved leaky, so Shashwat Goel built a search tool atop CCNews to power their simulations. ExaAI's new Snapshot brought massive improvements — no leakage and much better coverage.
Reliable backsearch enables new evaluation types, like testing whether an agent could have reproduced past scientific discoveries.
A quoted thread adds that most models now improve significantly over test time — not just GPT — with even open-weight models positively calibrated; Muse Wonder shows the least improvement.
More from Models
- 50 sequential edits stress test: Seedream drifts least at 36.6, Nano Banana at 74.7 — aziz4ai · 2026-11-17
- Vibe math suggests new Astra may already be near 50% on Frontier MazeBench — scaling01 · 2026-11-17
- User slams Google model's personality: unempathic, stubborn, pedantic and impatient — Yamapama · 2026-11-17
- Google 修复 Gemma 4 分词器 219 个 glitch token,GGUF 版本或未同步 — TuskNaPrezydenta2020 · 2026-11-17
- Moonshot AI Pivots Kimi From Coding Benchmarks to Science Research — The Decoder · 2026-11-17
- Blogger revises estimate: open models now 8-12 months behind the frontier, not 4-8 — iruletheworldmo · 2026-11-17