Models earn 9x more from switching ideas than refining, but 58% never find the best basin
AlexGDimakis · x · 2026-11-17
Bespoke Labs' deep dive into AutoResearchExam uses "idea basins" to map how models solve long-horizon ML research tasks. Most runs stick to one or two ideas; extra submissions go to refining rather than exploring, yet switching to a new basin gains 9x a refining step. Exploration peaks in the first six hours then drops sharply, and 58% of runs never reach the basin containing the best solution. Clustering follows Fork and Flush, with Nimble judging same-idea submissions. Alex Dimakis released a companion video.
More from coding & agent
- JetBrains bundles free Junie Lite with every account, ships baseline model via Junie CLI — SucceededMind · 2026-11-17
- heybox: Open-Source Touch ID Lock Stops AI Agents From Reading Your HEY Email — seanmcdonaldxyz · 2026-11-17
- GitHub Is Breaking Under Agent-First Workflows — Time for Anthropic to Rebuild It? — signulll · 2026-11-17
- Opensteer: open-source local Mac agent powered by your Claude subscription — dinotimm · 2026-11-17
- Fable 5.2 spent 7 hours and 8 critic rounds building a drivable Fiat Uno game in one 603 KB HTML file — Viktorkav · 2026-11-17
- Poland's engineering teams have stopped writing code by hand entirely — PawelHuryn · 2026-11-17