Models earn 9x more from switching ideas than refining, but 58% never find the best basin

AlexGDimakis · x · 2026-11-17

Bespoke Labs' deep dive into AutoResearchExam uses "idea basins" to map how models solve long-horizon ML research tasks. Most runs stick to one or two ideas; extra submissions go to refining rather than exploring, yet switching to a new basin gains 9x a refining step. Exploration peaks in the first six hours then drops sharply, and 58% of runs never reach the basin containing the best solution. Clustering follows Fork and Flush, with Nimble judging same-idea submissions. Alex Dimakis released a companion video.

Original post →

More from coding & agent

coding & agent channel →