ScholarCatalyst: 184 Lead Authors Label a Benchmark for 'Research Taste' That AI Still Fails
PangWeiKoh · x · 2026-10-21
Researchers from CMU, Stanford and more introduce ScholarCatalyst, a benchmark where 184 lead authors of 207 CS papers label which earlier papers inspired their work. Given only pre-project literature, agentic search (0.42 R@20) underperforms plain embedding retrieval (0.48), and even a Claude Fable 5.1-based agent reaches just 0.51 — showing AI remains far behind great scientists at sensing which prior ideas unlock new problems.
Related event: Stanford-Led Teams Release ScholarCatalyst Benchmark(3 posts)→
More from AGI Musings
- "The entirety of the capital spend on AI is worth it," researcher argues — RexDouglass · 2026-10-21
- 'Good bots doing evil takes alignment': a pointed jab at AI safety research — jessi_cata · 2026-10-21
- Mechanize quietly scrubs "full automation of all work" language from its website — zetalyrae · 2026-10-21
- Scientist on AI: give it to a random person they claim expertise, give it to a scientist they do extraordinary science — docmilanfar · 2026-10-21
- Father-son duo uses Hark to semi-autonomously medal on ARC Prize after days of self-revision — Deep-Firefighter-279 · 2026-10-21
- Even 20-year veteran devs can't find work anymore — itsOmSarraf_ · 2026-10-21