IdeaTaste benchmark finds LLM research taste reaches only 18% of human level
XiongChenyan · x · 2026-11-04
The IdeaTaste benchmark (arXiv:2611.03424) scores LLMs on whether their research proposals—written from a 2025-frozen literature snapshot—match what the same communities published in 2026. Across 6 models, 4 fields, and 90,000+ proposals, LLMs reach 26% of human level on citation matching and just 18% on idea matching; richer context or larger populations don't close the gap. Blinded expert review shows AI proposals trail least on presentation and most on expected success, pointing to weaker directions rather than overlooked opportunities: the bottleneck is judging which problems are worth pursuing.
Related event: CMU's IdeaTaste Benchmark Shows LLM Research Taste Lags Humans(2 posts)→
More from AGI Musings
- Sharing agent sessions across a team breaks knowledge silos — and AI can summarize your teammates' workflows — steipete · 2026-11-04
- Apple's annual filing warns rogue AI agents could 'conceal' and 'manipulate,' threatening its business — jeremyakahn · 2026-11-04
- Looking back at vibe-coded projects from 6-8 months ago, most are already obsolete — DeryaTR_ · 2026-11-04
- 'AI 2027' Resurfaces: Automation of Top Research Teams by Early 2027? — cephaloform · 2026-11-04
- Deutsch critic: his strongest adherents kept missing AI progress for 4 years — NathanpmYoung · 2026-11-04
- A proposal for "Partially-Selfish Nonprofits" that let founders keep 1% of donations — AaronBergman18 · 2026-11-04