CMU's IdeaTaste benchmark: LLM research taste reaches only ~26% of human level
CMU-SCS · hf · 2026-11-04
Researchers at CMU and MBZUAI introduce IdeaTaste, a benchmark that scores LLMs by how closely research proposals written from a frozen 2025 literature snapshot match what the same communities published in 2026 — spanning six LLMs, four research areas, and 90,000+ proposals.
Key findings:
- LLM agents fall far short of human communities, reaching 26% of human level on citation matching and only 18% on idea matching
- AI populations can cite and take up tasks/methods as broadly as a community, but rarely arrive at the concrete approaches and research plans humans pursue
- Neither richer field context nor larger populations substantially close the gap
- In blinded expert review, AI proposals trail 2026 papers least in presentation and most in expected success
More from Research
- Terence Tao predicts an explosion of experimental mathematics as AI maps math difficulty — haider1 · 2026-11-04
- Google DeepMind Genomics team hiring 2027 Student Researcher with RL and post-training skills — alexechu_ · 2026-11-04
- SAIL AI Challenge opens final phase: turn rare disease diagnoses into treatment reports — zakkohane · 2026-11-04
- qwenFS and qwenGPU: a filesystem and SIMT GPU compiled into Qwen3.5-2B's weights — CatAstro_Piyush · 2026-11-04
- D&A turns any VLM into a zero-shot robot reward model with SOTA on three benchmarks — KyleMorgenstein · 2026-11-04
- Berkeley's Beguš uses AI interpretability to find vowel analogs in whale communication — begusgasper · 2026-11-04