rTaste: 415 experiments show frontier research agents' rankings barely beat a random pick
omarsar0 · x · 2026-10-31
The rTaste paper (Art Kanke, U. Minnesota) probes whether research agents have taste — the judgment of which experiments are worth running before outcomes are known.
- Six frontier models, each given seven recent NLP papers, proposed minimum viable experiments (MVEs); an executor ran all 415, including the ones the models ranked last.
- Only 194 of 415 were judged sound, valuable, and new; just 73 were also beyond the seed paper and non-obvious.
- The agents' first choices matched auditors' most-valued experiments more often than a random pick, but showed no advantage over simply running the experiment written first.
- Forecasts beat a base rate but weren't measurably better than a fresh instance of the same model reading only the paper.
- Agent teams cost more with no clear improvement in proposal quality.
Conclusion: agents produce many worthwhile experiments, but evidence that their rankings improve which ones to run remains limited. Eval harness and dataset of all 415 MVEs are on GitHub.
More from coding & agent
- Dev builds Greia, an open-source Linux OS for toddlers with 45 learning activities, using Claude — orblabs · 2026-10-31
- 6 October models, same 5 trap-laden agent tasks: DeepSeek does it all for $0.02, hyped Flint passes 1/5 — latent_lurker8587 · 2026-10-31
- AI Video Rerolls Cost Dollars Each: Agentic Harness Claims 2-10x Savings on Seedance — ravisparikh · 2026-10-31
- Wes Roth builds an entire AI agent workforce with Grok Bot: chief of staff, analysts, routines — Wes Roth · 2026-10-31
- Skydive launches 3D virtual office where AI coworkers get their own computers, memory and tools — cneuralnetwork · 2026-10-31
- Dev calls GitHub Actions' 99.3% uptime claim "a complete lie": SLAs underreport outages — ctjlewis · 2026-10-31