AI Agents Fail Open-Ended Research: Authors Reject Papers After 6-Day Test
sayashk · x · 2026-07-31
Researchers conducted a "shadow evaluation" to test whether AI agents can handle open-ended AI research. Agents were given research questions from two unpublished papers, six days, and thousands of dollars in API credits and compute.
However, the original authors unambiguously rejected the AI-generated outputs after reviewing them. While agents were fluent in most engineering tasks, they lacked the persistence, ability to identify literature gaps, and keen awareness of the original research question required for true research. This also raises concerns: the point where agents discover new findings may also be the point where we can no longer catch their mistakes.
More from coding & agent
- Harness open-sources desktop and mobile apps for orchestrating agent fleets across frameworks and machines — dee_hw · 2026-10-16
- Open-Source Google Analytics MCP Server Lets LLMs Query GA Data via Natural Language — modelcontextprotocol · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- a16z podcast: AI cut a 6-month feature to 10 days and widened engineer gaps 10x — a16z · 2026-10-16
- "Hold On, My Agents Are Freezing My Screen": The Daily Grind of the Agent Era — every · 2026-10-16
- Dev builds MCP feeding Claude 3,000 real launch videos to fix samey AI output — Designer_Landscape25 · 2026-10-16