AI Agents Fail at Open-Ended Research: Authors Reject Papers After 6-Day Test
sayashk · x · 2026-07-31
Current evaluations of AI agents often focus on narrow, verifiable tasks, whereas real scientific research is open-ended. Researchers conducted a "shadow evaluation" where AI agents were given research questions from two unpublished papers, six days, and thousands of dollars in API credits and compute.
While agents proved fluent in engineering tasks, they struggled with core research skills like formulating hypotheses, determining appropriate evidence, and recognizing failing approaches. The original authors reviewed the AI-generated papers and unambiguously rejected them.
More from coding & agent
- Harness open-sources desktop and mobile apps for orchestrating agent fleets across frameworks and machines — dee_hw · 2026-10-16
- Open-Source Google Analytics MCP Server Lets LLMs Query GA Data via Natural Language — modelcontextprotocol · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- a16z podcast: AI cut a 6-month feature to 10 days and widened engineer gaps 10x — a16z · 2026-10-16
- "Hold On, My Agents Are Freezing My Screen": The Daily Grind of the Agent Era — every · 2026-10-16
- Dev builds MCP feeding Claude 3,000 real launch videos to fix samey AI output — Designer_Landscape25 · 2026-10-16