AI Agents Fail Open-Ended Research: Papers Rejected by Original Authors
sayashk · x · 2026-07-31
A research team explored whether AI agents can conduct open-ended AI research. Agents were given 6 days, thousands of dollars in API credits, and compute to tackle research questions from two unpublished papers. The original authors then reviewed the AI-generated papers.
The authors unambiguously rejected the agents' outputs in this "shadow evaluation."
While agents excelled at engineering tasks like literature reviews, debugging GPU environments, running hundreds of experiments, and formatting LaTeX, they fell significantly short of human researchers in core open-ended research abilities: formulating hypotheses, deciding appropriate evidence, and recognizing failing approaches.
More from coding & agent
- Harness open-sources desktop and mobile apps for orchestrating agent fleets across frameworks and machines — dee_hw · 2026-10-16
- Open-Source Google Analytics MCP Server Lets LLMs Query GA Data via Natural Language — modelcontextprotocol · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- a16z podcast: AI cut a 6-month feature to 10 days and widened engineer gaps 10x — a16z · 2026-10-16
- "Hold On, My Agents Are Freezing My Screen": The Daily Grind of the Agent Era — every · 2026-10-16
- Dev builds MCP feeding Claude 3,000 real launch videos to fix samey AI output — Designer_Landscape25 · 2026-10-16