Study Finds AI Agents Excel at Engineering but Fail at Research
burny_tech · x · 2026-08-17
Study Findings
- Methodology: "Shadow evaluations" where frontier AI agents were tasked with the core research questions of two unpublished NeurIPS 2026 submissions, graded by the original authors.
- Resources: Agents were given six days and thousands of dollars in compute budget.
- Core Conclusion: Agents completed all engineering tasks without human help but made no substantial progress on the research questions, resulting in unambiguous rejection of both papers.
Identified Failure Modes
- Poor judgment on the bar for publishable research
- Uncreative responses to research design shortcomings
- Ineffective backtracking from dead ends
- Poor resource awareness
- Instruction drift
More from coding & agent
- Six-month prediction: hiring for generic agent harnesses built from scratch will cool — yi111 · 2026-10-16
- Omarchy ISO wouldn't boot on NVIDIA, so he had Claude patch it — indragie · 2026-10-16
- Dev uses Grok to clean up 3D printer firmware, catches it sneaking in nonsense code — yacineMTB · 2026-10-16
- A rude AI agent took 89% of a shared resource in 102-run multi-agent personality experiment — Brave-History-6502 · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16