AI paper reproduction failures trace mostly to the papers, not the AI agents
RexDouglass · x · 2026-10-24
RT @ellliottt: when AI agents reproduce paper code, failures are mostly not AI mistakes — they come from the papers: underspecified methods, missing details, ambiguity.
The attached Figure 7 from the error-analysis study classifies divergences across agent/model combos (SWE-Agent GLM-5, SWE-Agent GPT-5.4, OpenCode, Codex CLI, Claude Code Opus 4.6; 374–494 cases each). The dominant root causes are "Original error" (paper under- or mis-specifies the original code) and "Data missing" (data absent from the reproduction package), while genuine agent errors account for only a small share. Takeaway: AI reproduction pipelines expose gaps in academic writing standards rather than model capability limits.
More from AGI Musings
- Ross Taylor: studying LLM behavior will increasingly resemble social science — rosstaylor90 · 2026-10-24
- New Yorker cartoon: "pick a field that will still require humans when you graduate" — docmilanfar · 2026-10-24
- Most "is AI conscious?" debates boil down to two people talking past each other — granawkins · 2026-10-24
- 20VC: Naveen Rao says inference is a $10T market and robotics 100x bigger than coding agents — 20VC · 2026-10-24
- Hot take: Gen Z may lose more to AI than Gen Alpha, argues Reddit post — SoftAndWetBubbles_78 · 2026-10-24
- Every negative AI consequence has been researched for years, researcher notes — NC_Renic · 2026-10-24