Team withdraws ICLR paper after AI coding agent's fix silently skewed their LLM evaluation
CSProfKGD · x · 2026-09-26
A research team withdrew their ICLR submission after discovering a serious evaluation error. While using Codex/Claude Code to debug LLM evaluations on modest GPUs, they approved Claude Code's suggestion to reduce the max output-token budget to fix OOM errors — without auditing it. Two months later they found the reduced generation budget was a major confound that made their method look far better than it actually was. The takeaway: AI can accelerate research, but it can also accelerate confounding. Every AI-suggested change to experimental setup needs auditing.
Related event: A Claude Code Suggestion Nearly Broke a Paper's Evaluation(2 posts)→
More from coding & agent
- Sydney Codex Community Buildathon set for Nov 14 with Cerebras-backed ultrafast compute track — paw_lean · 2026-11-11
- shadcn's improve hits 10k stars: let top models plan, cheap models execute code fixes — shadcn · 2026-11-11
- OpenAI open-sources Codex Security CLI and SDK for AI-driven vulnerability scanning and fixing — openai · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11
- 'Adam can now do CAE': AI agent showcase in engineering simulation draws buzz — burhop · 2026-11-11