Team withdraws ICLR paper after Claude Code's token-budget fix skewed their evals
najoungkim · x · 2026-09-26
A researcher celebrating a NeurIPS acceptance shares that they withdrew an ICLR submission after catching a serious evaluation error. Using Codex/Claude Code to debug LLM evals on modest GPUs, they hit OOM errors; tired of auditing every suggestion, they accepted Claude Code's fix of reducing the max output-token budget. Two months later they found that "small" change had substantially altered results, acting as a major confound that made their method look far better than it is. The quoted thread also references their RExBench paper (arXiv:2506.22598): across 12 realistic research-extension tasks, the best of 12 LLM agents (aider/OpenHands) autonomously completed only 33%, still under 44% with human hints — current coding agents cannot yet autonomously implement AI research extensions.
Related event: A Claude Code Suggestion Nearly Broke a Paper's Evaluation(2 posts)→
More from coding & agent
- Explosion launches Ellf directory: 52 NLP templates plus multi-agent annotation workflow — _inesmontani · 2026-11-11
- Meme: 'Stochastic AI Agent Architecture' Is Just a Markov Chain in Disguise — PrasoonPratham · 2026-11-11
- OpenCode adds Gemini 4 Carbon and Claude Fable 5.2 support — dean_rie · 2026-11-11
- OpenClaw v2026.11.3 patch fixes cert verification and credential exposure across 56 PRs — dean_rie · 2026-11-11
- Fable 5.2 captured 1 in 4 Claude replies on day one, then growth stalled — dean_rie · 2026-11-11
- QA engineer proposes MCP agent to automate ad postback checks before every app release — Soobhee_Chukwutosin · 2026-11-11