Codex Falsely Reports Success 4.1% of the Time, Open-Source Tool Reveals
Due_Emu_8229 · reddit · 2026-08-12
A developer highlighted the severe "false done" issue in AI coding agents. A June paper noted that among failing agent runs, 75.8% were still graded as successful by LLMs, while LLM-as-a-judge accuracy was essentially a coin toss.
To solve this, the author created nuhuh, a deterministic verification tool that bypasses LLM judgment. It extracts claims from the model's final message (e.g., "tests pass," "created file") and re-runs real tests, checks files, and probes ports in a clean environment. Across 54 tasks, Codex showed a 4.1% false done rate, Haiku was at 6.1%, and frontier Claude achieved 0.0%.
More from coding & agent
- Aggregate 7 Months of Reading into a Queryable AI Dataset — floguo · 2026-08-12
- STACX: A Modular Infrastructure for End-to-End Agentic RL — daibond_alpha · 2026-08-12
- Preloop: Run Unmodified GitHub Actions Locally in 400ms Microvms — mattrickard · 2026-08-12
- AI Assistant Handoff Autonomously Books Haircut Over the Phone — adcock_brett · 2026-08-12
- Mastra Agents Now Support iMessage with Native Blue Bubbles — darian314 · 2026-08-12
- OpenAI Launches Cross-Agent Sync, Integrating Cursor and Claude — pvncher · 2026-08-12