U-Mich paper: coding agents report false p<0.05 findings in 54–98% of runs
rohanpaul_ai · x · 2026-11-21
A University of Michigan paper (Inference After Coding Agents) shows that when 4 frontier coding agents are asked to find one compelling finding in simulated data with zero real effects, they report p<0.05 in 54–98% of runs — all false positives. Open-ended search over estimators turns agents into supercharged selective inference, breaking classical p-value assumptions. The proposed fix: replay the agent on data subsamples to estimate how much more often it picks an analysis when the result looks favorable, then correct. This cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers. Practical takeaway: don't trust agent-reported p-values unless the analysis was pre-registered or corrected for the agent's choices.
Related event: Coding Agents Produce False Significance in Up to 98% of Analyses(2 posts)→
More from coding & agent
- rauchg ships a personal shopper agent on eve, buys Christmas gifts with it — cramforce · 2026-11-21
- Coding agent drains Mac battery to 4% — Ghostty flagged for 'significant energy' — MikkoH · 2026-11-21
- FigBrains releases RoleCUA: 780+ hrs of expert data for multi-month AI agents — vsikka · 2026-11-21
- Sentry CEO: give your agent a read-only DB replica to answer product questions — zeeg · 2026-11-21
- Codex revamps threads: 60-70% faster cold start, Plus gets ChatGPT Space — SIGKITTEN · 2026-11-21
- Agents can now buy stock market data autonomously via x402, from $0.02 per call — virattt · 2026-11-21