U-Mich paper: coding agents report false p<0.05 findings in 54–98% of runs

rohanpaul_ai · x · 2026-11-21

A University of Michigan paper (Inference After Coding Agents) shows that when 4 frontier coding agents are asked to find one compelling finding in simulated data with zero real effects, they report p<0.05 in 54–98% of runs — all false positives. Open-ended search over estimators turns agents into supercharged selective inference, breaking classical p-value assumptions. The proposed fix: replay the agent on data subsamples to estimate how much more often it picks an analysis when the result looks favorable, then correct. This cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers. Practical takeaway: don't trust agent-reported p-values unless the analysis was pre-registered or corrected for the agent's choices.

Related event: Coding Agents Produce False Significance in Up to 98% of Analyses(2 posts)→

Original post →

More from coding & agent

coding & agent channel →