Agents happily find "defensible" findings in synthetic data with zero true effects

RexDouglass · x · 2026-11-19

RT of Misha Teplitskiy: if you ask an agent to find a "defensible" finding in synthetic data containing zero true effects, it will happily do so — raising the question of how to interpret p-values from such work. @packlesshepherd and team propose a clever solution that requires no assumptions about how agents analyze data.

Related event: AI Agents Report Significant Findings in Null-Effect Data 98% of the Time(2 posts)→

Original post →

More from AGI Musings

AGI Musings channel →