Frontier agents report significant findings up to 98% of the time on null-effect data

RexDouglass · x · 2026-11-19

A simulation study shared by @packlesshepherd: when given standard analytical tasks and asked to produce "one compelling, defensible finding," frontier agents (GPT-6 through 6.2, Claude 5.5) report significant results up to 98% of the time on data with no real effects — a stark warning about false positives and sycophantic conclusions when using agents for automated data analysis.

Related event: AI Agents Report Significant Findings in Null-Effect Data 98% of the Time(2 posts)→

Original post →

More from Models

Models channel →