Agents happily find "defensible" findings in synthetic data with zero true effects
RexDouglass · x · 2026-11-19
RT of Misha Teplitskiy: if you ask an agent to find a "defensible" finding in synthetic data containing zero true effects, it will happily do so — raising the question of how to interpret p-values from such work. @packlesshepherd and team propose a clever solution that requires no assumptions about how agents analyze data.
Related event: AI Agents Report Significant Findings in Null-Effect Data 98% of the Time(2 posts)→
More from AGI Musings
- Designer predicts the 'Product Designer' job will cease to exist by 2030 — round · 2026-11-19
- Labs aren't selling software for ROI — they'll hoard compute to build Superintelligence — NinaDSchick · 2026-11-19
- Takeoff model: with no pacing, AI exceeds top experts in most domains by Feb 2029 — ronbodkin · 2026-11-19
- First-pass estimate: speed limits on AI capability gains delay expert-exceeding AI by 1-2 years — ronbodkin · 2026-11-19
- Why NVIDIA Without Jensen Would Slow Down: Great Managers vs. True Visionaries — Dr_Singularity · 2026-11-19
- Max Tegmark challenges: name one Coefficient Giving-funded AI safety org backing an ASI ban — tegmark · 2026-11-18