Claude Opus 5.5 in agent mode gets multiple testing corrections backwards, user finds
Sauers_ · x · 2026-10-19
Sauers reports that Claude Opus 5.5 in default agent mode doesn't know how or when to properly correct for multiple testing. Given 10 independent tests all yielding p = 0.1, the model tightens the significance threshold — but under the null you'd expect a uniform p-value distribution, so 10 near-identical p-values is itself unusual. The statistically correct move is the opposite: a meta-analytic test combining evidence, which would push p far below 0.1. A concrete example of statistical reasoning blind spots in frontier agents doing data analysis.
More from Models
- Viral DeepSeek staffer chat claims benchmark scores rose on their own; Opus translates the humblebrag — teortaxesTex · 2026-10-19
- RTX 3090 runs 27B LLM 4.6x faster via pure inference optimizations, no new hardware — rickasaurus · 2026-10-19
- DeepSeek V4.1-Flash lands #2 on Terminal-Bench 4.0 at 1/16th the price of Opus 5 — zainhas · 2026-10-19
- OpenAI Pro 200 User Questions Missing Usage Reset After Promise, Posts Billing Screenshot — alexcovo_eth · 2026-10-19
- DeepSeek's minor update jumps to 6th on OpenRouter usage in one day — teortaxesTex · 2026-10-19
- Vals Science Index debuts: GPT-6 Astra tops at 65.09% while GPT-6.1 Sol costs 1/10th of Claude — 141_1337 · 2026-10-19