Independent audit of OpenAI's Astra safety report finds key data withheld, causal claim shaky
kevrussell · x · 2026-11-06
OpenAI Alignment claimed GPT-6 Astra completes 10% of safety-research tasks vs 16% under a capabilities framing (roughly 40% fewer). Honesty.Tools audited what it could without the files.
What held
- Internal arithmetic checks out: 6pp off a 16% baseline ≈ 37.5%; silent-withholding ratios 0.25%/0.11% → 2.3×; ethical-doubt flags 7.9%/2.9% → 2.7×.
- But no task pairs, rubrics, sample sizes, or confidence intervals were released, so significance is unknowable.
Key methodological concern
- The "ethical doubt" filter conditions on something the framing causes — doubt is 2.7× more common under safety framing, and filtering shrinks the gap to 0.75pp. That shows association, not a clean causal read.
Honesty.Tools explicitly does not vouch for the 10% vs 16% claim; a full recheck awaits the released files.
More from Models
- World Labs launches Frameboy: real-time exploration of Atlas world model at up to 15 fps — theworldlabs · 2026-11-06
- FrontierPhysics benchmark: Claude Opus 5.5 tops end-to-end physics research at 27.4% — xeophon · 2026-11-06
- Why AI writing feels annoying despite models winning literary competitions — cis_female · 2026-11-06
- METR chart reveals unreleased Anthropic Model 4 scoring 178.6 on AECI — ResultBackground2450 · 2026-11-06
- Marin's open 535B-A23B MoE finishes main pretraining on 18T tokens, checkpoints soon — dlwh · 2026-11-06
- Artificial Analysis launches Intelligence Index v5 and new Progress Index, reveals eval roadmap — ArtificialAnlys · 2026-11-06