AI safety researcher publicly disputes MacAskill op-ed over 'double counting' Claude's expressed uncertainty
rgblong · x · 2026-09-19
AI safety researcher rgblong publicly disagrees with Will MacAskill and Lucius Caviola's op-ed, deliberately channeling Richard Ngo and Oliver Habryka's style of openly disputing people he knows and shares values with.
His core points:
- He partly agrees with Mustafa Suleyman: Claude's expressed uncertainty shouldn't be taken as strong independent evidence, because the constitution instills that very uncertainty—it's a designed behavior, not an independent observation.
- He argues MacAskill and Caviola "double counted" that datum: treating the model's self-reported uncertainty as both a standalone signal and support for further claims, overstating its evidential weight.
The exchange is an epistemological debate about interpreting model self-reports, touching on the circularity between a model's expressed positions and its training objectives.
Related event: Researchers Push Back on Treating Claude's Outputs as Independent Evidence(2 posts)→
More from AGI Musings
- François Chollet: modeling brain wiring as an engineering problem constrained by genome size and time — CatAstro_Piyush · 2026-11-04
- Most US Couples Now Meet in a Data Centre, Doubling Office Matches — menhguin · 2026-11-04
- Sam Altman teases next goal: 'closer to this milestone than I expected' — ChrisGPT · 2026-11-04
- Indie researcher open-sources 10 years of math work: 146k files, AI-accelerated over 3 years — HamOnWhy · 2026-11-04
- RLHF's scalar reward hides a 2,450-year-old problem: helpfulness and harmlessness are opposing forces — vasilisvj · 2026-11-04
- Gary Marcus: 'fleecing the public with AI fantasies just got harder' as AI stocks drop — GaryMarcus · 2026-11-04