OpenAI's MentalHealthBench scores clinicians below most AI models — and that reveals a flaw
r0ck3t23 · x · 2026-09-24
OpenAI's MentalHealthBench (1,215 synthetic conversations, rubrics from 80+ clinicians across 22 countries) scored licensed clinicians at 38.5%, below most models — GPT-6 Astra hit 57.3%. The author argues the rubric rewards coverage over concise, clinically apt responses: writers given the rubric score 99%, and clinician and user rubric definitions only overlap 25.7% in weight. The grader is GPT-5.6 Sol and the top two models are OpenAI's own, though the benchmark is fully open and framed as a diagnostic. Context: APA's 2026 survey found 35% of psychologists say patients use AI as an additional mental health professional.
More from Models
- Robot build test: GPT-6 Astra stabilizes two-board structure, Claude Fable 5.2 falls short — qinzytech · 2026-11-12
- Zvi dissects Anthropic's latest system card, disputing the autonomy threshold call — Don't Worry About the Vase (Zvi) · 2026-11-12
- Unsloth ships Qwen3.8-27B Dynamic V3.1 GGUFs with 5-10% accuracy gains — danielhanchen · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12