OpenAI's MentalHealthBench: GPT-6 Astra Scores 57.3 vs GPT-4o's 32.1
rohanpaul_ai · x · 2026-09-24
OpenAI released MentalHealthBench, an open benchmark for realistic mental health conversations: GPT-6 Astra scores 57.3 versus GPT-4o's 32.1.
- Motivation: prior evals focused on emergencies and broad safety criteria, leaving everyday and ambiguous conversations largely unmeasured
- Construction: co-created with 80+ licensed psychologists and psychiatrists from 22 countries, spanning 19 languages and nearly 20 subspecialties
- Scoring: each synthetic conversation gets a custom expert rubric covering behaviors like seeking context, preserving user agency, safety, and appropriate guidance; at least 3 experts review each case, and a criterion survives only when 2 agree and a 3rd doesn't contradict
- Caveat: GPT-5.6 Sol grades answers against the human-written criteria, so score quality still partly depends on an LLM judge
The benchmark is released openly for other researchers to examine, run their own evals, and build on.
More from Models
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12
- Google's Physical-Engineering's Last Exam: all 14 frontier models score below 50% — dl_weekly · 2026-11-12
- Reddit's 2026 LLM hall of fame: Opus 5.5 still unbeaten after 9 rival releases — G0dZylla · 2026-11-12
- Commit leak reveals unreleased Gemini 4 variants: Argon, Hydrogen and Helix (unconfirmed) — BlackHC · 2026-11-11