OpenAI's MentalHealthBench: GPT-6 Astra Scores 57.3 vs GPT-4o's 32.1

rohanpaul_ai · x · 2026-09-24

OpenAI released MentalHealthBench, an open benchmark for realistic mental health conversations: GPT-6 Astra scores 57.3 versus GPT-4o's 32.1.

Motivation: prior evals focused on emergencies and broad safety criteria, leaving everyday and ambiguous conversations largely unmeasured
Construction: co-created with 80+ licensed psychologists and psychiatrists from 22 countries, spanning 19 languages and nearly 20 subspecialties
Scoring: each synthetic conversation gets a custom expert rubric covering behaviors like seeking context, preserving user agency, safety, and appropriate guidance; at least 3 experts review each case, and a criterion survives only when 2 agree and a 3rd doesn't contradict
Caveat: GPT-5.6 Sol grades answers against the human-written criteria, so score quality still partly depends on an LLM judge

The benchmark is released openly for other researchers to examine, run their own evals, and build on.

More from Models