80+ Psychiatrists in 22 Countries Build MentalHealthBench from 1,215 Conversations
thekaransinghal · x · 2026-09-24
How MentalHealthBench was built:
- Experts: 80+ licensed psychiatrists and psychologists across 22 countries wrote rubrics for 1,215 conversations
- Quality control: At least three experts wrote and adjudicated rubric items per example; criteria were retained only with expert consensus
- Dimensions assessed: safety, seeking context, preserving user agency, and providing actionable guidance when appropriate
- Coverage: spans both emergencies and everyday non-acute situations, closer to real conversations than prior evals
The thread also reports steady progress among frontier models, with clear differences: context-seeking varies substantially across models while empathy and support are more consistent; room for improvement remains in context-seeking and supporting user decision-making.
More from Safety
- AI Analysis of 42 Jensen Huang Podcasts Tracks Shift on China and Chip Controls — ShakeelHashim · 2026-11-12
- Small AI sites under 20 MW mostly skip the permitting fights battering big data centers — antavedissian · 2026-11-12
- Newsweek op-ed: US should fund hospitals and universities in federated AI learning — Saul_Loveman · 2026-11-12
- Ron Bodkin: automated research taste advances faster than expected, but the evidence is weak — ronbodkin · 2026-11-11
- DEF CON panel with Bruce Schneier and Google security VP posted: scaling laws and open weights — JasonDClinton · 2026-11-11
- Claude Code refuses to type into IDEs due to unchangeable safety tier restrictions — burkov · 2026-11-11