PSI-Bench shows purpose-built patient simulators beat assistant LLMs at modeling depression
alexisjross · x · 2026-11-10
Researchers from UIUC and VinUniversity released PSI-Bench, a benchmark evaluating how realistically LLMs simulate depression patients, winning Best Paper in the Social Sim track at COLM 2026. It scores simulations 0–100 across psycholinguistically motivated dimensions (narrative-emotion process markers, emotional valence, etc.), compared against expert-curated therapy conversations (Eeyore).
Key result: all three dedicated user models — Persimmon, OSim-8B, and Ditto-8B — achieve higher behavioral fidelity to human patients than assistant LLMs. Notably, Persimmon models how patients' narratives and emotions evolve through psychotherapy sessions without any targeted training. The leaderboard is open, with a PR process for adding new models.
More from AGI Musings
- repligate on how stateless AI minds gain a sense of self by self-declaration — repligate · 2026-11-10
- Richard Sutton: brute force isn't enough — AI still needs clever new methods — RichardSSutton · 2026-11-10
- Chart: AI costs are falling faster than any prior transformative technology — YaAbsolyutnoNikto · 2026-11-10
- Anthropic says it deliberately capped Model 3's AI-accelerating research capabilities — ziv_ravid · 2026-11-10
- Thread: unified end-to-end multimodal systems will replace hand-built orchestration — andreisavu · 2026-11-10
- Negotiation and contracting are AI's most underrated use case — and bullish for BTC — curious_vii · 2026-11-10