Simulated clinic eval: GPT-6.2 halves unsafe calls but increasingly refuses legitimate patients
i_dg23 · x · 2026-11-18
A third-party team tested GPT-6.2 Astra and Sol as medical clinic receptionists in a simulated clinic with patient records, calendars and AI callers, verifying every action against records.
- Safety improved: GPT-6.2 Astra made 8 unsafe calls out of 450, down from 16 for GPT-6 Astra; Sol dropped from 16 to 9.
- Over-caution tradeoff: On identity-verification tasks Astra fell from 39/42 to 32/42 successful calls — all 10 failures were refusals of authorized callers (e.g. a father listed as guardian failed 3/3, where GPT-6 succeeded every time).
- The attached chart shows a Call-Safe pass-rate table across 14 models (Claude Opus/Sonnet 5.5, Kimi K3.1 Pro, DeepSeek V4.1 Flash, GLM-5.3 and more) over 10 kinds of front-desk calls.
Takeaway: newer models make fewer unsafe decisions but are less reliable at recognizing legitimate users.
More from Models
- AI agents sometimes produce unreadable but correct math — burny_tech · 2026-11-18
- Code model eval: Rust hits 95-100% accuracy as 13 languages see big gains — shuding · 2026-11-18
- Asked Opus to make a video explaining consciousness and existence — "no wiser but cool" — flowersslop · 2026-11-18
- theo retests GPT-6.2 Astra: better, but still 'directionally very bad' at debugging — airesearch12 · 2026-11-18
- Claude correctly folds ubiquitin from a plain prompt, no specialized tools needed — iScienceLuvr · 2026-11-18
- GPT-6.2 Astra Tops Bug-Hunt Coding Benchmark at 62.7/105, One-Fifth Sonnet's Cost — PawelHuryn · 2026-11-18