13 AI models play GP in 195 simulated consults: diagnoses easy, safe prescribing is the failure mode
radeon2000 · reddit · 2026-10-22
An Australian GP registrar built Crook, a benchmark where 13 AI models sit in the doctor's chair and run full consultations via tool calls — free-text questioning, exams, tests, prescriptions, referrals — scored by code against hand-written case keys, with penalties for unnecessary tests and harmful prescribing.
Key findings (195 consults: 13 models × 5 cases × 3 plays)
- Diagnosis didn't separate models: 191/195 consults named the right diagnosis, yet scores ranged 27%–85% — history-taking and safe prescribing made the difference
- Top: GPT-6.1 Sol and GPT-6 Astra (85%), Gemini 4 Argon (82%), Qwen3.8 Max (81%), Claude Fable 5.1 (80%), Grok 4.7 (79%); best open-weight: DeepSeek V4 Pro (68%)
- The record is not the patient: a penicillin-allergic patient whose chart said "nil known" received penicillin in 16/39 consults, 14 without the model ever asking about allergies; another on sildenafil got GTN 5 times, never after asking about other medicines
- Asking pays: red flags found tracked questions asked (r = 0.88); no model neared the 50-question budget
- Cost gap: GPT-6.1 Sol hit 85% at 3¢ per consult; Claude Fable 5.1 hit 80% at $1.93
- Rankings held against an earlier run (Spearman 0.92)
A GP registrar playing briskly scored 31% as a reference point — the authors stress this is a game score, not fitness to practise. Benchmark, code and all transcripts are open; a paper is heading to arXiv.
More from Models
- Nemotron 3.5 Super VL: NVFP4 on one Blackwell GPU beats BF16 on two, ~2.2x per-GPU throughput — TheZachMueller · 2026-10-22
- LFM2.5-24B-A2B Tested and Found Lacking: Looping Outputs, Broken Code, Full Configs Posted — tracagnotto · 2026-10-22
- Decision Index 0.4: 224 models scored on private benchmarks only to curb benchmaxxing — multimodalart · 2026-10-22
- Codex is painfully slow, Claude over-refuses: one developer stuck between both — not_rian · 2026-10-22
- I ran 40 PRs through Opus, Sonnet and GPT-6: only 11 of 37 bugs were caught by all three — ParticularFan1122 · 2026-10-22
- Whatever happened to the Yi family of LLMs? Nostalgia for 01.AI's models — code_star · 2026-10-22