127,833 real clinician queries show LLM benchmarks miss actual clinical use
NYU-OLAB · hf · 2026-10-13
An NYU-led team analyzed 127,833 queries from 6,342 clinicians across 35 specialties during an eight-month rollout of an institutional LLM assistant, using their RCQ-Map taxonomy. Documentation/administration (36.2%) and knowledge retrieval (28.9%) dominated, diagnosis was just 3.7%, and over a third of queries were poorly answerable as posed. Comparing against 58 public benchmarks (the Clinical AI Benchmark Atlas), the median benchmark contained zero documentation tasks and shared only 31% of the real-use task mix — below a uniform spread. Benchmarks designed to mimic clinical practice fared no better than frontier-report benchmarks. Benchmark scores thus say little about real-world clinical performance; evaluation should match actual usage.
More from Models
- Nace AI's Drex 2.0: a 35B decision-only model with 3B active params beats Jev on Decision Index — techNmak · 2026-10-13
- Meta's Alexandr Wang confirms Muse Code quota bug, promises usage resets — alexandr_wang · 2026-10-13
- Muse Spark subscribers hit quota limits within minutes of launch — MatthewBerman · 2026-10-13
- Mollick: Muse Wonder impresses, but Meta's thinking model trails the Big Three — emollick · 2026-10-13
- Early User: Meta's New Muse Model Beats Rivals in Text and Agentic Tasks — intellectronica · 2026-10-13
- Jev 1.13 aces 153/153 real-length clinical notes, fastest top model at 345ms — MaziyarPanahi · 2026-10-13