SurgKnowBench: 376k-sample NeurIPS benchmark shows general AI models can't judge surgery

maier_ak · x · 2026-11-17

SurgKnowBench: Testing What General AI Knows About Surgery

A new paper from LMU Munich, TUM, Oxford, Stuttgart and ZESS (accepted at NeurIPS 2026 Datasets & Benchmarks track) introduces SurgKnowBench: 11 public surgical datasets covering 7 procedures (gallbladder removal, gastric bypass, prostate, pituitary, cataract surgery), 376,602 samples total, with 7 tasks of increasing difficulty — instrument detection/localization, who-does-what-to-which-tissue, phase and action recognition, plus two expert judgments: the critical view of safety checklist and surgical skill rating.

Stage one tested 17 general-purpose models with zero adaptation. Results were sobering: even the best models struggled on tasks requiring judgment.

Stage two mimics surgeon training: give the model a library (textbooks, guidelines, recorded operations) plus tools for close inspection, without surgical fine-tuning. Caveats noted by the author: no frontier models as agents, a small hard set, offline frames rather than live video. Bottom line: tools help with seeing, not yet with judging.

Related event: Surgical AI Benchmark: 17 Multimodal Models Score Under 5% Zero-Shot(3 posts)→

Original post →

More from Research

Research channel →