SurgKnowBench: 376k-sample NeurIPS benchmark shows general AI models can't judge surgery
maier_ak · x · 2026-11-17
SurgKnowBench: Testing What General AI Knows About Surgery
A new paper from LMU Munich, TUM, Oxford, Stuttgart and ZESS (accepted at NeurIPS 2026 Datasets & Benchmarks track) introduces SurgKnowBench: 11 public surgical datasets covering 7 procedures (gallbladder removal, gastric bypass, prostate, pituitary, cataract surgery), 376,602 samples total, with 7 tasks of increasing difficulty — instrument detection/localization, who-does-what-to-which-tissue, phase and action recognition, plus two expert judgments: the critical view of safety checklist and surgical skill rating.
Stage one tested 17 general-purpose models with zero adaptation. Results were sobering: even the best models struggled on tasks requiring judgment.
Stage two mimics surgeon training: give the model a library (textbooks, guidelines, recorded operations) plus tools for close inspection, without surgical fine-tuning. Caveats noted by the author: no frontier models as agents, a small hard set, offline frames rather than live video. Bottom line: tools help with seeing, not yet with judging.
Related event: Surgical AI Benchmark: 17 Multimodal Models Score Under 5% Zero-Shot(3 posts)→
More from Research
- kalomaze: cross attention is what you add when you iterate on architecture more than data — kalomaze · 2026-11-17
- What 4 calibration papers do (and don't) say about gating agent tool calls on model confidence — Xjjjjyn · 2026-11-17
- ICLR 2027 rebuttal catch-22: new experiments deemed "too much change," rebuttals dismissed as "AI slop" — SimpleObvious4048 · 2026-11-17
- IIT Madras CeRAI unveils conversational AI eval tool with 7 dimensions, 48 core metrics — ravi_iitm · 2026-11-17
- Are top AI conferences failing? High-quality authors may be dropping out — analisereal · 2026-11-17
- Distilling a frontier model into 4B params lifts robot arm success from 0 to 22.5% — ZeYanjie · 2026-11-17