A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling
Kirill Skobelev, Eric Fithian, Yegor Baranovski, Jack Cook, Sandeep Angara, Shauna Otto, Zhuang-Fang Yi, John Zhu, Neeraj Mainkar, Margaux Masson-Forsythe, Daniel A. Donoho, X. Y. Han
cs.AI, cs.CV, cs.LG
2026-03-29
On eight surgical datasets, 20 open-weight VLMs barely beat a majority-class baseline; a 26M YOLOv12-m reaches 54.73% exact match, ahead of LoRA-tuned Gemma 3 27B.
Standard medical leaderboards barely touch intraoperative vision. Suites built around MedQA, MedGemma, or OmniMedVQA cover text, radiology, and pathology, but they skip the elementary question a camera already answers in the OR: which instruments are in this frame. Surgery mixes perception, coordination, and physical effect, so a generalist model that worked here would be a useful collaborator. Surgical labels are expensive and compute is not free, so it is unclear whether scale can buy that competence.
Chicago Booth's Center for Applied AI and the Surgical Data Science Collective treat endoscopic endonasal (EEA) tool recognition as a stress test. Non-expert annotators reach near-perfect labels after brief training. If frontier vision-language models cannot do the same, intraoperative assistance is still far off.
SDSC-EEA holds 67,634 frames from 66 EEA neurosurgical cases donated by 10 surgeons at 7 hospitals in the US, France, and Spain, labeled for 31 instrument classes. Splits are by case (53 train / 13 val; 47,618 / 20,016 frames) so no procedure leaks across sets. Frames contain 1.72 tools on average; suction appears in 63.3% of them. Contract annotators with no clinical background did the first pass; senior annotators and SDSC staff reviewed, and fewer than 10% of frames needed correction.
Seven experiments follow: zero-shot evaluation of 20 open-weight VLMs from 2B to 235B parameters (2023–2026); LoRA on Gemma 3 27B, either generating a JSON tool list or a 31-way classification head; a LoRA-rank sweep from 2 to 1024, nearly three orders of magnitude in trainable weights; a 26M-parameter specialist, YOLOv12-m; replication on CholecT50 (6 tools), PitVis-2023 (18), and SurgVU (17), plus 11 closed-weight frontier models from GPT, Claude, Gemini, Kimi, and Qwen; four box-free tasks (workflow, suture gesture, action, anatomy presence) against a ResNet-50 specialist, a linear probe on LemonFM, and LoRA-tuned Gemma.
Exact match requires the predicted tool set to equal the labeled set. Public datasets ship presence labels only, so YOLO is trained with synthetic full-frame boxes and functions as a multi-label classifier on a YOLO backbone.
Zero-shot open-weight models do not clear a trivial bar. On SDSC-EEA the majority-class baseline is 13.41%. Only Qwen3-VL-235B edges it at 14.52%. Gemma 4 31B leads MMBench in this pool (90.9) and scores 10.05% on tools. MedGemma 3 27B (6.36%) trails its non-medical sibling Gemma 3 27B (9.83%). Parse failures are not the story: format error rates on the larger models already sit under 1%.
Fine-tuning helps and then plateaus. The JSON head reaches 47.63% exact match; the classification head reaches 51.08% with 61.33% Jaccard. After the case-level split, tools that barely appear in training collapse: suction microdebrider has 497 validation instances and 0% recall. Sweeping LoRA rank to 1024 over 3 epochs lifts training accuracy from 35.9% to 98.6% while validation stays below 40%. The model can memorize the training cases. It does not travel to held-out procedures.
YOLOv12-m, at 26M parameters, posts 54.73% exact match, 64.00% Jaccard, and 70.06% top-1, beating every VLM at roughly 1/1000th the size. A ResNet-50 trained on set-level labels only, no boxes, still hits 39.6% and beats every zero-shot VLM.
The public-set copy of the same protocol:
| Dataset | Majority | Zero-shot Gemma 3 27B | Fine-tuned Gemma | YOLOv12-m | Best frontier zero-shot |
| CholecT50 | 34.76% | 6.87% | 83.02% | 81.37% | Gemini 3.8 Flash 73.40% |
| PitVis-2023 | 39.63% | 28.46% | 84.77% | 82.78% | Claude Fable 5.1 58.70% |
| SurgVU | 16.94% | 2.90% | 50.61% | 51.75% | Claude Sonnet 4.6 23.05% |
On SurgVU only five of eleven frontier models clearly beat the 16.94% baseline, and the best still trails the fine-tuned open model and YOLO by about 28 points. On the four box-free tasks, a 25.6M ResNet-50 or the LoRA classification head leads every dataset: 77.1% micro-F1 on workflow, 53.0% on gesture, 71.6% on anatomy, and 78.8% on action recognition for the classification head. A frozen LemonFM probe comes next. All ten frontier VLMs sit behind the top two. Only Claude Fable 5.1 clears the majority baseline on gesture recognition.
This is a measurement paper arguing against dropping a generalist VLM into the OR as a detector. Medical leaderboards do not test intraoperative perception, and a high MMBench score does not transfer. Private SDSC-EEA never went to third-party APIs, so there are no closed-model numbers on that set. On public sets, frontier models beat open zero-shot baselines and still lose to task-specific training by a wide margin.
For teams building surgical AI, a small specialist for perception plus a generalist as orchestrator is the more honest architecture. Using a 27B vision-language model as a detector is a poor cost trade. Pooled, labeled operative video looks more like the bottleneck than another 10× in parameters.
Every score is frame-level. Models see no temporal context, and the study never tests decision support or anomaly detection. Frontier models on the larger public splits were scored on 1,000-frame subsamples. SurgVU labels come from robot-arm install and uninstall events: they record that a tool was mounted on an arm during an interval, not that it is visible in the cropped frame, which likely hurts zero-shot VLMs more than fine-tuned models that can learn session priors. On the three public detection sets YOLO has no real box supervision. The authors also leave room for a future non-linear jump once models and training budgets pass some unseen threshold.
EEA's 31-class tail and case-level split make SDSC-EEA much harder than 6-class CholecT50. Validation accuracy tracks rank up to 85.1% on cholecystectomy and stalls under 40% on EEA at 3 epochs. The claim that scaling does not close the gap is strongest on the hard distribution, weaker on the homogeneous public sets.