NeurIPS 2026 surgical benchmark: top multimodal models score under 5% zero-shot
maier_ak · x · 2026-11-17
A NeurIPS 2026 benchmark from LMU, TUM, Oxford, ZEISS and others tests 17 multimodal models on 11 surgical datasets. Zero-shot, they score under 5% on instrument-action-tissue triplets, with phase recognition below the majority-class baseline.
- Equipping models as agents with a segmentation tool and a curated library of textbooks, guidelines, and tutorial videos improves instrument/action recognition, but safety checklists and skill ratings do not improve—extra evidence sometimes misleads.
- Caveats: no frontier models as agents, a small hard set, offline frames rather than live video.
- Takeaway: tools help with seeing, not yet with judging.
Related event: Surgical AI Benchmark: 17 Multimodal Models Score Under 5% Zero-Shot(3 posts)→
More from Embodied
- CoRL fireside chat: Fei-Fei Li and AMD CEO Lisa Su on stage together — PeterStone_TX · 2026-11-17
- Distilling a frontier model into 4B params lifts robot arm success from 0 to 22.5% — ZeYanjie · 2026-11-17
- Musk retweets: Tesla Giga Berlin to build Cybercab — elonmusk · 2026-11-17
- Behavior Graphs turns raw robot policy rollouts into interpretable graphs — drmapavone · 2026-11-17
- Neuralink hackathon adds hands-free speech: iPhone calls, lock-screen controls, and a 'Hey Neuralink' Grok agent — XFreeze · 2026-11-17
- Six frontier models drive real robot arms into hazards; LEASH guardrail cuts harm to zero — 新智元 · 2026-11-17