OSWorld-Science benchmark: 12 VLMs average just 34.3% on real scientific software tasks
jiqizhixin · x · 2026-10-21
A team led by Tsinghua University with Yale and other institutions presents OSWorld-Science, a computer-use agent benchmark testing whether AI agents can operate real scientific software (Jupyter, RStudio, ImageJ, etc.).
Key points:
- 146 expert-designed tasks across 7 scientific domains and 14 software configurations (medical imaging, bio, EEG, chemistry, statistics, geoscience, linguistics, physics), evaluated under a unified prompt and interaction protocol
- 12 vision-language models average only 34.3% task score (partial credit, not just success rate)
- Per the accompanying score/cost chart: Claude Fable 5.1 leads (72%), Claude Opus 5 58%, GPT-6 Astra and GPT-5.6 sol in the 50-57% range; Gemini 3.1 Pro scores 17% while burning 150K+ output tokens per task, Kimi K3 30%
- The sobering takeaway: a misconfigured option, unsaved file, or suboptimal intermediate result cascades — agents that want to do real science must plan and execute correctly inside scientific tools step by step
More from Models
- Broke devs turn to Qwen: can the free Qwen Coder replace pricey Claude and ChatGPT? — OldZookeepergame1927 · 2026-10-21
- Dev threatens org-wide OpenAI ban over API credits expiring in under a year — keithmifsud · 2026-10-21
- Reddit user ditches US AI for Chinese models: cheaper, fewer refusals, no watermarks — Revolutionalredstone · 2026-10-21
- Fireworks, Microsoft and NVIDIA host SF buildathon: beat a frontier model, win $10K — sophiamyang · 2026-10-21
- Experiments show Qwen3 can compile arbitrary patterns into layer-by-layer steps with plan training — Sauers_ · 2026-10-21
- Hugging Face quietly added model file size display, a big QoL win — vboykis · 2026-10-21