OSWorld-Science benchmark: 12 VLM agents average just 34.3% on 146 real research-software tasks
机器之心 · wechat · 2026-10-13
A Tsinghua-led consortium with UCLA, Yale, CMU, Stanford, UTokyo and others released OSWorld-Science, a computer-use agent benchmark of 146 expert-designed tasks across 7 scientific domains and 14 research applications (Mnova NMR, SAS, QuPath pathology, ANSYS simulation), validated by actual file outputs and application state rather than self-reports.
Findings:
- 12 VLMs average just 34.3% task score; Claude Fable 5.1 leads at 73.7%, Claude Opus 5 and GPT-6 Astra 58% — but per-domain variance is huge, with only a few models exceeding 20% on chemistry tasks.
- 61% of failed runs left no scoreable artifacts; one model clicked the same wrong location 39 times until budget exhaustion.
- Self-consistency checking fails when it loops around the same erroneous representation — reliable scientific verification needs independent evidence (raw data, physical constraints).
- Highest-scoring models weren't the most expensive; more output tokens didn't mean better results (Gemini 3.1 Pro: 157K tokens for 17.2%).
Benchmark, data, and code are open; the team invites community task contributions.
More from coding & agent
- Hallmark v2 questioned: how does it check taste without a human looking? — petrusenko_max · 2026-10-23
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23