SPU benchmark: frontier models fail at predicting 4,207 real wet-lab experiments
量子位 · wechat · 2026-11-14
UniPatAI has released Science Prediction Universe (SPU), a benchmark that skips quiz-style testing: models get experimental conditions and must autonomously search, compute, and predict the outcome ranking of real wet-lab experiments within 8 hours (8-core CPU, 16GiB RAM, one H20 GPU). It covers 116 tasks and 4,207 experiments across chemistry, materials science, biology, and medicine (32 subfields), with advisors from Stanford, Cornell, Harvard and others; dataset and code are open-sourced.
The first round evaluated 12 frontier models in 13 configurations, with sobering results: the best average score was Claude Opus 5.5 at 0.486, and only 34%52% of tasks per configuration beat the randomness threshold. Several counterintuitive findings emerged:
- Higher reasoning effort ≠ better scores: raising GPT-6 Astra from medium to max effort increased runtime and API cost 5x but barely moved the score—logs show answers were typically settled at a median 26% of the run.
- More tools ≠ better: max-effort runs used a median of 4 method categories vs 1, with no significant score difference; Monte Carlo sampling mostly just added error bars.
- GPU underused: fewer than one-fifth of GPU compute attempts succeeded.
- Per-run API costs ranged from $0.22 to $65.91—a 300x spread.
Failure analysis shows errors often occur at the "choosing the right basis" step: in a solvent-ranking task, Qwen3.8 ranked by ethylene solubility while Opus 5.5 considered miscibility and catalyst compatibility (0.874 vs -0.126). The team suggests testing models on SPU tasks matching your target measurement before deploying them in research. SPU joins UniPatAI's broader suite including ScienceVision, PaperBenchX, SaaS-Bench, and EvoCode-Bench.
More from Research
- EMBER gene foundation model: frozen gene representations probe well for perturbation prediction — jmuiuc · 2026-11-16
- 158k AI answers studied: page tweaks explain 2.8% of citations, brand memory acts as a ranking factor — 5le · 2026-11-16
- USTC Demonstrates Autonomous Trapped-Ion QEC, Logical Qubit Lasts 11.7 ms — jiqizhixin · 2026-11-16
- Asimov Press founder: academic journal innovation is strangled by indexers like Clarivate — NikoMcCarty · 2026-11-16
- dspy-wsp Lets a Frontier LLM Edit Small Model Weights Directly: +18.3 Pts for ~$5-8 per Run — stanfordnlp · 2026-11-16
- Song Han Team Proposes Coco, a Proactive Desktop Agent with On-Device Context Observation — stanfordnlp · 2026-11-16