ScienceArena benchmark: LLMs score 64.5% on chemistry tasks needing structural diagrams vs 74.1% without
geoffwolfe · x · 2026-09-27
Researchers from Qiyuan Tech, Peking University, Tsinghua and HKU introduced ScienceArena, a benchmark built on 13 years of recent science olympiad contests to counter benchmark saturation and data contamination. Headline finding: chemistry turns requiring structural diagrams earn only 64.5% of available points versus 74.1% without them — models can narrate plausible reaction stories but falter when they must commit to a precise molecular structure. The authors argue evals should grade actual structural commitment, not fluent scientific prose.
More from Models
- Leaked Anthropic remarks: Opus 5.5 cost over $10B, AI market is "winner take all" — A_K_Nain · 2026-11-11
- ChatGPT users hit mass "could not load conversation" errors across devices while status page stays green — CleanSignalLab · 2026-11-11
- Anthropic engineer's Fable 5.2 one-shot 'AAA-level' game sparks AI-renaissance debate — RileyRalmuto · 2026-11-11
- Age-verified paying user calls out ChatGPT for still blocking consensual adult fiction — Scary_Fix_1295 · 2026-11-11
- User reports hitting usage limits after just ~15 messages on flash-lite — Weird-names · 2026-11-11
- Developer fights Claude's "clever" phrasing; standing instructions keep drifting — jkubicki · 2026-11-11