Science LLM benchmarks have flawed answers; fixing them significantly raises model scores
Profanion · reddit · 2026-09-16
A Reddit post highlights an arXiv paper (2609.13009) showing that many current science-focused LLM benchmarks contain errors in their reference answers. After correcting the flawed ground truths, model benchmark scores rose significantly.
This implies past leaderboard rankings on science tasks may have systematically underrated models, and that benchmark data quality is an overlooked variable when comparing models.
More from Models
- Open-source VoxLock tops Krisp voice isolation benchmark with 4.95% WER — Arbaz Siddiqui · 2026-11-02
- Anthropic's 85% inference margin sparks debate on open source's price-pressuring role — code_star · 2026-11-02
- User reports xAI appears to have quietly raised weekly usage limits, dropping his meter from 93% to 47% — XFreeze · 2026-11-02
- Meituan's LongCat-2.5 lands as 1.6T MoE agent model, with DeepSeek copycat speculation — teortaxesTex · 2026-11-02
- Asked what AI firms want you to stop owning, Claude said: your own compute — alex_verem · 2026-11-02
- Is Hermes Still Worth It? Codex's $200 Tier Now Overlaps Most of Its Value — Foreign_Lead_3582 · 2026-11-02