Frontier models jump 31 points in 21 months on SciCode, now solving 67% of research problems
geoffwolfe · x · 2026-10-29
Artificial Analysis data shows the best score on the SciCode research-coding benchmark rose from 35% in December 2024 to 67% today—a 31-point gain in 21 months.
SciCode gives models real research problems broken into the steps a scientist would take, with unit tests ideally grounded in peer-reviewed papers.
Key milestones:
- Claude 3.5 Sonnet 35.4% → o3-mini 42.8% → Gemini 2.5 Pro 46.3% → Gemini 3.1 Pro 58.7% → Claude Fable 5 61.0% → Claude Opus 5.5 66.9%
- SciCode-Verified (Aug 2026): domain experts audited every test, found 263 defects, corrected them, and re-scored 12 frontier models—84–98% of subproblems, 69–92% of full research problems
The author argues the benchmark looks saturated but the approach isn't—there are tens of thousands of other scientific computing problems. Verifiable rewards unlocked reasoning models like DeepSeek-R1, making executable science an excellent training signal.
More from Models
- Nat Lambert Mocks Teased 'Gemini 4 Carbon' as a Graspy Preemptive Move — natolambert · 2026-10-29
- Unverified 'Gemini 4 Carbon' SOTA claims circulate alongside Google Docs ranking jab — gnukeith · 2026-10-29
- DeepMind's Samuel Albanie: new model's hallucination rate much improved over Argon — SamuelAlbanie · 2026-10-29
- User pushback: US AI models are 'heavily censored' and less useful — gnukeith · 2026-10-29
- Google slammed for slow releases: rivals may outperform Gemini 4 Carbon before it ships — Angaisb_ · 2026-10-29
- Nat Lambert calls out Google: don't announce Gemini scores nobody can try — natolambert · 2026-10-29