SciCode benchmark audit finds 263 defects; 91% of problems wrongly reject correct solutions
teortaxesTex · x · 2026-09-04
teortaxesTex cites a domain-expert audit of the SciCode benchmark: all 65 test problems contain 263 defects, 192 of which (spread across 91% of main problems) wrongly reject correct, instruction-following solutions. He adds that on Verified, V4-Flash-0731 outscores Fable 5, underscoring how unreliable current coding/science benchmarks — and their leaderboards — may be.
More from Models
- Mistral's Voxtral 3 speech models land #3 on STT leaderboard at $13/1M chars — ArtificialAnlys · 2026-10-21
- Perplexity ships Decider V1.2, tops Vals AI's new ClaimBench decision-model benchmark — bo_wangbo · 2026-10-21
- Omnii-1 genome model debuts: first AI-designed phages ever given to a human patient — JosephJacks_ · 2026-10-21
- Nous Portal Pricing Revealed: Unused Credits Roll Over, Up to $100 Cap on $200 Tier — alexcovo_eth · 2026-10-21
- 1.3B open-source doc parser OXR tops three benchmarks, parsing 100 pages in 101 seconds — s_mohinii · 2026-10-21
- How LLMs store facts: three learning phases, attention circuits, and hallucinations — scott_linderman · 2026-10-21