DeepSWE's 81.6% Coding Score Misleading — Random Baseline Hits 73.7%
yifeiwang77 · x · 2026-09-24
Yi Wang flags the widely cited 81.6% agent coding number as misleading: the model served as a verifier/classifier of answers produced by another model (e.g. Fable), not as the answer generator, so claiming "81.6% on DeepSWE" is hard to justify.
He adds that a random selection baseline already achieves 73.7% on this task, further undermining the significance of the headline score.
Related event: 8B Model's 81.6% Coding Score Questioned: It's a Verifier, Not a Solver(2 posts)→
More from Models
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12
- Gemini re-renders images as text when vision is removed, dropping only 6% WSR — zacharynado · 2026-11-12
- Bare-harness coding benchmark: Fable 5.2 best overall, Gemini 4 Carbon best value — Semantic_meaning · 2026-11-12
- Google's Physical-Engineering's Last Exam: all 14 frontier models score below 50% — dl_weekly · 2026-11-12
- Reddit's 2026 LLM hall of fame: Opus 5.5 still unbeaten after 9 rival releases — G0dZylla · 2026-11-12
- Commit leak reveals unreleased Gemini 4 variants: Argon, Hydrogen and Helix (unconfirmed) — BlackHC · 2026-11-11