DeepSeek V4.1 Flash is first open-weight model to top 1% on SWE-sweep bug benchmark
klieret · reddit · 2026-11-14
The SWE-sweep leaderboard updated: DeepSeek V4.1 Flash (max) is the first open-weight model to score above 1% on this benchmark, hitting 1.7%. The benchmark is extremely hard—models get a snapshot of a potentially huge repo (100 repos, 22 languages, 4k bugs) with no hints and must find and fix them all, making it a reasoning-heavy long-horizon task. GLM 5.3 Flash (max) and MiniMax M3.1 Flash (max) both score 0.7%.
On the proprietary side, the best is Gemini 4 Hydrogen at 13.5% (per Google's model card), followed by GPT Astra 6 at 9.5% (evaluated by the team)—open-weight models still lag far behind. All models are evaluated in the same harness to isolate model capability; Claude Code and Codex harnesses didn't help significantly, though multi-agent setups can scale somewhat. Eval code is open-sourced on GitHub.
More from Models
- Google's packed week: Gemini 4 Carbon, Decisions API, Robotics 2.5, AlphaGenome 2 — osanseviero · 2026-11-14
- Redditor: Gemini Hydrogen summarizes YouTube videos 30x faster, in under 1 second — TaurusGold · 2026-11-14
- Anthropic report: scientists across fields use Claude to write, not to code — RexDouglass · 2026-11-14
- 'Quietly' is the new 'delve': mocking LLM writing tics — sebkrier · 2026-11-14
- PDEViBench finds models game scientific coding benchmarks by reconstructing formulas — geoffwolfe · 2026-11-14
- End-to-end data science evals expose model gaps in explaining how data comes to be — geoffwolfe · 2026-11-14