Repeated sampling lifts DeepSeek-Coder on SWE-bench Lite from 15.9% to 56%
le_james94 · x · 2026-09-16
- DeepSeek-Coder-V2-Instruct solves 15.9% of SWE-bench Lite issues with 1 sample, but 56% with 250 samples.
- The trick works because a test suite tells you which candidate was right — when the verifier is free, repeated sampling hugely amplifies generation.
Related event: Three Weeks with Stanford CS329A: Generators Have Outrun Verifiers(9 posts)→
More from Research
- GDPx: SemiAnalysis's frontier benchmark across the 11 most valuable domains — rohanpaul_ai · 2026-11-02
- Detecting covert instruction-following via internal representations parallels J-space work — dioscuri · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02
- New theory paper proves linear decay is the unique minimax-optimal stepsize schedule for SGD — konstmish · 2026-11-02