With retries and pooled selection, Qwen3.8 27B hits 92.04% on DeepSWE 1.1, ~18 pts above GPT-6 Astra

S_Conradi · x · 2026-09-16

A hands-on experiment shows Qwen3.8 27B can reach 92.04% on DeepSWE 1.1 — about 18 points above GPT-6 Astra's reported 74% — when retries are combined with cross-config candidate pooling.

The author ran 20 configs across harnesses, thinking levels and quantizations. Coverage varied wildly: two runs scoring identical 31.86% reward disagreed on 42/113 tasks, mainly with quantized models.
Pooling rule: a task passes if any single candidate passes all new tests; no patch mixing.
Best-of-k results: k=1 → 81.43% F2P / 37.21% reward; k=10 → 98.31% / 86.25%; k=20 → 99.18% / 92.04%.
Caveat: this is oracle-selected, cross-config coverage, not single-run accuracy. A 27B model can produce passing solutions for most tasks; the hard part is reliability and candidate selection. With strong independent acceptance and regression tests, a generate → verify → retry loop is less unrealistic.

Original post →

With retries and pooled selection, Qwen3.8 27B hits 92.04% on DeepSWE 1.1, ~18 pts above GPT-6 Astra

More from Models