Most open models give up halfway through complex agentic coding tasks
zyxciss · reddit · 2026-11-05
A Reddit user systematically tested a dozen models while refactoring their Betelgeuse engine (a stubborn PP bug in a custom kernel) and found most models don't fail — they quit: given four subtasks, they finish one and wrap up. Models that persisted and fixed the bug: Opus 5.5 Max, GPT-6 Astra, Sonnet 5.5 Max, Kimi K3.1 Pro, Muse Wonder Max, and Haiku 5.5 (identified but didn't finish). Quitters included Gemini 3.8 Flash, GLM 5.3, DeepSeek V4.1 Flash, the Qwen 3.8 family, MiMo V2.6 Pro, and MiniMax M3.1 Flash (worst); Qwen 3.8 Flash-Next burned 178K thinking tokens and Qwen 3.8 27B exhausted its 262K context. GPT-6 Astra later boosted prefill from 7 to 70 PP/s (over 10x) and decode from 10 to 14 TG/s on just an RTX 3060 + 16GB RAM + NVMe. Takeaway: frontier closed models show far better long-horizon persistence; DeepSeek V4.1 Flash and GLM 5.3 were the closest open models but still missed things.
More from Models
- Mistral launches Le Chonk model and Studio managed agents, workflows and apps — MistralAI · 2026-11-05
- StepFun's Liu Licheng teases new work, community hopes Step 5 skips the Preview label — teortaxesTex · 2026-11-05
- HF dev questions Thinking Machines' SWE-2 Flash weight-sharing program over simple HF gating — QGallouedec · 2026-11-05
- RAG 101: One Untuned Setting Drops Answer Accuracy from 100% to 33% — dima806_dima · 2026-11-05
- Dev Praises DeepSeek's Eval Harness: 'They Really Cooked' — mariofilhoml · 2026-11-05
- Lebrel launches uncensored frontier-model inference with signed, Solana-verified receipts — HeyAmit_ · 2026-11-05