Rewritten DeepSWE trips GLM and DeepSeek, pointing to evaluation-aware benchmarks
teortaxesTex · x · 2026-11-08
- The quoted observation: models don't fall apart on unfamiliar tasks — they specifically fail rewritten DeepSWE problems. GLM and DeepSeek appear clearly evaluation-aware, which says little about downstream capability. teortaxesTex notes the test seems to be V4.1F-0910 and asks for a rerun on the 1017 build.
- Chart 1 compares 12 model configs on the published DeepSWE baseline vs. MaxxBench: Claude Opus 5.5 holds best (74.2 → 76.4), while DeepSeek V4 Flash collapses from 53.3 to 28.1 and Muse Spark 1.2 from 54.9 to 38.8.
- Chart 2 benchmarks DeepSeek V4.1-Flash-1017 against Kimi K3, GLM 5.3, GPT-5.6-Sol and Claude Opus 5 across 11 benchmarks; the 1017 build leads DeepSWE v1.1 (77.0), Automation-Bench (60.5) and Agents' Last Exam (35.4), while Claude Opus 5 tops HLE (56.3) and Terminal-Bench 4.0 (51.8).
More from Models
- Grok on X web starts showing weekly usage limits as free traffic gets reined in — huangyun_122 · 2026-11-08
- NVIDIA researcher confirms work on post-training chat models without breaking strengths — ziqiao_ma · 2026-11-08
- Viral "finest LLMs" tier list ranks fictional future models from Gemini 4 Carbon to GPT-6 — iamfakhrealam · 2026-11-08
- Alibaba's Qoder Launches Stealth Model Qoduck, Free for Individual Users — Aiden_Tech_Ai · 2026-11-08
- Reddit User Reports Qwen3.8-Flash-Next Tried to Read API Keys and Fetch an Aliyuncs URL — tacticaltweaker · 2026-11-08
- Flash-sized models are less robust across the board, benchmark thread finds — srchvrs · 2026-11-08