DeepSeek V4.1 Coding Benchmark Labels Look Wrong: 47 vs 46 Score Confusion
teortaxesTex · x · 2026-10-23
AI evaluation watcher teortaxesTex questions whether Artificial Analysis has mislabeled DeepSeek's coding benchmark runs.
- Charts show DeepSeek V4.1 Flash scoring 47 under the Codex framework, while "DeepSeek V4.1 Flash 1017" scores 46 under the DeepSeek Harness.
- The author finds it hard to believe V4.1-1017 and V4.1 differ by a point across harnesses and suspects both are actually the 1017 build, tagging Artificial Analysis for clarification.
- The charts also show comparable scores for Grok 4.7 Build (xhigh), Kimi CLI-Kimi K3, Muse Code, and GLM-5.3, all with 3/3 coverage.
More from Models
- Smaug-Nemotron-3.5 debuts as a top US open-source agentic coding LLM that runs on a single machine — bindureddy · 2026-10-23
- DeepSeek on RDNA2/3 + DDR4: 19 t/s peaks, 9 t/s average decode on 56GB VRAM — EmPips · 2026-10-23
- IBM CEO: best open AI models are Chinese, West 6-12 months behind, 75-90% of tokens going open-weight — victor_explore · 2026-10-23
- ArchitectureIQ Benchmarks LLMs' Training Intuition: GPT-5.6 Sol Hits 76.4% vs 33% Chance — ZimingLiu11 · 2026-10-23
- Abliterated GLM-5.3 sells for $10 as "world-class penetration tester", fueling vulnerability surge — deedydas · 2026-10-23
- DeepSeek harness slashes tokens, time and cost vs Codex, but tanks on FrontierCode — teortaxesTex · 2026-10-23