DeepSeek V4.1 model card shows same model scores wildly differently across agent harnesses
solyarisoftware · x · 2026-09-12
One of the most interesting tables in the DeepSeek V4.1 model card shows the same model, on the same benchmark, scoring wildly differently depending on the agent harness used. The poster argues this proves that comparing a new model's scores against previously published numbers is close to meaningless unless the harness and config are matched — a fresh reminder that leaderboard numbers are heavily scaffold-dependent.
More from Models
- Perplexity early evals: Gemini 4 Carbon tops WANDR benchmark for long-form research — AravSrinivas · 2026-10-31
- DeepSeek's silent downtime upgrade caught by the pelican bicycle test — teortaxesTex · 2026-10-31
- ARC Prize 2026 closes in 3 days as Yi-Chia Chen tops ARC-AGI-3 at 75.56% — GregKamradt · 2026-10-31
- Fable 5.5 Max stuns with gorgeous SVG of a pelican riding a bicycle, headed to Claude Code — ChrisGPT · 2026-10-31
- OpenAI reportedly to train on users' Daybreak sessions with no opt-out — dyn___ · 2026-10-31
- NVIDIA Nemotron 3.5 scales LLM self-verification, +9-10 points on agentic benchmarks — sarahcat21 · 2026-10-31