DeepSeek V4.1 model card shows same model scores wildly differently across agent harnesses
solyarisoftware · x · 2026-09-12
One of the most interesting tables in the DeepSeek V4.1 model card shows the same model, on the same benchmark, scoring wildly differently depending on the agent harness used. The poster argues this proves that comparing a new model's scores against previously published numbers is close to meaningless unless the harness and config are matched — a fresh reminder that leaderboard numbers are heavily scaffold-dependent.
More from Models
- Qwenputer: a full RISC-V CPU inside Qwen3.5-4B weights, running C in forward passes — armand_ruiz · 2026-10-31
- OpenAI blocks weapons details for teens, adds auto-ending chats fixated on violence — rohanpaul_ai · 2026-10-31
- Granite Switch 4.1.1 ships: update models like you update software — krvarshney · 2026-10-31
- Mistral Open-Sources Le Chonk: Top Open Model on FrontierCode, Cyberdefense-Tested by CrowdStrike and Cloudflare — tensorqt · 2026-10-31
- Stanford CS336 Lecture Explains Why DPO Compares Against a Reference, Not Two Current Probabilities — stanfordnlp · 2026-10-31
- ik_llama.cpp PR #2808 adds native runtime for China Telecom's Xing4.0-29B-A4B MoE with mHC and MLA — Any_Bus9205 · 2026-10-31