Kimi K3.1 Pro regresses on Debate Benchmark while DeepSeek 1017 gains at lower cost
teortaxesTex · x · 2026-10-27
Two new entries on Lech Mazur's Debate Benchmark split in opposite directions: Kimi K3.1 Pro (high) scored 1603, below Kimi K3 (high)'s 1691 — a notable regression — while using 66% fewer reasoning tokens; DeepSeek V4.1 Flash 1017 (high) improved to 1593 from 1577 with 29% fewer tokens. The benchmark tests sustained multi-turn adversarial argumentation across hundreds of topics, with side-swapped reruns and three judges. teortaxesTex finds the Kimi regression surprising given Debate's sensitivity to scale. The accompanying price-vs-performance chart places DeepSeek V4.1 Flash 1017 in the ideal high-rating/low-cost region.
More from Models
- Reflection AI's Beam-1.0 tops US open-weight models, hitting 88% of Qwen3.8-2.4T with 23B active params — ryanshrout · 2026-10-27
- GPT-6 Pro reruns crime research: GTA V's 2013 launch tied to violence dips in four regions — emollick · 2026-10-27
- Dev unfreezes MathTutor project: Opus 5.5 finally makes lessons good enough — lucasmeijer · 2026-10-27
- Why temperature 0 isn't deterministic: 1,000 runs of Qwen3-235B yield 80 outputs — JensHonack · 2026-10-27
- EMNLP study: LLMs attribute biased text to groups, refusing only 0.4% of the time — ZhijingJin · 2026-10-27
- OpenAI launches GPT-6.1 Terra, a cheaper model built for fast computer use — sama · 2026-10-27