Cognition's FrontierCode 2.0 tested Gemini 4 Argon on Devin CLI, not its first-party harness, sparking fairness complaints
pvncher · x · 2026-10-31
Cognition released FrontierCode 2.0, claiming it separates model tiers more clearly than 1.1. Rankings shifted: Muse Wonder 1.0 and Gemini 4 Argon scored highest on DeepSWE v1.1 but land mid-pack on FrontierCode 2.0. pvncher spotted via Reddit that while Astra and Opus were tested in their first-party harnesses, Gemini 4 Argon ran through Devin CLI without being tested in Antigravity, raising fairness concerns about the benchmark.
More from Models
- LLMs are fluent in others' work but unreliable when pressed on details — AlexTensor · 2026-10-31
- 576GB 'go f*ck yourself Sam Altman' mirror spreads as OpenAI docs leak — retr0jirachi · 2026-10-31
- Codex 28-day challenge teases day-28 reset: OpenAI may be saving a new model for the finale — haider1 · 2026-10-31
- GPT-6 ASTRA LAW spotted in OpenAI's API model list, launch looks imminent — imjustnewatai · 2026-10-31
- Grok now accepts Alipay and UnionPay, opening paid access for Chinese users — op7418 · 2026-10-31
- Asking DeepSeek and Hy to predict their own next models yields two very different roadmaps — lucky-adventurer- · 2026-10-31