Cognition benchmark: Gemini 4 Carbon scores 40% near-SOTA at $4.04 per task
scaling01 · x · 2026-10-30
Cognition's FrontierCode 2.0 coding benchmark shows Google's Gemini 4 Carbon scoring 40% — near SOTA — at $4.04 per task, behind Claude Opus 5.5, GPT-6 Astra, and Claude Sonnet 5.5. Gemini 4 Argon scores 26.3% at $6.41 per task.
- FrontierCode measures "mergeability": would a maintainer actually merge the PR, judged on correctness, test quality, scope discipline, style, and codebase standards
- 20+ world-class open-source maintainers hand-crafted tasks, 40+ hours per task, graded via unit tests, rubrics, and verifiers with a QC pipeline
- Internet use is allowed like an engineer would (docs, error search), but consulting solution-bearing sources scores zero
- v2.0 expands to 132 tasks, adds security patches and performance optimizations, and raises thoroughness standards
More from coding & agent
- Claude API docs reveal beta Agents, Sessions and Environments APIs with managed sandboxes — aigclink · 2026-10-30
- Graph embeddings in one line: FastRP by default, GraphSAGE for attributes — JeremyCMorgan · 2026-10-30
- CodeRabbit CEO: $8M seed rounds no longer enough as 'software factories' need tokens; Garry Tan pushes back — garrytan · 2026-10-30
- GBrain Cloud launches a CLI: one command spins up a cloud AI workspace — garrytan · 2026-10-30
- Designing a multi-bot AI automation system to run one person's life and three businesses — Xeriphy · 2026-10-30
- "The agents of the internet of things" — a catchy frame for device-side agents — sull · 2026-10-30