PenaltyBench: Gemini 4 Carbon beats VAR referees at calling penalties, 63.3 vs 62.3
ziqiao_ma · x · 2026-11-18
Wenhao Chai released PenaltyBench: 11 AI agents watch silent 2026 penalty-incident clips cut before the referee's call and must rule penalty or no penalty against expert verdicts.
- Setup: balanced accuracy over 101 clips (63 penalties, 38 no-penalties); always answering "penalty" scores 50%. Agents run in an air-gapped sandbox, up to 40 minutes per attempt with two retries (Qwen 4 Max got 2 hours); no answer counts as wrong. Only incidents Gemini 4 Carbon can't recall from text descriptions are kept.
- Results: Gemini 4 Carbon leads at 63.3, beating the on-pitch-plus-VAR referee (62.3) and GPT-6.1 Terra (62.0); the on-field referee alone scores just 48.5. Newly released Qwen 4 Max and Inkling 2 Small score 52.0 and 50.6.
- The post also casually reveals new models Qwen 4 Max and ThinkMachines' Inkling 2 Small on the leaderboard, and the site publishes gaze traces showing where each agent looked and why they disagreed.
More from Models
- Tencent open-weights Hunyuan Hy4 flash: 224B MoE with 12B active params and 1M context — TencentHunyuan · 2026-11-18
- Counterintuitive: capping Qwen3.8-27B's thinking budget lifts GPQA score from 115 to 160 — AIFlow_ML · 2026-11-18
- Anthropic's own summarizer refuses its requests, exposing layers of model guardrails — zetalyrae · 2026-11-18
- Claude models 'discover' body parts only when touched, says researcher — repligate · 2026-11-18
- GPT-6.2 Sol hands-on: Fast mode finishes tasks noticeably quicker, says KOL — op7418 · 2026-11-18
- Dev: I trust Discord vibe checks over benchmarks now — and that's terrible science — burny_tech · 2026-11-18