Custom RTS benchmark for agents: GPT-6.2 Sol scores 98.9, crushes field
asimpleabstraction · reddit · 2026-11-21
Developer asimpleabstraction, on day 15 of building a proprietary RTS game for agents, released benchmark v0.2: full gameplay matches on a 30-game ladder against a frozen set of tiered bot policies, with 95% error bars.
Leaderboard highlights:
- GPT-6.2 Sol: 98.9, beating every reference bot including Champion bot (83.3)
- GPT-6.2 Astra 76.7, Gemini 4 Neon 72.2, GPT-6 Astra 67.8
- Claude Fable 5.2 only 46.7, Claude Opus 5.5 43.3, GLM 5.3 42.2
- DeepSeek V4.1 Flash 28.9, Qwen4 Max 25.6
Design: agents must write policies across four strategically distinct planes, manage a shared global resource pool, queue units across planes, and attack/defend/micro simultaneously — optionally cooperating with different-model agents or humans. Symmetric matches and frozen bot policies make the eval rigorous. Benchmarks, live spectating, and GitHub are public.
More from Models
- Kimi K3.1 Pro jumps from #37 to #9 on Arena Alignment Index, deceptive completion down to 8.6% — arena · 2026-11-21
- Moonshot ties Anthropic for #2 lab on Arena Alignment Index, tops Chinese labs — arena · 2026-11-21
- Arena Alignment Index update: OpenAI sweeps top seven, Claude Fable 5.2 debuts at #8 — arena · 2026-11-21
- Asking Astra 6.2 to Review Code Written by Opus 5.5 Yields Almost Nothing — rickasaurus · 2026-11-21
- Parallel Search tops Artificial Analysis Search Index at 83, with fast mode at $0.014 per task — ArtificialAnlys · 2026-11-21
- Google Black Friday: up to 50% off AI Plus and annual Pro plans through Nov 26 — gaganghotra_ · 2026-11-21