DIY RTS benchmark for agents: GPT-6.2 Sol posts 98.9 win rate, tops all rivals
asimpleabstraction · reddit · 2026-11-21
A Reddit developer building a proprietary RTS game for agents released ACBench v0.2, running model agents against a frozen ladder of tiered bot policies (30-game matches, 95% error bars) in fully symmetric games that demand multi-plane strategy, shared-resource management, and even cooperation with heterogeneous agents or humans.
Leaderboard highlights: GPT-6.2 Sol leads at 98.9, beating even the built-in Champion bot reference (83.3) and Elite bot (78.3); GPT-6.2 Astra scores 76.7, Gemini 4 Neon 72.2, GPT-6 Astra 67.8; Claude Fable 5.2 (46.7), Claude Opus 5.5 (43.3) and GLM 5.3 (42.2) sit mid-pack, while DeepSeek V4.1 Flash (28.9) and Qwen4 Max (25.6) trail. The author argues symmetry makes per-version bot behavior invariant, and is soliciting suggestions for more models to test.
More from Models
- Google Black Friday: up to 50% off AI Plus and annual Pro plans through Nov 26 — gaganghotra_ · 2026-11-21
- Model so DeepSeek-like that every AI tool misidentifies it as DeepSeek — cephaloform · 2026-11-21
- Claude's 'Riemann Hypothesis proof' unravels: a 250,000-page slop book and a tautological Theorem 1 — basedjensen · 2026-11-21
- GPTZero scans millions of citations: hallucination rates flat or rising at CVPR, ICLR, EMNLP — HanchungLee · 2026-11-21
- Investor slams per-token pricing: verification burns millions of tokens, calls for pay-per-outcome — jasondeanlee · 2026-11-21
- Custom RTS benchmark for agents: GPT-6.2 Sol scores 98.9, crushes field — asimpleabstraction · 2026-11-21