LOLPlay benchmark: League of Legends agents get worse with more reasoning effort
Zhaojian Yu · hf · 2026-11-04
LOLPlay benchmarks real-time visual agents in native League of Legends: 36 champion-specific tasks spanning resource management, laning, cooperation, and objectives, with agents acting via screenshots and mouse-keyboard actions, automated initialization, and 82 executable subgoal checks.
Across 324 task evaluations of three models at three reasoning-effort settings:
- GPT-6 Astra hits 25.0% task success at low/medium effort and the best 40.3% subgoal success at medium effort;
- High effort improves neither metric for any model;
- For Astra, longer requests leave fewer interaction rounds — extra reasoning carries a real cost in real-time control. Trajectories also reveal incomplete subgoal composition and actions executing after the observed scene.
More from coding & agent
- Orgo, a YC-backed platform giving AI agents their own cloud desktops, claims 1,000+ teams — Scobleizer · 2026-11-04
- Founder: Vibe-coded a PMF-successful product, then hired pro devs to fix the mess — sujingshen · 2026-11-04
- One prompt to audit cross-region traffic: dev cut trans-Atlantic latency by moving a service — DanielLockyer · 2026-11-04
- Agent silently completed only a quarter of a 200-module build and stalled waiting for confirmation — Promptmethus · 2026-11-04
- ChatGPT Agent mode can scan LinkedIn and Indeed and build a tailored job-application spreadsheet — nikola_mr64990 · 2026-11-04
- Blogger auto-applied to 500 jobs via ChatGPT and got 12 interview calls in 24 hours — nikola_mr64990 · 2026-11-04