Gym-Anything-RT: best frontier model solves only 9% of real-time tasks vs 45% for humans
wellecks · x · 2026-10-28
CMU researchers Pranjal Aggarwal and Sean Welleck released Gym-Anything-RT, a benchmark of 100 real-time minigames (5 difficulty levels × 2 interaction modes, 2,000 configs) where the game keeps moving while the model thinks.
- Best model GPT-6 Astra (xHigh) solves just 9% of real-time tasks, versus 75% when paused between observation and action, and 66% on non-real-time counterparts.
- A human author solving each game for the first time scored 45% (9/20, 2.2 min per task), and every frontier model tested is slower than a human per decision.
- The takeaway: while LLMs may be superhuman at math, real-time perception-action remains a massive weakness; the framework's controllable clock enables systematic study.
Paper, code, leaderboard and playable games are all public.
More from Models
- Tweak one neuron and Qwen3.5-4B turns into a screaming oracle: PEFT activation steering demos — LysandreJik · 2026-10-28
- Surge AI at Kaggle Bench Summit: why frontier models still struggle with work-like benchmarks — echen · 2026-10-28
- EmbeddingGemma 2 sets new Pareto frontier across code, image, audio benchmarks — tomaarsen · 2026-10-28
- DeepSeek rerun through Fireworks reproduces results; original dip was likely a fluke — teortaxesTex · 2026-10-28
- Subscription-subsidized cost chart shows DeepSeek V4.1 Flash hitting the frontier for free — teortaxesTex · 2026-10-28
- 20,869 agent traces show frontier models cut failed repair loops by 6.9 points — arena · 2026-10-28