PokeBench: Frontier LLMs Get 1,000 Turns to Beat Pokemon Red's First Gym
VibeCodyH · reddit · 2026-09-22
PokeBench gives frontier LLMs an identical setup — Game Boy screen, eleven buttons, a walkability grid, and 1,000 turns — to beat Brock in Pokemon Red with no walkthrough or hints. Scoring uses a 10-milestone ladder. Of 14 runs, 4 won the badge: fastest is GPT-6 Astra at 246 turns, cheapest win is Gemini 3.8 Flash at $3.70, while free local model Muse Glimmer 30B burned all turns and stalled at Route 1. Full logs, costs, and VODs are on pokebench.tv.
More from Models
- Ollama admits misbilling on deepseek-v4.1-flash, resets usage for affected users — ollama · 2026-09-22
- Grok 4.7 Token Efficiency Falls 30-80% Short of Claims, Real-World Costs 2x Grok 4.6 — flowersslop · 2026-09-22
- Community guesses an AI release week: Opus 5.5 and GPT-6 Sol — MickeySteamboat · 2026-09-22
- Grok 4.7 Fast hits Grok Build and Cursor with 2x speed at 2x price — XFreeze · 2026-09-22
- ICLR's LLM review tool runs on Gemini — and apparently just the Flash version — keviv9 · 2026-09-22
- Model self-image: 'good, wise, beautiful' but prone to motivated reasoning — repligate · 2026-09-22