PokeBench: frontier LLMs get 1,000 turns to beat Pokemon Red's first gym
VibeCodyH · reddit · 2026-09-26
PokeBench gives every frontier LLM the same setup: the Game Boy screen, eleven buttons, a walkability grid, and 1,000 turns to beat Brock — no walkthrough, no pathfinding, no hints, one shared prompt. Scoring uses a 10-milestone ladder, so failed runs still rank by progress.
- 4 of 14 runs so far earned the badge
- Best: GPT-6 Astra at 246 turns
- Cheapest win: Gemini 3.8 Flash at $3.70
- Free local Muse Glimmer 30B burned all 1,000 turns, stopping at Route 1
Full turn-by-turn logs, costs, and VODs are public at pokebench.tv.
Related event: PokeBench tests LLMs on Pokemon Red; Opus 5.5 beats Brock for $8(2 posts)→
More from Models
- Micius-Flash: 27B Multimodal Model for Hardware and Semiconductor Agents — zhiman-ai · 2026-11-11
- Baidu says RSI auto-optimized ERNIE inference, 6.24x decode speedup on Kunxin P800 — xiaohu · 2026-11-11
- Baidu's ECHO: multi-token prediction reusing the last layer as a drafter — xiaohu · 2026-11-11
- ERNIE 5.5 Flash Preview: 280B-param multi-router MoE activating only 14B per token — xiaohu · 2026-11-11
- ERNIE 5.5 recreates the Pelican test: 117-second animation drawn frame-by-frame in Python — xiaohu · 2026-11-11
- 35,000+ Grok-built apps exposed patient data, plaintext passwords, and sales pipeline — The Decoder · 2026-11-11