PokeBench: frontier LLMs get 1,000 turns to beat Pokemon Red's first gym
VibeCodyH · reddit · 2026-09-26
PokeBench gives every frontier LLM the same setup: the Game Boy screen, eleven buttons, a walkability grid, and 1,000 turns to beat Brock — no walkthrough, no pathfinding, no hints, one shared prompt. Scoring uses a 10-milestone ladder, so failed runs still rank by progress.
- 4 of 14 runs so far earned the badge
- Best: GPT-6 Astra at 246 turns
- Cheapest win: Gemini 3.8 Flash at $3.70
- Free local Muse Glimmer 30B burned all 1,000 turns, stopping at Route 1
Full turn-by-turn logs, costs, and VODs are public at pokebench.tv.
More from Models
- Yoav Goldberg: Model Excels at Sokoban-Like Puzzles—Trained on Them? — yoavgo · 2026-09-26
- Claude Opus 5.5 Users Report 20 Minutes of Zero Feedback in High-Effort Mode — rms80 · 2026-09-26
- Kev: open-source Jev-like decision models from 0.8B to 27B run locally for free — alexcovo_eth · 2026-09-26
- Tev1 0.8B open-sourced: a tiny Jev-like classifier running locally on Mac at ~50ms — iamrobotbear · 2026-09-26
- Delip Rao: finetuned judge models like Jev won't deliver true judge diversity — deliprao · 2026-09-26
- Genomics researcher: Claude and Codex always fail on assay design questions — anshulkundaje · 2026-09-26