Open-source PokeBench makes AI models play real Pokemon Red; one reached the Elite Four
afk-sapien · reddit · 2026-10-21
A hobbyist open-sourced PokeBench, running the real Pokemon Red ROM in an emulator to compare models. Each turn the model gets screenshots, party/inventory/map state, an outcome log and a self-written notebook, plus skill shortcuts that compress menu work so tokens go to decisions. Two tracks: 47 short tasks (0–1 scored) and PokeBench Long — full playthrough, 50 milestones, 1000 points, capped at 2,000 decisions for fairness. Under the cap Astra progresses fastest; unrestricted it earned all 8 badges and beat 3 of 4 Elite Four members by decision 7,627 (run ended by expired subscription). Sonnet, Haiku and Luna looped until the no-progress guard ended them. Notably, Haiku nearly matches Sonnet on short tasks yet scored 10 on the full game — short-task skill and hours-long progress seem to be separate abilities. Caveats: single run per model via official CLIs, no error bars, unequal effort settings.
More from Fun
- REK's new humanoid robots spotted on-site, human hangs alongside to mimic them — edgarpavlovsky · 2026-10-21
- 'PHPcolypse': More UAF Bugs Found in PHP Core, Dev Suggests Rewriting with Codex — basedjensen · 2026-10-21
- OpenAI LLM flags ML paper title 'Deep Ignorance' as biological risk, refuses task — burkov · 2026-10-21
- Danish national ID password was 123456, this week an intermediate CA's key was 4 — kingslyj · 2026-10-21
- AI meme puts Matt Damon in a diner: 'Fortune favors the GPU poor' — SSj_Enforcer · 2026-10-21
- Don't mention recent AI math results to Opus 5.5 if you want it to focus — repligate · 2026-10-21