RuneBench: Benchmarking AI Coding Agents by Playing RuneScape
charliermarsh · x · 2026-10-20
A developer released RuneBench, a benchmark where AI agents accomplish tasks inside the game RuneScape. Agents read and act on game state via a TypeScript SDK against an emulated server, and can search markdown docs extracted from the game wiki (skill guides, item stats, quest walkthroughs), testing multi-step planning in an "orient, decide, act" loop.
Scoring: The original metric — total XP gained in a fixed window — punished exploration, since pure grinding won. The final design measures max XP rate per 15-second window, rewarding discovery of higher-level strategies; winning runs combined many locations, tricks and methods — "models are incredible optimizers."
Limitations: long runtimes; shorter tasks punish front-loaded planning (models spend minutes reading docs) and slow inference; environment complexity plus small samples create noise and false negatives. Shorter micro-tasks remain future work.
More from coding & agent
- Free book 'Build Your Own Software Factory' on trusted agent delivery systems — prasanna_says · 2026-10-20
- Rachel Thomas: optimizing only for autonomous coding agents won't yield better collaborators — math_rachel · 2026-10-20
- Position paper: humans are missing from AI coding agent research — math_rachel · 2026-10-20
- REA gains 214K stars in a week, tops GitHub's fastest-growing AI repos — sven_ai · 2026-10-20
- Dev goes 'graph pilled': graphs will anchor every company's world model — _AustinCalvert_ · 2026-10-20
- A 5-step loop for debugging AI agents in production: trace everything, watch for silent failures — Tracify · 2026-10-20