Prime Intellect Residents Release Fully Open Agent Evals: MazeBench, PretrainGym, OpenTTD-Bench
xeophon · x · 2026-10-30
Prime Intellect shared a set of fully open benchmarks built by its residents:
- MazeBench: maze-based tasks
- PMPP-Hard: hard parallel programming problems
- PretrainGym: pretraining-focused evaluation
- OpenTTD-Bench: built on the business sim OpenTTD, where frontier LLM agents run competing transportation companies for profit, measuring real-time, long-horizon ability
All are open for the community to use and reproduce.
More from coding & agent
- MIT paper shows inbox rules fix coordination for LLM agent societies of 16,000 — omarsar0 · 2026-10-30
- "Claude will get lung cancer with how often it runs smoke tests" — dev joke hits a nerve — alec_helbling · 2026-10-30
- Jack&Jill Matches Closed-Model Quality with Together AI's Open-Model Post-Training — togethercompute · 2026-10-30
- Dev runs Cursor cloud agents from his phone with a small repo script — malliktwts · 2026-10-30
- Free search APIs keep dying for local agents — what are you paying per 1,000 queries? — CO_Oked_COO · 2026-10-30
- Install Codex on a separate SSD via junction to stop I/O contention with Windows — Yamapama · 2026-10-30