UWaterloo's IGMBench tests world editing in Minecraft and Terraria; best agent solves 78.2% of tasks
UWaterloo · hf · 2026-10-07
University of Waterloo researchers formalize "world editing": intervening on an existing executable world while preserving properties that shouldn't change, with intervention depth describing how strongly an edit couples entities, dynamics, and systems.
Key points:
- Built via industry-grade game modding: IGMWorld plus IGMBench with 110 tasks and 1.1K+ executable state/behavioral criteria across Minecraft and Terraria
- Tasks span property, entity, dynamics, and system interventions, evaluated on deterministic executability, behavior, preservation, and visuals
- Frontier coding agents already show substantial capability: the best config solves 78.2% of tasks at strict task-level criteria, 94.8% at criterion level
- Reliability drops with intervention depth; most failed edits still build and load — the hard part is making the world behave as requested
- Visual consistency is a separate weakness: all configs score below 50% joint visual pass rate
Conclusion: world editing is a capability distinct from world generation and interaction, and executable games make a practical testbed.
More from coding & agent
- Coinbase's Lincoln Murr on paying AI agents via the x402 protocol — MurrLincoln · 2026-10-07
- Resend logs 3M MCP calls last month, up ~30x since April — dsp_ · 2026-10-07
- Dev vibes-clones his 2-year game with Claude: looks right, plays wrong — IanArawjo · 2026-10-07
- Y Combinator-backed Linc launches discovery layer that turns enterprise tribal knowledge into SOPs — ycombinator · 2026-10-07
- Don't delete your agent memory systems yet: focused .md files still win — gregbarbosa · 2026-10-07
- A no-RAG agent memory system with plain markdown saves ~155k tokens per run — gregbarbosa · 2026-10-07