MineExplorer shows top multimodal models collapse on long-horizon open-world tasks

美团技术团队 · wechat · 2026-07-23

MineExplorer benchmarks long-horizon exploration in Minecraft-like open worlds

Meituan's LongCat team introduced MineExplorer, a benchmark for testing multimodal models in a dynamic open world rather than static screenshots.

813 manually validated tasks, spanning 1-hop to 4-hop long-horizon problems.
A rule-based milestone evaluator for continuous 3-minute interactions in a Minecraft sandbox.
A multi-agent data synthesis pipeline with open code to generate training tasks.
The benchmark is designed to measure general exploration and planning, not Minecraft-specific wiki knowledge.

They evaluated 18 top models across Claude, GPT, Gemini and others. The best result, Claude-Opus-4.6, reached only 41% overall task success rate.

Key findings:

Performance drops sharply as hidden prerequisites increase: the best model falls from 77% on 1-hop tasks to 12% on 4-hop tasks.
Models are stronger at perception than reasoning; the bottleneck is turning what they see into a multi-step plan.
Navigation is the main failure mode, accounting for nearly 60% of failures in the best model.
Adding more steps or more memory does not reliably fix the problem; outdated observations can even hurt performance.

The team argues MineExplorer exposes the gap between seeing a world and exploring it, and it is fully open sourced for benchmarking and training.

More from Multimodal