GlobeBench: 1,632 tasks show LLMs still struggle as text-based world models
PatronusAI · hf · 2026-10-20
Patronus AI released GlobeBench, the first standardized benchmark for text-based world models: 1,632 tasks across five domains and nineteen simulation skills, testing whether models can predict an environment's response to an action given the full agent trajectory—understanding prior state changes, following environment rules, and predicting effects on related and unrelated objects. Using a hybrid deterministic-gate + semantic-judge evaluator, among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra topped out at 49.57% task success. Accuracy degrades as actions affect more of the environment: current LLMs struggle to propagate action effects across dependent objects.
More from Research
- CASTER formalizes zero-shot robot manipulation from synthetic videos as constrained optimization — mangahomanga · 2026-10-20
- What makes human video generation good for robot manipulation? A constrained-optimization analysis — mangahomanga · 2026-10-20
- New MDP paper constructs learning algorithms for the bias-to-Blackwell optimality hierarchy — _onionesque · 2026-10-20
- DeepMind's Gemini agent discovers and Lean-verifies new acceleration rules for 11 BB(6) Turing machines — omarsar0 · 2026-10-20
- Atomwell debuts: 9B foundation model shows clean scaling laws on 80M experimental measurements — chaitjo · 2026-10-20
- SFT vs RLVR: how fine-tuning paradigms shape LLM confidence calibration in high-stakes use — burkov · 2026-10-20