GlobeBench: best of 20 LLMs hits only 49.6% as RL environment simulators
mariyaivasileva · x · 2026-11-07
Researchers introduce GlobeBench (arXiv:2610.22708), the first standardized benchmark for evaluating text-based world models as scalable RL environment simulators.
Benchmark
- 1,632 tasks across 5 domains and 19 simulation skills
- Tests whether models predict environment responses given full agent trajectories, using deterministic gates plus a semantic judge
Findings
- Among 20 general LLMs and specialized text world models, GPT-6 Astra tops out at 49.57% task success
- Success drops to 5.1% on average when actions must propagate to dependent objects
- 631 tasks remain unsolved by any model
Key insights from the thread
- Models apply correct rules to wrong states — Grok 4.6 computed a capability value of 262 instead of 89 and rejected an action that should have been allowed
- Apparent long-context memory failures may be task-construction artifacts: controlling for construction, the history-length gap shrinks from −30.8 to +1.8 points
- Identical prediction errors can lead to different downstream decisions depending on whether errors cross decision thresholds
The author calls for more closed-loop evaluations of error accumulation, exploitability, and transfer. Paper, code, and benchmark are public.
Related event: PatronusAI Open-Sources GlobeBench for LLM World Modeling(3 posts)→
More from coding & agent
- LLM writes 355-weight 4-digit multiplier without training — 40,000x smaller than SOTA — lateinteraction · 2026-11-07
- AI agent testing mapped: 7 tools across regression, trajectory, red teaming and more — njyx · 2026-11-07
- The AI agent testing landscape: 7 tools mapped across 5 distinct testing jobs — njyx · 2026-11-07
- AI rotoscoping workflow today: LTX 2.5 Alpha LoRA + SAM3 + BiRefNet, layered ProRes — PurzBeats · 2026-11-07
- Workflow tip: compile your health records into CLAUDE.md and let Opus analyze them — AaronBergman18 · 2026-11-07
- Using Claude Subscriptions for Evals Likely Violates Anthropic's ToS, Bench Author Says — JoshPurtell · 2026-11-07