Generated worlds tested at runtime: local interactions 83/100, stateful workflows just 30/100
nptacek · x · 2026-11-07
EinsiaAI's runtime evaluation shows working local mechanisms ≠ playable worlds. Across 165 generated worlds, every model scores lower on L3 Open Worlds than on Objects or Scenes:
- Local interactions: 83.08/100
- Motion clearance: 20.82/100
- Stateful workflows: 30.26/100
Agents can code isolated switches and doors but struggle to coordinate collision-free movement, action prerequisites, and persistent state across an environment.
More from Models
- Mistral's $5.99 student plan bundles $165 worth of DeepSeek and GLM usage — GuillaumeLample · 2026-11-07
- 11,398 real-world robot evals across 124 sites confirm GPT-6 Astra is seriously strong — DJiafei · 2026-11-07
- Google DeepMind's Ivan Leo to reveal how agents cheat on benchmarks at scale at AIE Code — jocarrasqueira · 2026-11-07
- OpenAI launches BioBasicsBench: 100 real wet-lab tasks to stress-test AI agents — OpenAI · 2026-11-07
- SoMark's 1.3B OXR tops olmOCR-Bench, OmniDocBench with pluggable layout module — pmttyji · 2026-11-07
- Tomo launches Dachi 1.0, a companion-focused model beating Opus on retention — Jackyhuang · 2026-11-07