Karpathy Tests Opus 3D World Generation: Spatial Skills Impress, But Lacks Visual Feedback

AI寒武纪 · wechat · 2026-08-02

Andrej Karpathy recently evaluated the spatial understanding and generation capabilities of LLMs. He provided a model with roughly 1 million tokens of context (costing about $10) to render the opening of The Lord of the Rings into a 3D scene using Three.js. The model took nearly two hours to write 5,500 lines of code, successfully placing assets in 3D coordinates and generating animations.

Karpathy noted that while the result was a bit rough, AI's "infinite patience" turns highly customized tasks that humans wouldn't normally bother with into a "why not do it" scenario. He envisioned a future where highly customized worlds are generated on-demand, allowing players to enter novel storylines as NPCs.

However, the experiment exposed a significant flaw: the model cannot efficiently perceive video or "play" within the generated scene to verify its output. Instead, it relies on extremely slow screenshots, leading to imperfections. Karpathy emphasized that the lack of multimodal perception and interactive capabilities remains a major weakness for LLMs.

The article's author conducted a similar test comparing DeepSeek and Kimi in generating 3D simulations. DeepSeek demonstrated blazing-fast generation and solid logic, but its lack of native vision prevented it from self-correcting during the process, resulting in lower final quality compared to vision-equipped models. The author suggested that adding vision capabilities would make DeepSeek a formidable model.

Original post →

Karpathy Tests Opus 3D World Generation: Spatial Skills Impress, But Lacks Visual Feedback

More from coding & agent