WorldStream: Streaming Interactive 3D Worlds with Hierarchical Planning and Persistent State
Xiaoxiao Ma, Weiyang Jin, Yanwen Ma, Haoran Li, Zhixiang Wei, Songchun Zhang, Yaowei Li, Junhao Zhuang, Zeyue Xue, Yuming Li, Zheming Fu, Haoyang Huang, Nan Duan, Feng Zhao
cs.CV, cs.GR
2026-10-21
USTC and JD's WorldStream pairs hierarchical LLM planning with deterministic code to stream an endless 3D world; records keep edits across revisits, language grounding hits 95%.
Learned generative world models render responsive views from player actions, and their consistency horizon keeps extending. But their state lives in frames and latents. There is no persistent place identifier: once a region is evicted from memory it can only be resynthesized, and no individual object can be referenced for later modification. Procedural, code-driven 3D systems hold the opposite position. Scenes are explicit and every object is inspectable and editable, but generation cost is paid before play begins, over a scene fixed in advance. Neither line answers what an open world needs: which part of an unbounded map to prepare next, and what should persist once a prepared region is unloaded.
Open-ended exploration imposes three requirements at once. Terrain must extend past any pre-specified map while per-frame cost stays flat as explored extent grows. Open-ended language must be interpreted against the entities and terrain that currently exist, including regions not yet generated. And a visited place must come back, with its accepted edits, after its geometry has been evicted to bound render memory. WorldStream, from USTC and JD's Joy Future Academy, is a runtime built around those three requirements.
The core move separates thinking from building, with both sides operating on one persistent record.
The world is cut into fixed-size tiles. Each tile holds a record: geographic evidence, the accepted semantic plan, boundary commitments shared with neighbors, object state, user edits, a monotonically increasing revision, and generator/asset versions. Rendered geometry is a compiled view of that record. Eviction deletes only the view; an edit bumps the revision; a revisit recompiles.
Planning runs at two scales. Macro planning reads regional relief, drainage, shoreline extent, and accepted neighbor state, then fixes settlement gradients, ecological structure, transport corridors, and water organization. Micro planning specializes that contract into tile semantics: land-use roles, boundary connections, facility anchors, and generator parameters. Both scales are required. Local-only planning yields terrain-compatible tiles that are regionally unrelated; a regional plan alone does not say how its structure crosses each local boundary. The ablations reproduce both failures.
The decision model (called Jev in the paper) returns, for every typed field, a preferred value plus a distribution over permitted alternatives. The decisive step is joint resolution: search the feasible set for an assignment that maximizes summed log-probabilities plus a cross-tile consistency term, subject to terrain, water, Macro-role, and adjacency constraints. A plan is proven executable before any geometry is built. Taking each field's top probability independently invalidates 63 percent of plans, the hardest number in the paper.
Execution goes to deterministic local code. Terrain, roads, settlements, vegetation, and entities are all compiled from the accepted plan; canonical road ports plus continuous height and water sampling make independently compiled tiles meet seamlessly. One model decision can govern many concrete objects while code resolves coordinates and geometric constraints deterministically.
Streaming uses two frontiers. A semantic frontier plans ahead along the velocity-predicted corridor, supplemented by camera-facing speculative candidates; a narrower realization frontier compiles only the tiles about to enter the active area; geometry behind the player is released. Separate planning and construction budgets cap each stage, with priority ordered by predicted arrival time. Asynchronous consistency rests on three mechanisms: a shared boundary record, a schedule-independent plan key, and a revision stamp on every background job, so stale results are dropped.
Language interaction reads and writes the same records: commands to existing entities, including stateful NPCs that update as they act; edits to the present region, routed through the same compilation pipeline; and intent attached to unplanned regions, consumed by later planning. Commits are optimistic: a mismatched revision or an invalidated target is rejected.
| Experiment | Baseline / comparison | Result |
| Plan executability | Independent decoding 37% | Joint resolution 100% (100 model distributions) |
| Language grounding | 80 bilingual requests | 76 correct (95.0%), errors concentrated in temporal and spatial scope |
| Waiting on 4 km route | No-prefetch and plan-ahead variants (values in Fig. 8) | Directional prefetch cuts total wait to 3.52 s over 100 replays |
| Boundary jump across tiles | No Micro-neighbor context 0.361 | Full hierarchy 0.089; buildings drop from 2,686 to 250 |
| Dense-forest tile compile | 10.78 s before optimization | 3.32 s after, a 3.24x speedup |
Per constraint, independent decoding is weakest on biome (55%) and terrain (69%); joint resolution brings every constraint to 100%. In the hierarchy ablation, the Micro-only variant never forms the requested city and publishes no content. On language, no-action requests and unavailable targets score perfectly, with no entity substitution. On engineering, replacing two compile threads with two persistent worker processes raises warm throughput from 0.234 to 0.464 tiles/s; across 12 newly published tiles, compilation averages 2.134 s with a 3.318 s maximum.
For anyone building interactive 3D environments, games, embodied simulation, or content tools, this architecture splits three usually entangled concerns into independently evolving interfaces: planning writes intent, generators realize it, rendering is a transient view. Upgrading visual fidelity, and the paper's outlook explicitly names a streaming visual decoder consuming the low-poly realization, would not require redoing state management. The transferable finding is broader: structured output is not executable output. The 63-percent figure applies to any pipeline that feeds LLM-generated structured config to a downstream system, and joint constrained resolution is a required step, not polish. The project page and code are public.
The authors' own list: planning ablations run one complete pass per variant with no estimate of model-sampling variance; prefetch replays measure backend tile readiness only, excluding browser installation; the interaction benchmark measures language interpretation against fixed world facts and does not isolate a full edit-evict-revisit cycle; the generation vocabulary is bounded and terrain plausibility rests on curated rules; storage policy for long-running worlds and simulation policy for actors outside the active neighborhood remain open. Further concerns from reading: rendering is low-poly Three.js, far from generative-model fidelity; every evaluation is a self-ablation, with no head-to-head against a learned or procedural baseline; per-tile planning latency and model invocation cost are not reported; a fixed 4 km route with a 3x3 required neighborhood is a narrow streaming test; and the 80 language requests form a self-authored set.