Magpie: Real-Time World Renderer for Interactive Games
Xiaoyu Zhan, Xinyu Wang, Xiaohong Zhang, Huanjie Zhu, Tengjiao Sun, Pengcheng Fang, Jiaxing Yu, Yanwen Guo, Dongjie Fu
cs.CV
2026-08-27
Magpie keeps gameplay in the engine and paints from white-box frames with Wan2.2-5B: ~32.2 FPS on one H100, ~1.55 s action-to-frame, ~300 hours of Unreal play.
A game is not a video that unrolls on its own. The same actions and states have to keep stable meaning under authored rules, or designers cannot tune numbers, reproduce a scene, or grow a prototype into a product. Collision volumes, cooldowns, and hidden triggers barely show up in the final pixels, and they are still the source of truth for gameplay.
Video generators already produce convincing materials and lighting. Used alone as a game, they do not hold rules and state. Keyboard-conditioned or text-event world models also cannot guarantee compliance with authored mechanics. If a prototype still has to finish modeling, texturing, rigging, lighting, and optimization before anyone can judge the feel, visual assets stall iteration.
Magpie splits the jobs. Designers lay out scenes and write rules in the engine. The Game Engine resolves player actions, keeps world state, and emits a white-box frame: final materials and complex lighting stripped out, occupancy, collision bounds, main silhouettes, and visible state changes kept. An independent Render Server turns that observation into the displayed image.
At session start the Render Server takes a text prompt and a first-frame image to set style and character appearance. After that, white-box frames are the only continuing denoising condition. Camera poses are used only to retrieve history for the current viewpoint. Player actions, object properties, and event signals stay in the engine and never enter the generator. A visual error can muddy feedback; it cannot change collision or progression.
The renderer is built on Wan2.2-TI2V-5B with Helios-style bounded multi-scale context, hierarchical generation, and few-step distillation. Three ways of injecting the white-box condition were tried: packing it into the noisy latent (tight structure, crushed detail), AdaLN (works inside a chunk, jumps at chunk boundaries), and cross-attention (the practical quality/structure trade-off). The deployed model uses cross-attention. History is assembled in a fixed order: an early anchor block holds initialization appearance, FOV-overlap retrieval supports revisits, and recent generated chunks keep local continuity. Distillation compresses a multi-step teacher into a 3-step student, then self forcing trains on the model's own generated history. Deployment uses LightTAE and FP8.
Training data come from 30+ Unreal scenes and about 300 hours of human play at 1920×1080 and 60 FPS, with paired high-fidelity and white-box streams plus camera poses. Operators play like players: walk, run, jump, drive, sit, collide, and a large idle share. Dense combat and multi-character cooperation are not yet covered at similar scale.
This is a systems paper. It does not compare visual quality against Matrix-Game or DreamX under a shared protocol. What it reports is runtime.
On one NVIDIA H100, the distilled 5B renderer spends about 620 ms on a steady-state chunk of 20 frames, or about 32.2 FPS on the compute side, with peak GPU memory around 34 GB at 1280×768. The interaction pipeline runs at 24 FPS. End-to-end latency is gated by chunk boundaries: the engine records 20 white-box frames in about 0.83 s, transfer and encoding take about 0.1 s, inference and decoding take 0.62 s, so the matching visual response arrives after about 1.55 s. Later chunks display over an 830 ms window.
The authors evaluate generative rendering on long-horizon memory, adherence to the white-box condition, and overall look. Figures show the base autoregressive model and the distilled real-time model. LPIPS, FVD, and a quantitative structure-alignment score are not reported.
For gameplay designers, a white-box prototype can approach the intended look before production assets exist, while rules and state remain inspectable, editable, and reproducible. For video world-model work, the paper draws a hard boundary: the model should not guess the rules; it should paint an outcome the engine has already resolved.
It is not a production renderer yet. 1.55 s of feedback is unusable for shooters or fighters that need immediate feel, and 32.2 FPS of compute throughput is still short of console expectations. Closer uses are playtests, style exploration, and early experience reviews.
The authors are direct. Chunk-wise pre-recording dominates latency; a frame-wise streaming path is needed so engine, encode, generate, and display overlap. A single RGB white-box frame is depth-ambiguous, so thin structures and same-colored regions can land in the wrong space. Under fast motion, large viewpoint changes, or occlusion, generated geometry and character placement drift. Appearance supervision is limited, there is no audio, and memory is still a bounded 2D history rather than an updatable 3D scene. 34 GB also will not run on the edge.
The paper also never turns "does the generated frame obey the white-box" into a reportable structural error. Without that number, it is hard to know how tightly the system actually locks the rules.