HelloWorld: Enabling Socially Interactive Characters in Video World Models
Liangyang Ouyang, Ruicong Liu, Xuangeng Chu, Kaipeng Zhang, Yoichi Sato
cs.CV
2026-08-06
Video world models steer cameras, not characters. HelloWorld keeps the base model's social prior by self-distillation, then pins each interaction to the button-press window with a training-free attention mask.
Video world models can already steer cameras, respond to keyboards, and generate events, but the characters in the frame are just moving pixels that never greet the viewer. Existing social world models are mostly text-only LLMs, lacking multimodal visual interaction. HelloWorld fills this gap: press a button and the character turns to the camera, waves, nods, or says a short greeting inside a precisely timed window, while the camera keeps following its trajectory.
The base model is LTX-2.3, and the authors find it already generates rich social interaction natively; what it lacks is camera control. So the real problem is how to add camera control without losing that interaction prior. Three pieces.
A self-distillation pipeline. Four-part prompts with both interaction and camera motion make the base model synthesize clips; Pi3X recovers per-frame camera poses; the first frame is lifted to a point cloud and reprojected along the trajectory into a warp video that serves as explicit camera-motion conditioning. Reprojection holes are dropped by a visible-token selector, and a rank-32 LoRA is finetuned under flow-matching loss. Crucially the camera prompt never enters the loss text, so camera following is controlled solely by the warp video, preventing prompt leakage. This uses only 156 self-synthesized clips and 2000 steps.
Why interaction and camera motion must be trained together: training on real camera-only video (no interaction) only teaches the model to fill warp holes, and the resulting actions are not directed at the viewer. Co-training preserves the interaction prior while learning camera conditioning.
A training-free temporal attention mask. A button press opens a time window; a mask is added to the DiT cross-attention logits so frames outside the window cannot attend to the interaction-prompt tokens. The interaction takes effect only inside the window; outside, the character stays in ambient motion. No training, negligible overhead.
HelloWorldBench. 120 high-quality images (humans, animals, toys, robots) times 4 camera trajectories times early-mid-late timing, yielding 400 samples, 264 characters, 101 interaction types. Three social metrics (action accuracy, timing accuracy, gaze deviation toward the viewer) plus three conventional ones (background consistency, aesthetics, camera controllability).
The decisive advantage is temporal control.
| Method | ActAcc | TimeAcc | GazeDev | CamCtrl |
| LingBot-World | 50.5 | 39.5 | 59.0 | 62.6 |
| LTX-2.3 base | 42.5 | 52.6 | 38.1 | 31.4 |
| HelloWorld | 41.4 | 81.7 | 40.2 | 82.9 |
TimeAcc of 81.7 versus roughly 30 to 39 for every baseline (three-way choice is 33% random) is the mask's win. Aesthetics 5.27, camera controllability 82.9, and background consistency 96.9 are all the best in the field. But ActAcc of 41.4 is not the highest; LingBot-World reaches 50.5, which the authors concede. An ablation shows removing the mask actually raises ActAcc to 42.5, because the character performs the action throughout, at the cost of timing localization; that is a deliberate tradeoff. Inference is only about 20% slower than the LTX-2.3 base per clip.
For anyone building interactive video and world models, button-triggered, precisely timed, viewer-directed social interaction is a new capability axis, achieved with just 156 self-synthesized clips and a training-free mask, so the cost to adopt is low. The warp-video camera conditioning and the attention-mask temporal localization are two reusable components.
The authors concede it is not real-time: generation runs on preset trajectories and interaction scripts, constrained by the base model and compute; persistent identity and sustained multi-round interaction are future work. Current interactions are short gestures like waving, nodding, or a quick hello, with no sustained dialogue; the focus is single-character, not multi-character conversation in one clip. Only 156 training videos means generalization breadth is an open question. ActAcc loses to LingBot-World; the wins are timing control and image quality, not raw action recognition. Gaze deviation is measured on only 217 human samples, so the viewer-directedness claim is not quantified for animals and toys. The whole benchmark is self-built, with no third-party reproduction yet.