ME-World jointly denoises two ego streams, reaching real-world Senv of 0.468

Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction

Dahyun Chung, Siyoon Jin, Hyunwook Choi, Honggyu An, Junyoung Seo, Hyunsung Kim, Seung Wook Kim, Seungryong Kim

cs.CV, cs.AI

2026-10-09

ME-World denoises two ego videos in one sequence with shared poses and scene memory, reaching real-data Senv 0.468 and Supdate 0.466.

What problem this solves

Egocentric world models predict the next first-person frames from an agent's actions. Most of them model a single self: one pair of hands, one head, one future. Multi-person VR and embodied simulation put two people in one changing scene. When one person reaches for a pan, the hand has to land in their own view, and the other view has to show the same body and the pan's new place at the same time.

Recent multi-agent world models compress actions into locomotion, camera paths, or discrete commands. Those controls can line up viewpoints. They do not carry hands, arms, body pose, and head turns. The paper names three failures, each drawn in Fig. 2. The action must match across views. Layout, appearance, and objects must not diverge. A state change caused by the interaction has to show up in every stream.

Method

ME-World fine-tunes the Cosmos-Predict2.5 2B multiview video DiT. The video VAE and text encoder stay frozen. Real and synthetic data are fine-tuned separately. The formulation allows N agents; the experiments use two. Inputs are both agents' future head and body motion, text, the first frame, and a shared observation history. The output is two future ego videos.

• Joint denoising. Noisy latents from both streams are concatenated on the token axis and denoised in one self-attention pass, so each view can read the other before either commits to an outcome. Both streams share RoPE, which lines up their space-time grid. A learned agent embedding marks who each token belongs to. Text keeps a separate cross-attention path per stream, because the two people can look at different parts of the room. The moved pan is resolved once.
• Shared action conditioning. A stream that sees only its own action renders the other person inconsistently. Each stream therefore receives every agent's motion. Ego-visible hands and arms are drawn as a skeleton. Other agents' 3D poses are projected into that camera. Head motion becomes a Plücker ray map in one canonical frame, so the two camera paths are geometrically comparable. Pose features come from the frozen VAE, ray features from an MLP, and both are channel-concatenated with the noisy latent.
• Shared environment memory. History frames come from old viewpoints, so they are weak as raw references. The model picks the history image with the largest valid reprojection, warps it with the target rays, and uses SAM3 to cut out people so a stale pose is not pasted onto the new view. Warping covers only the overlap and messes up clean appearance. A second path greedily picks unwarped history frames as anchor tokens, appends them to the joint sequence, and lets both streams attend to the same clean references. The synthetic pipeline uses 8 anchors.

Training is rectified flow with AdamW, learning rate 3e-5, and a global batch of 4 on four A100s. Inference uses 35 denoising steps at guidance scale 7.0. Actions are given. The model does not choose what either person does next.

Results

Real video is CoMind, synchronized two-person egocentric recordings. The synthetic set retargets 1,822 two-person motions from Inter-X and InterHuman onto 100 avatars, 7 scenes, and 53 placements, giving 4,736 paired 77-frame clips at 504×504 and 10 FPS.

Consistency is DINOv3 cosine similarity. Co-visible points come from metric depth and ground-truth cameras, and are kept only when reprojected depth agrees within 10 cm. Senv scores regions the history warp does not cover. Supdate scores regions that changed relative to a valid warp. Sid, synthetic only, masks the other person with SAM3 and takes the max similarity to reference renders from 12 azimuths.

Released checkpoints in the first comparison are not fine-tuned on this data. Multi-view baselines also see the other stream's ground-truth video. Lower rotation error and FVD are better.

On real data, ME-World also reports translation error 0.042, Hand-F1 0.888, Hand-mIoU 0.531, full-body F1 0.822, PSNR 19.849, and LPIPS 0.289. GEN3C still receives a ground-truth reference stream, and its hand mIoU is 0.087. LingBot-World's FVD of 689.9 sits close to 637.9, with a weaker room and partner pose. Synthetic Sid is 0.561, next to 0.518 for JointControlVideo. Synthetic Senv is 0.441 against GEN3C at 0.414, a narrower gap than on real video. Synthetic rotation error is 9.057, well above the real-data 2.256.

With the same backbone, training data, and shared action conditioning, adapted MetaWorld reaches Senv 0.421, Supdate 0.413, rotation error 4.086, Hand-mIoU 0.379, full-body PCK 0.787, and FVD 704.4. ME-World's matching numbers are 0.468, 0.466, 2.256, 0.531, 0.881, and 637.9. Full-body F1 is 0.816 versus 0.822. Solaris and γ-World fall to 0.146 and 0.188.

On the real-data ablation, fine-tuning Cosmos alone leaves Senv at 0.290. Adding the ego hand skeleton and history warp raises Senv to 0.401 and drops FVD to 558.6. Shared action conditioning moves partner full-body PCK from 0.416 to 0.809, while FVD rises from 554.7 to 651.9. The full model's FVD is 637.9, worse than 541.7 for joint generation plus environment memory without shared action conditioning.

Three-agent clips and a 221-frame rollout are qualitative only. A third agent is another token stream. Long videos are chunked, and the next chunk is conditioned on one or two frames from the end of the previous one. No consistency numbers are reported there.

Why it matters

Given both people's future body and head poses, one sample produces ego videos that agree on the scene. The backbone is 2B, and the fine-tune runs on four A100s at batch 4.

Against untuned GEN3C, full-body PCK moves from 0.456 to 0.881. GEN3C was not trained on CoMind, so domain shift sits inside that gap. Against MetaWorld adapted to the same data and the same action interface, Senv is 0.468 versus 0.421 and rotation error is 2.256 versus 4.086. That is a clear step on the same track. Actions still come from outside the model, so this is not a closed-loop planner.

Limitations

The scored benchmark is two people and 77-frame clips. The three-person and 221-frame demos have no consistency numbers; they show that another stream of tokens can be appended. Joint attention grows with head count and duration. The paper defers larger groups to more efficient cross-agent communication and memory.

Frame count and conditioning in the public-checkpoint comparison do not match. GEN3C pads the camera path to 121 frames and cuts it back to 77. EgoSim emits 61 frames. Every adapted baseline in the controlled comparison inherits ME-World's shared action conditioning, so that table compares communication and scene memory, not the original systems. MetaWorld has no public code and was reimplemented from the paper.

Senv and Supdate ask whether co-visible regions look like the same place, and the correspondence uses ground-truth depth and cameras. Neither score knows whether a hand actually grasped the object. Sid does not cover real people. There is no curve for what happens when history coverage shrinks. The full model also loses on FVD to a thinner ablation: tighter cross-view agreement is not the same thing as a closer video distribution. Thirty-five denoising steps plus reprojection remain far from the real-time multi-agent simulator mentioned in the conclusion.

Terms

• ego stream: a first-person video whose visible region is set by that agent's head motion.
• Plücker ray map: a per-pixel encoding of camera rays, used here for head-driven viewpoint change.
• Senv / Supdate: DINOv3 similarities for newly generated scene regions and for regions changed by an interaction.
• rectified flow: a training objective that straightens the path from noise to clean video latents.
• PCK@10: the fraction of the other person's body joints that fall within 10% of torso scale.

Source

• arXiv 2610.12299v1
• PDF
• Hugging Face
• GitHub · https://github.com/cvlab-kaist/ME-World

Related papers

• Lag-chosen gain cuts unguided SiT-XL ImageNet FID from 28.0 to 12.2 at NFE 25
• StreamMAE: self-supervised pretraining on 95 hours of continuous video matches i.i.d. training
• VibeEdit edits images from marks drawn on the canvas, beating text-prompted baselines 79.9 vs 67.4
• Pumpire finds depth leaderboards do not predict tape-measured distances
• Open models top out at 23% while GPT-Image-2.5 reaches 56.6%
• OneSearch-VL-8B gains 20.2 and 17.6 points on multi-image and video research

All paper explainers