MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation
Zhijian Qiao, Xinjiang Wang, Jiajie Chen, Haoming Huang, Meng Li, Chih-Chung Chou, Jing Wang, Shaojie Shen
cs.CV, cs.RO
2026-09-01
MeRoPE maps metric camera displacement into rotary phases so attention logits stay bounded. CamMC is 2.32 on nuScenes vs UCPE 2.45; PanShot TransErr 12.66 vs UCPE 17.91.
Camera-controlled video generation writes extrinsics and per-token viewing rays into attention. Homogeneous encodings such as GTA, PRoPE, and UCPE pack rotation and translation into a non-orthogonal 4×4 block. Translation then enters the score as an unnormalized inner product. Stretch the physical baseline and both logits and feature norms grow without bound; distant frames steal attention from nearby covisible ones.
Three properties cannot hold together: full metric relative pose, strict per-token factorization, and norm-preserving action. An appendix theorem says any continuous finite-dimensional unitary encoding that factorizes per token is blind to metric translation. Something has to give. This paper drops strict per-token factorization and groups by query camera.
MeRoPE splits relative pose into two orthogonal operators and adds a disparity prior.
A minimum-rotation (MinRot) frame carries the optical axis onto each calibrated unit ray. UCPE's cross-product frame degenerates as vertical FoV approaches 180°; MinRot is singular only looking backward. Relative rotation between query and key rays is a repeated 3×3 orthogonal block, so feature norms stay exactly one.
Metric translation is not a homogeneous add. The 3D displacement in the query frame becomes multi-frequency rotary phases on x, y, and z, with log-spaced wavelengths. Each 2D rotation is orthogonal: a longer baseline spins the phase faster and does not scale the logit.
Static-scene correspondence uses spherical disparity anchors. As depth on a key ray runs from infinity to zero, directions sweep a great-circle arc from the infinite-depth ray to the epipole. Bounded angular fractions along that arc become extra MinRot blocks from the query ray. Attention is not masked to pixel epipolar lines, and no dataset-specific metric depth bins are required. Degenerate cases (coincident centers, epipole parallel to the infinite ray) collapse every anchor to the infinite-depth direction.
The full operator is the orthogonal direct sum of disparity, rotation, translation, and the backbone's native spatiotemporal RoPE. Integration follows UCPE: a parallel camera-attention branch with a zero-initialized residual into a frozen video DiT.
All methods share Wan2.2 TI2V-5B. Training mixes nuScenes driving clips with World Engine synthetic trajectories (347.7k sequences) for 20k steps on 64 H20 GPUs. Poses are recovered from generated frames with VGGT-Ω.
| Method | rot°↓ | tr%↓ | CamMC↓ | FID↓ |
| No camera PE | 5.57 | 5.94 | 7.96 | 21.06 |
| URoPE | 1.66 | 2.67 | 2.53 | 19.98 |
| UCPE | 1.39 | 2.95 | 2.45 | 20.20 |
| MeRoPE | 1.37 | 2.84 | 2.32 | 19.82 |
CamMC is the headline metric because it scores rotation and translation together. MeRoPE is lowest, and FVD 134.15 is the best point estimate in the table. URoPE still wins tr% and AUC; dropping metric translation can even raise AUC, which ignores translation magnitude.
On PanShot (pinhole through fisheye), with near-depth normalization off so raw metric displacements are kept, MeRoPE records TransErr 12.66 and CamMC 15.56 against UCPE's 17.91 / 20.13. UCPE remains slightly better on rotation (4.40 vs 4.82). Query-camera grouping costs 12.47 ms per CamSA block versus 5.03 ms for UCPE; full-model training throughput drops only 3.3%.
Driving and simulation need commanded trajectories in meters, not after a global scale wipe. MeRoPE shows metric translation can live in attention if it is a rotary phase rather than a homogeneous add. The comparison is a matched-backbone swap, so the gain is the encoding. It is an adapter on a pretrained DiT. Translation control moves more than rotation, and the gap widens on large baselines and odd optics.
Grouping by query camera relaxes per-token factorization. Isolated CamSA memory is about 3× UCPE; a fused kernel is future work. MinRot is singular looking backward. URoPE still leads nuScenes tr% and AUC, so disparity anchors do not dominate every directional score. Recovered poses come from VGGT-Ω, tying generation quality to that extractor. Training includes 244.1k synthetic clips; real long-tail motion remains thin.