FactorJEPA: Factorizing Monolithic Futures into Layout-Agent-Interaction Channels for Crowded and Chaotic Global South Urban Worlds
Kapil Wanaskar, Gaytri Jena, Aman Chadha, Vinija Jain, Vasu Sharma, Amitava Das
cs.AI, cs.CV, cs.LG
2026-08-02
FactorJEPA factors JEPA's future into layout, agent, and interaction channels, separating from the next best method on all four diagnostics over a 1,000-hour Indian-street dataset.
A world model predicts what happens next from observed video, and JEPA (Joint Embedding Predictive Architecture) is a leading approach: instead of reconstructing future pixels, it predicts a future embedding in an abstract representation space, which avoids the waste of modeling reconstruction noise. This works well on lane-structured, low-density driving footage.
What JEPA evaluations have largely skipped is a very different setting: the streets of India and South Asia, where lane markings barely matter, pedestrians, cars, motorbikes, rickshaws, carts, and animals share the road, occlusion is constant, and progress depends on moment-to-moment social negotiation. The authors call this regime DENSEWORLD and point to an overlooked failure mode. In dense scenes, layout, crowd density, visibility, and interaction pressure are tightly correlated, so a monolithic predictor that compresses the future into one latent takes shortcuts: it uses crowd texture as a stand-in for interaction and road geometry as a stand-in for motion, without recovering the scene's actual structure. To study this they release DENSEWORLD, about 1,000 hours of video across 22 Indian Tier-1 and Tier-2 cities from drive-through, walk-through, and aerial viewpoints, with faces and license plates blurred.
FactorJEPA replaces the monolithic predictor with three explicit channels:
The three channels recombine in a block structure, so the future embedding is a sum of a layout term, an agent term, and an interaction term. A separation loss penalizes linear and nonlinear leakage between channels, so the model cannot quietly mix them back together and resume shortcutting. The visibility gate is calibrated with its own binary cross-entropy term.
The authors call the whole change predictor surgery: only the monolithic predictor is replaced, while the online encoder, momentum target encoder, and masking policy are carried over from pretrained V-JEPA. Training follows a staged curriculum, Layout then Agent then Interaction, that progressively unfreezes the top encoder blocks. Supervision targets are not hand-labeled; a frozen DINOv2 pipeline extracts regions, visibility, and relative motion from the video and converts them into layout, agent, and interaction targets.
Four diagnostics are used: Future-frame L1 (future-latent accuracy, lower is better), Causal L1 (sensitivity to interventions, lower is better), Mask-ratio slope (error rise as the frame is progressively masked, lower is more robust), and Motion cosine (linearly decodable motion information, higher is better).
The bluntest absolute result: on the motion probe, ten mainstream frozen encoders, including the 2-billion-parameter V-JEPA 2.1, all land in a 37.5 to 44.4 percent Action top-1 band, against a 19.5 percent majority baseline. The strongest frozen model is clearly above the majority class but far from usable; after adaptation the band only reaches 50.3 to 53.2 percent. Frozen motion-cosine values sit at 0.004 to 0.019, essentially zero.
For FactorJEPA versus the next-best method, the multipliers measure separation in units of the paired 95% confidence interval, meaning how robustly the ranking holds, not how many times the raw metric improved. The authors stress this explicitly. Under a stratified 10k-clip protocol on the 2B and 1B backbones, Future-frame L1 separates by 6.3x and 4.8x and Causal L1 by 2.3x and 2.7x, while Mask-ratio slope separates only at 1B (1.9x) and not at 2B (0.9x). Move to full 115k-clip training at 1B and all four diagnostics separate, reaching 43.3x for Mask-ratio slope, 33.2x for Future-frame L1, 20.0x for Motion cosine, and 13.9x for Causal L1. Method rankings track closely between 2B and 1B, with Spearman rho from 0.895 to 0.978 and Causal L1 the highest at 0.979.
The cost is a trade-off: on Motion cosine, plain fine-tuning beats FactorJEPA. Structured prediction buys a firmer grip on future-latent fidelity and intervention sensitivity while partly sacrificing linearly decodable motion signal, and only full-scale training closes that gap.
Two pieces of value. On method, this makes world structure a first-class prediction target instead of burying it in one latent for the model to organize on its own, which gives a reproducible path to interpretable, intervention-sensitive prediction in dense and occluded scenes. On data, DENSEWORLD fills a geographic blind spot in JEPA evaluation. For anyone building autonomous driving or robotic perception in the unstructured traffic of South Asia, Southeast Asia, Africa, or Latin America, this is a ready-made baseline and diagnostic suite, with data and weights released.
The authors are candid. Interaction is the main open problem: only sparse pairwise coupling is modeled, while group motion, multi-agent negotiation, and temporally persistent interaction events are all out of scope, and street negotiation is precisely those. Factor semantics are aligned only at the channel level, produced by a DINOv2 pipeline rather than coordinate-level identification, so a different teacher could yield a different but equally predictive partition. The evaluation targets predictive structure, not full causality: Causal L1 measures intervention sensitivity, not causal identification or counterfactual reasoning. Geographic coverage is one large-scale realization across 22 Indian cities, not a stand-in for the whole Global South. Full training runs only at 1B, the 2B model is evaluated on the stratified 10k subset, and the latent-to-RGB decoder can blur, drop small agents, or collapse uncertainty into a single future.
One honest caveat for readers: the headline 43.3x is a statistical separation multiplier, not an absolute gain, and the large numbers invite misreading as a 43-fold performance jump.