JEPA-Anything: Learning Predictive Models across Different Worlds
Taoyong Cui, Zhongyao Wang, Xinyue Xu, Weiyang Liu, Zhaochen Yu, Yuying Zhang, Qiang Gao, Mengyue Yang, Wanli Ouyang, Pheng Ann Heng, Yingcheng Wu, Zhenfei Yin, Ling Yang
cs.CL
2026-09-18
JEPA-Anything splits JEPA targets into orthogonal factors, beating matched JEPA on all 10 dynamics tasks and cutting Pong intervention error by 34.8%.
A world model predicts another state of the same system in representation space: a masked patch, the frame after an intervention, the next clinical events, atomic coordinates a hundred steps out. JEPA already does this with a context encoder, an EMA target encoder, and a latent predictor, but it usually stuffs the requested world state into one vector and one pathway. When local and global structure, multiple entities, and changes at different scales share that slot, easy or high-variance directions can dominate, and weaker structure gets conflicting gradients.
JEPA-Anything, from PhAI Labs with CUHK, Fudan, Oxford, Princeton and others, keeps the interface fixed and slices the target into complementary factors, each with its own predictor, then synthesizes a full latent state. "Anything" names the breadth of that interface, not one universal weight file.
A domain adapter turns observations into tokens and structural descriptors, then a view sampler picks context and targets. An online encoder reads context; an EMA target encoder supplies the latent to predict. Orthogonal predictive factorization (OPF) learns projectors that split the target into K subspaces of width r with Kr equal to the hidden size, and a dedicated predictor for each factor. Training asks for near-orthonormal bases within a factor, near-zero overlap across factors, and a variance floor so coordinates do not collapse. Synthesis uses a pseudoinverse; when projectors are near-orthogonal, factor error is not amplified.
Downstream use splits. Ordinary readouts discard the heads and keep the online encoder. Rollouts, planning, and intervention prediction keep the projectors and feed the synthesized state forward. Factor coordinates also serve as a diagnostic: intervention nomination, spectral modes.
Seven domains share this core. Adapter, backbone, splits, and base loss stay matched; only the monolithic predictive term is swapped for OPF.
On controlled visual binding, OPF-pretrained DINOv3 and SigLIP2 improve injective held-out-cell accuracy, collapse, and grid recovery versus matched standard JEPA, by a few thousandths. On single cells, PBMC finetuned AvgBIO rises from 0.7830 (Cell-JEPA) to 0.8301, Norman perturbation Pearson from 0.787 to 0.814. A synthesized future patient state is read into more than 1,000 event risks; mean PRAUC beats the monolithic health-state JEPA, with ranking shown in a figure rather than a tabled mean.
On CITRIS Interventional Pong, single-intervention one-step MSE falls from 0.009541 to 0.006218 (−34.83%), combined-intervention −12.90%, six-step free rollout −8.58%. A geometry audit drives cross-factor overlap to numerical noise and the condition number to about 1, versus 438 for unconstrained multi-head. All 10 matched dynamics tasks improve. APEBench Burgers late-segment MSE drops about 49.5%. Across water, quartz, paracetamol, and benzene, one-step error and 100-step RMSD are the lowest among compared methods.
Planning is not uniform: CEM return is higher on Walker2d and HalfCheetah, while Hopper favors standard JEPA. Masking any factor raises 20-step rollout error. Wet-lab work nominated IL-18 plus CD73 blockade from factor coordinates and then saw stronger killing in co-cultures, organoids, tumor fragments, and mice. On simulated orbits, latent modes fit a Keplerian slope of −1.4991, close to −1.5.
This is a drop-in training principle, not another domain-specific world model. Existing JEPA pipelines can add the OPF regularizers and swap adapters for new observation geometry. For representation work, it makes capacity allocation an auditable set of subspaces. For scientific use, factor coordinates can nominate interventions and be checked against known laws. Orthogonality itself does not prove causation.
Hopper planning return is worse, so factorization is not free. Clinical PRAUC is figured, not tabled. The wet-lab result is experimental support for a nominated combination, not a randomized proof that a factor equals a mechanism. The Kepler diagnostic uses unlabeled simulated trajectories to recover a known law, not to discover a new one. Shared architecture does not imply aligned factor semantics across domains. The paper states the same limit: orthogonal factors do not by themselves establish causal mechanisms.