Fast-JEPA: Advancing the Success-Latency Frontier for Latent World Models
Yao Zhang, Bin Yu, Shijie Lian, Haishan Liu, Changti Wu, Yu Wu, Kai Chen
cs.RO, cs.AI
2026-11-19
Fast-JEPA scores complete candidates with one terminal prediction instead of recursive rollout: 96.5% success at 10.88 ms planning across four control tasks, 2.41x faster than INTACT.
Latent world models let a controller compare action sequences by predicting their outcomes in representation space. The bill comes due at test time: the standard way to score a candidate is a recursive rollout, where each predicted state is fed back as the next input. A sequence of H chunks costs H world-model calls, multiplied by candidate count and refinement rounds. Prior systems either live with hundreds of milliseconds of planning latency or drop the search and commit to a single proposal.
For robot control this is a real constraint. The environment keeps moving while you plan, so every millisecond of planning widens the gap between the world you planned for and the world you act in. Fast-JEPA pushes the success-latency Pareto frontier up and to the left: keep the benefits of candidate search, cut the per-candidate evaluation cost from a chain of predictions to one query.
The paper rests on an asymmetry in what the planner knows. While actions are still being constructed, later chunks are unspecified, so intermediate outcome predictions carry information for choosing them. Once a candidate is complete, all its actions are known, and under a terminal visual-goal objective, comparison only needs the predicted endpoint. Intermediate states add nothing to the score.
Fast-JEPA allocates prediction accordingly, in three stages:
Training is shared and cheap: one forward world model supervised at multiple horizons from the same observed anchor, and an action model trained against two conditioning targets (next latent state and segment endpoint). Total cost is 5.95 GPU-hours versus 6.67 for INTACT.
In complexity terms, rollout-based CEM scales as O(NIHW) while Fast-JEPA lands at O((H+NI)W): each candidate costs one prediction instead of H.
Four tasks (PushT, Reacher, Cube, TwoRoom), 100 episodes per task averaged over three independently trained models, warmed batch-one latency on an RTX PRO 6000:
| Method | Avg. success | Planning latency |
| Fast-JEPA | 96.50% | 10.88 ms |
| INTACT | 95.58% | 26.19 ms |
| INTACT Guarded A | 96.33% | 127.09 ms |
| Fast-LeWM + SC | 84.58% | 586.15 ms |
That is a 2.41x speedup over INTACT with a slightly higher success rate. The ablations isolate the mechanism: removing proposal construction drops success from 96.33% to 68.17%, so search needs a good starting point; dropping refinement keeps 93.83% at 5.53 ms, so refinement buys about 2.5 points. The sharpest evidence is the scoring comparison: recursive scoring gets 96.08% at 20.54 ms while direct terminal scoring gets 96.33% at 10.78 ms, with the scoring step itself falling from 11.16 to 2.32 ms. Recursive rollout carries no extra information under this objective. Across a 32-configuration search-budget sweep, Fast-JEPA stays within 9.7-16.8 ms while INTACT Guarded A spans 71.8-197.2 ms.
On the new DynaGoal benchmark (original-task models reused, no fine-tuning): 73.83% average under the Shift protocol (changing goals and geometry) versus 56.25% for INTACT, including 85.33% versus 45.33% on the moving-doorway TwoRoom task; 87.67% under the Deadline protocol, where planning time counts against the task budget, versus 75.33%.
For anyone building model-based controllers, the transferable finding is that candidate evaluation does not need recursive rollout under a terminal objective. Any CEM/MPC-style pipeline with a world model can audit where its prediction budget goes. The 10.88 ms operating point, paired with full candidate search, sits at the upper-left edge of the frontier among all locally evaluated configurations.
DynaGoal itself is a solid contribution. Shift, Deadline, and Delay tie planning speed to planning quality under changing conditions, and Deadline in particular charges planning time against the task budget, forcing methods to expose their true cost.
Honest framing: on the static tasks the headline gain is incremental, 96.50% against 95.58%. The real jumps are in latency and in success under dynamic conditions.
The authors acknowledge several: retaining the proposal guarantees the predicted goal cost cannot worsen, not that realized success improves; success on Reacher and Cube falls sharply as response delay grows; PushT remains below Fast-LeWM + SC (90.00% versus 92.00%).
A few more caveats from the numbers. Local reproductions diverge widely from published figures: Fast-LeWM reports 85 ms in its own paper but 389.99 ms in local runs here, so cross-paper latency comparisons are unreliable. All four tasks are pixel-based simulation with modest dimensionality, with no real-robot or high-dimensional action-space validation, and the complexity analysis assumes action-generator and world-model calls cost the same. Under the Delay protocol, Fast-JEPA's Reacher and Cube results collapse as delay grows, which says the proposal-plus-refinement structure tolerates environment drift only on some tasks, not in general.