Grounding LLMs with JEPA world models trained in simulation: a research proposal

Full_Promotion4522 · reddit · 2026-09-03

A Redditor proposes a research idea: LLMs learn statistical token relations ("falls"↔"gravity") without grounded physical intuition — essentially the Mary's Room problem. The proposal:

Train a JEPA-style model inside physics simulation (MuJoCo or a simple 2D env), predicting representations of future states in an abstract embedding space rather than pixels or tokens. Wrong physics yields unforgiving loss, unlike next-token prediction.
Let the embedding space encode real physical structure — object permanence, momentum, trajectories — because that's what makes prediction possible.
Freeze those representations and attach them to an LLM as a conditioning signal, giving it linguistic physics knowledge plus grounded intuition it can "run" forward — a computational primitive rather than a propositional fact.

The hypothesis: downstream learning speeds up significantly since the LLM needn't rediscover that objects fall. Adjacent work: V-JEPA (predicts future frame representations for video), DreamerV3 (latent world models for RL), but this exact combination appears unexplored.

Open questions posed to the community: any missed prior work? What's the right interface (prompt concatenation vs cross-attention)? Will the sim-to-real gap kill transfer? Worth building a small prototype?

Original post →

Grounding LLMs with JEPA world models trained in simulation: a research proposal

More from AGI Musings