Adding a pixel-reconstruction loss makes latents unable to decode hopper state

hisspikeness · x · 2026-10-08

Tweet 6/ of a research thread: adding a pixel-reconstruction loss to the same latent predictive model makes the learned latents partly encode nuisance factors, so the system state can no longer be linearly decoded. The parent tweet (5/) describes the setup: a MuJoCo hopper with non-deterministic dynamics plus nuisance — random colors, lighting, camera viewpoints, noisy backgrounds — where the latent predictive model's representation allows linearly decoding the hopper's pose and velocity. The contrast shows predictive objectives learn representations closer to the true state than reconstruction objectives.

Related event: New theory shows predictive self-supervised learning provably separates stochastic signals from distractors(8 posts)→

Original post →

More from Embodied

Embodied channel →