kalomaze: Diffusion autoencoders are lossy tokenizers — data is gone before training

kalomaze · x · 2026-11-04

kalomaze argues that essentially all latent diffusion models learn an autoencoder for tractability that is effectively a lossy tokenizer: the bottleneck latents are too crude to preserve underlying information, so much distinguishing detail is destroyed before the model ever trains on it — likely unrecoverable even from the training data as the model actually saw it. Almost nobody runs diffusion on raw pixels or audio at scale. This adds a overlooked technical wrinkle to debates about memorization and training-data extraction: the bottleneck is the encoder, not the model.

Original post →

kalomaze: Diffusion autoencoders are lossy tokenizers — data is gone before training

More from Multimodal