A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
Antonio Sclocchi, Alessandro Favero, Matthieu Wyart
stat.ML, cond-mat.dis-nn, cs.CV, cs.LG
2024-02-27
A phase transition in diffusion: past a critical noise level the class is recovered only at random while low-level detail changes smoothly; on ImageNet the class drops at t≈T/2.
Diffusion models generate high-quality images, but what each step of the reverse process actually does is unclear. A natural intuition is that real data is hierarchical: high-level features (what the image is) and low-level features (how it looks locally) live on different levels. The paper asks whether different time steps of the diffusion process correspond to different depths of that hierarchy.
The authors use an analytic hierarchical generative model (Random Hierarchy Model, RHM): a class label expands layer by layer into features and sub-features down to pixel-level tokens, with tree depth L, branching s, and vocabulary size v. They then derive, at a given noise level, how high in the tree the reverse process can recover.
They model "can we still infer the original class?" as a message-passing iteration up the tree and get a fixed-point equation. A key parameter sf (the effective branching ratio per layer) decides whether a phase transition exists.
The theory gives a critical noise ε (a critical time t):
High-level features (class) collapse in a sharp jump at t; low-level features (texture, local detail) evolve smoothly throughout.
On ImageNet they trained a class-unconditional DDPM and used a ConvNeXt-Base probe (96.9% top-5 accuracy), measuring cosine similarity between initial and generated images across layers on 10,000 images. Class similarity drops sharply at t≈T/2 (about step 500 of 1000), while low-level similarity stays smooth. Past the transition the generated image has a new class but can still reuse the original's low-level elements: a leopard's eyes, nose, and ears recomposed into a wolf.
It turns "diffusion works on different hierarchy levels at different noise levels" from a metaphor into a falsifiable phase-transition claim. For practitioners, noise scheduling and layered control (fix the class, then fill in detail) now have a theoretical anchor, not just intuition.
RHM is a toy; real image hierarchies are far messier than a fixed tree. The transition time t depends on model and data, and T/2 was checked on only one ImageNet setting. Whether the result transfers directly to text-conditioned generation, where the prompt replaces the class, is open.