IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves
Feyza Yavuz, Mert Bülent Sarıyıldız, Diane Larlus
ECCV 2026
cs.CV
2026-08-25
Optimizing Dead Leaves with four teachers hits 78.2% ImageNet at 1M vs 66.9% raw; at 1K, IDeaL beats a 1K ImageNet subset on classification.
Multi-teacher distillation tries to pack several visual backbones into one student. UNIC, RADIO and DUNE all assume a pile of real images: the teachers' own training sets, or a web scrape such as DataComp-1B. Foundation models are increasingly trained on data that cannot be shipped, so that assumption is failing.
This ECCV 2026 paper from NAVER LABS Europe takes the constraint to the limit: no real images at all. Earlier work swapped ImageNet for class-conditional Stable Diffusion renders. Here even the generator is gone. Images start as procedural noise and are then optimized, using teacher attention, into a distillation set.
The distillation loop copies UNIC. Four ViT-B/16 teachers, all trained on ImageNet: self-supervised DINO and iBOT, supervised DeiT-3, and a classification-finetuned dBOT-ft. The student is also ViT-B/16 and matches each teacher's CLS and patch features. Shared teacher data means later gaps can be blamed on the distillation images.
Three procedural stand-ins are tried first: Gaussian noise, FractalDB, and Dead Leaves. Dead Leaves are overlapping colored shapes drawn from a formula. Their power spectrum resembles natural photos; they carry no object semantics. They beat the other two noises, but still lag real images.
IDeaL starts from Dead Leaves, treats pixels as parameters, and backprops through frozen teachers with two decorrelation losses:
Both losses are summed over all four teachers. Generation and distillation are decoupled: optimize the images, freeze them, then train as if they were a real set. No labels, no classification heads, so self-supervised teachers can join. Implementation: pools of 250, random subsets of 40 for 10 steps, 4000 steps per pool, Adam at learning rate 0.1.
At 1M samples, the IDeaL student reaches 78.2% ImageNet top-1, against 66.9% for raw Dead Leaves and 83.2% for UNIC distilled on 1M real ImageNet images. Mean transfer over 15 datasets is 70.6% (65.5% / 73.1%), ADE20K mIoU 34.5% (29.1% / 39.0%), NYUd depth RMSE 0.573 (0.632 / 0.531). Relative to Dead Leaves those are roughly +17%, +8%, +18%, +9%. Transfer, segmentation and depth all beat the weakest teacher. ImageNet classification sits at 78.2%, a hair under DINO's 78.4%.
At a 1K budget, IDeaL beats a 1K ImageNet subset on classification: 74.1% vs 72.6% on ImageNet, 68.6% vs 65.3% on transfer. Dense tasks still favor real photos (mIoU 33.9 vs 34.2, RMSE 0.594 vs 0.566). Distilled into a ViT-S/16 student, 1K IDeaL wins all four metrics against 1K ImageNet.
Joint optimization over four teachers beats stitching four independently optimized 2.5K sets: 77.0% vs 75.1% ImageNet at 10K. Ablations say patch decorrelation does most of the work; image decorrelation is a complement; both together win.
The old synthetic-data scaling problem remains. From 10K to 1M, IDeaL barely moves; real ImageNet keeps climbing. Past a small budget, procedural noise does not ride scale.
This is a controlled demonstration that real images can be removed from multi-teacher distillation without collapsing classification or dense transfer. For teams that cannot ship teacher data, 1K to 10K optimized noises already beat a random 1K ImageNet draw. The set is generated once and reused across student sizes; the ViT-S run supports that.
It is not a drop-in replacement for RADIO-scale distillation on a billion web images. All four teachers saw only ImageNet. Distilling CLIP or SigLIP teachers trained on private data is a different job.
There is no standalone limitations section; the tables draw the border. IDeaL does not scale with set size: 100K and 1M are nearly tied, while real images keep gaining from 10K to 1M. The study is locked to UNIC's four ImageNet-only ViT-B/16 teachers. No heterogeneous resolutions, no teachers pretrained on different corpora. The objective is cosine decorrelation of attention features, with no reconstruction and no class conditioning, so the images are not meant to look real. That is the design, and it also means the student never aligned to teachers on photo-like inputs. At 1K, depth still loses to real images. The abstract's "match or surpass" claim is about classification.