Fourier Self-Supervision for Fine-Grained Generalized Category Discovery
Sarah Rastegar, Mina Ghadimi Atigh, Pascal Mettes, Yuki M. Asano, Cees G. M. Snoek
ECCV 2026
cs.CV, cs.AI, cs.LG
2026-08-10
Fourier Self-Supervision adds low- and high-pass views to contrastive GCD; FourEx reaches 82.7% novel-class accuracy on CUB with DINOv1.
Generalized Category Discovery (GCD) asks a model to keep recognizing labeled known classes while clustering novel classes that sit in the same unlabeled pool. The default recipe is contrastive learning: two augmentations of one image as a positive pair. That works when classes look obviously different. On fine-grained sets such as bird species or aircraft variants, the telltale cues live in a beak, a feather pattern, or an air intake, and random crops plus color jitter wipe those cues out.
Most GCD methods also pull and push in a single flat embedding. Fine-grained labels usually sit on an implicit hierarchy: forest bird versus seabird first, species later. Low Fourier frequencies behave like the parent nodes; high frequencies carry the species-level residuals. This ECCV 2026 paper from the University of Amsterdam and the University of Technology Nuremberg turns that frequency split into a plug-in self-supervision head.
Fourier Self-Supervision runs an FFT on each image, applies a low-pass and a high-pass filter in parallel, then inverts back to pixels. The low-pass view is a blur of color masses. The high-pass view is edges and texture. Both are treated as extra positive views of the original image.
Cutoffs are data-dependent. A sweep on the training set picks the frequency where signal-to-noise ratio (SNR) sits near 15 dB, a video-quality convention reused here as "still category-readable." Training samples a random cutoff below that cap. A 5×5 Gaussian blur is applied before filtering to damp Gibbs ringing.
The latent vector is split:
A classification loss then requires the original, the low-pass reconstruction, and the high-pass reconstruction to predict the same known-class label. The total loss is the host GCD loss plus these three terms, all at weight 0.1 by default. Dropped onto SimGCD it is FourSim; onto SelEx it is FourEx. Backbone is ViT-B/16 with DINOv1 or DINOv2; FourEx fine-tunes the last three blocks.
With DINOv1, FourEx lifts SelEx when both fine-tune the last three blocks:
| Method | CUB All / Known / Novel | Aircraft | Cars | Mean All |
| SelEx† | 76.4 / 72.4 / 78.4 | 61.2 / 67.7 / 58.0 | 56.9 / 76.9 / 47.3 | 64.8 |
| FourEx | 80.2 / 75.1 / 82.7 | 65.9 / 67.9 / 64.9 | 59.2 / 76.9 / 50.6 | 68.4 |
Mean gains against the respective hosts: All +4.3, Known +4.0, Novel +4.4. FourSim moves SimGCD on CUB from 60.3 to 70.3. When the class count is estimated rather than given (Vaze's estimator, 231 classes on CUB), FourEx still reaches 77.3 All versus SelEx at 72.0.
DINOv2 shrinks the gap: CUB All 87.8 versus 87.4, Aircraft 81.5 versus 79.8, Stanford Cars 80.1 versus 82.2 (a drop). The paper treats Cars as coarser, so high-frequency views help less. On long-tailed Herbarium19, FourEx All is 44.5 against SelEx 39.6; Oxford-Pet is a wash (92.8 vs 92.5). All three losses contribute; setting any coefficient to 1.0 hurts. Among SNR caps from 5 to 25 dB, 15 dB is best. On coarse CIFAR-10/100 and ImageNet-100, FourEx mean All is 87.7, in the same band as SPTNet at 88.0. Wall time: 90.69 s per epoch for FourEx versus 84.29 s for SelEx, same parameter count and FLOPs.
This is a drop-in frequency view, not a new clustering head. For open-set fine-grained work, specimen sorting, or defect taxonomies where novel classes look like known siblings, splitting hierarchy into low frequencies and discrimination into high frequencies is a cleaner bet than yet another pseudo-label trick. The DINOv2 shrinkage is the caveat: once the backbone already sees texture, extra Fourier views add little, and on coarser sets they can subtract.
Failures cluster on occlusion, specular highlights, and color mix-ups. High-pass strips chroma, so hue becomes a weak spot. The DINOv2 plus Stanford Cars regression shows the fine-grained premise is not universal. Class-count estimation still uses Vaze's older method (231 / 230 versus true 200) with no sensitivity study. The 15 dB cutoff is borrowed from subjective video quality, which is not the same as class separability. The claim that the left latent half encodes a parent class is motivated, not probed with a direct hierarchy readout.