Invisible Shortcuts: Why Vision Encoders Know Your Camera
Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos, Noa Garcia, Giorgos Tolias
ECCV 2026
cs.CV, cs.LG
2026-08-06
Vision encoders use invisible metadata (JPEG quality, focal length) as prediction shortcuts; stronger correlation means more sensitivity and worse shifts, but mitigation improves OOD robustness and AI-image detection.
Vision models take shortcuts, latching onto simple cues correlated with labels but not necessarily causal, like object-background ties or texture bias. Prior work mostly studied visible biases. This paper points to a new source: invisible metadata traces hidden in pixels, in two families. One is image-processing parameters (JPEG compression, sharpening, resizing, interpolation), the other is acquisition parameters (camera model, exposure, aperture, ISO, focal length). Earlier work (Ramos et al.) showed that shifting image metadata shifts the predictions of pretrained vision models, but the cause was unclear. The hypothesis here is that this sensitivity comes from metadata-semantics correlations in pretraining data (football photos tend to have long focal lengths), which models learn as shortcuts.
The paper defines three diagnostics: MP (metadata prediction, whether a linear probe can read metadata from features), SPD (semantic prediction distraction, whether metadata interferes with semantic retrieval), and SP (semantic prediction, whether classification still works). It then tests the hypothesis two ways.
First, by measuring existing correlations. The ImageNet JPEG-semantics correlation by Cramér's V is 0.047 for IN1k and 0.067 for IN21k, and models trained on IN21k encode more metadata and are more distracted. On about 40 million Exif-tagged images from Re-LAION-2B, distinct topics like football versus plots of land have clearly different focal-length distributions.
Second, by controlled injection. Assign each semantic class a JPEG compression with probability pi, so Cramér's V roughly equals pi, then train ResNet50; sensitivity rises monotonically with pi. Acquisition metadata works the same way: LAION subsets with correlations 0.396, 0.255, 0.166 yield models that are most sensitive at the strongest setting.
Mitigation runs two ways. During training, the DINOv2 augmentation cocktail (color jitter plus grayscale plus blur) cuts sensitivity sharply. After training, an adversarial linear layer strips metadata using only JPEG labels, yet it also reduces sensitivity to unseen metadata types.
The key numbers: IN1k and IN21k correlations are 0.047 versus 0.067, and the latter encodes more metadata and is more distracted. In controlled runs sensitivity rises monotonically with correlation strength; on a test set transformed with the same metadata, accuracy rises instead (the model reuses the pixel signal it learned), while on a clean set it does the opposite.
A positive side effect: models more sensitive to metadata are also better at detecting AI-generated images, because both need a grasp of pixel-level detail. A ResNet50 trained under strong JPEG correlation lifts generated-image detection from an overall mean of 57.3 to 60.4, and on ProGAN from 69.1 to 77.2. Stable Diffusion images still carry traces (MPp-Resizing 61.58). Mitigating metadata sensitivity also improves out-of-distribution generalization: on original IN1k, ResNet50 rises on ImageNet-C from 44.2 to 45.0, on ImageNet-R from 34.5 to 37.7, and on ImageNet-Sketch from 26.4 to 30.0.
This explains a known puzzle: why do frozen CLIP features detect AI-generated images so well? Because they encode pixel-level metadata. It also tells practitioners that if a vision encoder degrades under distribution shift, metadata shortcuts may be the cause, and you can mitigate them with a cheap linear layer added after training (no retraining) or the right augmentations, gaining OOD robustness as a bonus. It also overturns the earlier claim that mitigating one bias amplifies others: JPEG-targeted mitigation also reduces sensitivity to resizing, camera, and other metadata.