Doomed to Re-Annotate, Forever: The ImageNet Story
Illia Volkov, Nikita Kisel, Tetiana Mishkina, Klara Janouskova, Jiri Matas
cs.CV
2026-08-14
CTU Prague reannotated all 50k ImageNet-1k val images: ~12% original labels wrong, 33.3% multilabel. MLLMs gain 5–6 points; the strongest supervised model barely moves.
Top-1 accuracy on the ImageNet-1k validation set is still the number vision papers quote most often. The 2012 labels have been dirty for years: missing objects, wrong classes, overlapping synonyms, a train/val distribution shift. Earlier reannotation passes (ImageNet ReaL, Shankar et al., Tsipras et al.) improved coverage but kept the original WordNet taxonomy, so the corrected labels disagree with each other.
Label noise also shuffles model rankings, which then leak into which checkpoint gets used downstream. The Visual Recognition Group at Czech Technical University in Prague reannotated all 50,000 validation images from scratch. The result, ReImageNet, adds multilabel boxes, class definitions rewritten from the actual images, and five attributes: rendition, reflection, text-recognition, crowd, and dominant.
The task is no longer "pick one class for the image." Annotators mark every ImageNet-1k object a person notices on first viewing, and draw a box around it. An earlier protocol tried to label every visible object. Tiny blobs a few dozen pixels across, identifiable only from context after zooming in, made the label set depend on how hard each annotator chose to look. The rule became: at least whatever a person notices within five seconds.
Seven in-house, non-expert annotators (ages 16–50, backgrounds in Czechia, Ukraine, Greece, and Latvia) worked in 97 WordNet-parent groups, median seven classes per group, so nobody had to keep 1,000 names in working memory. Each group started with a familiarisation pass: look at the images, rewrite class names and definitions where the original WordNet gloss did not match the photos, then label image by image. OWLv2 top-5 boxes and OpenCLIP top-20 predictions sat on screen as optional hints. Fine-grained species went through iNaturalist.
Guidelines kept changing mid-stream. Is a toy gun a toy or a gun? Does a face in a mirror count as a face? Those questions produced rendition, reflection, and text-recognition. Crowd and dominant exist to keep dense scenes tractable. A second verification pass, still running, swaps in stronger MLLM proposals with SAM3 boxes. The authors expect only small edits from here.
A prior case study is the quality ceiling in one number. On hard images where GPT-4o disagreed with the humans, annotators accepted the model or admitted they were wrong about half the time, and were right the other half. Crowd one-shots with frozen guidelines do not work at this scale. Trained humans plus MLLMs plus tooling is the current ceiling.
About 12% of the original ImageNet-1k labels are wrong: 5.5% single-label disagreements, 2.8% multilabel disagreements, 3.8% images with no ImageNet-1k object at all. 33.3% of images are multilabel. Prior reannotations topped out at 21.6% multilabel (Tsipras et al.); ReaL sat at 14.9%.
Evaluation uses ReaL accuracy: a prediction counts if it lands in the legal set, with near-synonyms merged. On the 48,114 images that have at least one valid class:
| Model | Original labels | ReImageNet | Delta |
| GPT-5.4 | 72.54 | 77.64 | +5.10 |
| Qwen3-VL | 70.95 | 77.04 | +6.09 |
| Gemma 4 | 69.48 | 74.87 | +5.39 |
| SigLIP2-g | 86.74 | 88.99 | +2.25 |
| DINOv3 | 86.49 | 87.69 | +1.20 |
| EVA-02 | 91.33 | 91.38 | +0.05 |
Models that depend less on ImageNet supervision move more when the labels change. EVA-02, the strongest supervised model in the pool, barely moves. On the hard slices where ImGT and ReGT disagree, every model sits in the 29–46% range.
Attributes split the families. Rendition (toys, drawings, stylized depictions) hurts everyone; supervised models drop hardest (EffNetV2 −9.8 points, DINOv3 −7.3), MLLMs only 1.3–2.3. On text-recognition images, where the class is readable from letters in the photo, Qwen3-VL gains 15.6 points and EffNetV2 loses 6.7. Language-aligned models can use visible text as a cue; purely supervised vision models cannot.
Tight crops around each box produce 96,051 samples at 33% of original area on average. Supervised models collapse when every crop is scored independently: EVA-02 −33.5 points, DINOv3 −32.3, Gemma 4 only −15.7. Expanding each box until it touches a neighbor, then dropping overlaps, yields 38,628 ExCrops (82% of original area, exactly one annotated object) and recovers about 10 points. ImageNet-supervised models were trained on full scenes; they are not good at isolated objects. In about 10% of images the model misses the largest object and still gets a smaller, less salient one right. Center-crop evaluation looks worse after that.
Errors travel. Using the subset where GPT-5.4 and SigLIP2-g agree, estimated error rates on derivative sets run 1.7–5.8× the ImageNet-1k rate. ImageNet-A is the high end; ImageNet-Sketch is estimated to hold 2,861 mislabels. The failure mode is structural.
ImageNet top-1 is a looser number than the leaderboard implies. For MLLMs it is especially loose: 5–6 points of "error" were labels that had not caught up with objects the model already recognized. Supervised models gain little because they were trained on the same noisy labels.
The release is usable: annotations, class definitions, guidelines, analysis code, and a preview tool. Detection, counting, and attribute slices need the boxes. Strict single-label comparisons should use ExCrops, not the permissive ReaL numbers, which the paper says are not comparable to a decade of top-1 scores.
The methodological claim is the one that travels. Frozen-guideline crowdsourcing plus majority vote loses to one trained annotator on fine-grained semantics. If new MLLM benchmarks are built the same way, or labeled by the models they are supposed to test, the next generation of scores will inherit the same noise.
The authors list the biases. All annotators are European; classes such as abaya were learned from the web. Fine-grained wildlife was labeled by non-experts, a task domain experts already get wrong. A shared definition table keeps labels consistent, and a wrong definition poisons every image of that class. Anonymized model hints can still nudge decisions. Only the validation set is covered; class names were rewritten from val images and may not match the training set, which already sits on a different distribution. The photos are from 2012.
Verification is unfinished. The second pass is where MLLM plus SAM3 arrived. The authors expect only small changes, but 12% and 33.3% are not final until that pass closes.
Two estimates sit on thin ice. The derivative-set error bound assumes the two models behave the same on ImageNet and on ImageNet-A / Sketch / ObjectNet; that may be false, and defining label quality by model agreement writes current inductive biases into the ground truth. ReaL scoring is also permissive: any legal label counts, so ReGT accuracy is structurally higher than single-label top-1. ExCrops patch that. The +5–6 points in the main table should not be read as models getting stronger.
Reannotation does not touch ImageNet's known problematic categories or image provenance.