Why ViTs beat CNNs: 113k-citation paper shows scale trumps inductive bias
moschles · reddit · 2026-11-19
A Reddit post by moschles recaps why Vision Transformers took over, anchored on Google Research's classic paper An Image is Worth 16x16 Words (arXiv:2010.11929, 113k+ citations).
- Transformers lack CNN inductive biases (translation equivariance, locality), so trained directly on mid-sized datasets like ImageNet they underperform comparable ResNets.
- Pretrained on 14M–3000M images, the picture flips: large-scale pretraining trumps inductive bias. ViT pretrained on ImageNet-21k or JFT-300M hits 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the 19-task VTAB suite.
- The post links in He et al.'s MAE paper (arXiv:2111.06377): splitting images into patch tokens for masked self-supervised prediction is what makes ViT scalable.
- Bottom line: the token-izing approach is worse than CNNs on small data but wins decisively at scale — and the current CIFAR-10 SOTA is itself a pretrained ViT.
More from Research
- Open models transcribe 8h38m of mock GP consultations in 18m38s on a single GPU — 27.8x faster — MaziyarPanahi · 2026-11-19
- Pink noise in conversation semantic similarity predicts enjoyment, finds Science Advances study — MacrinePhD · 2026-11-19
- ZK Podcast: ARIA's Nicola Greco on generative cryptography and AI scaling trust — DavideCrapis · 2026-11-19
- StatePortal synthesizes agent checkpoints for GUI deep-state testing, 0%→27.5% completion — SJUT1 · 2026-11-19
- SADL plug-in loss claims to cut JEPA training collapse from 50% to 0% — TheTuringPost · 2026-11-19
- PNAS Study: LLMs Amplify Gender Bias Both Writing CVs and Screening Them — AnnaCiaunica · 2026-11-19