Evaluation Resolution Confounds Learning-Rule Comparisons in Model-Brain RSA of Early Visual Cortex
Nils Leutenegger
q-bio.NC, cs.LG
2026-08-11
CNNs trained at 32px reverse V1 RSA ranking: the untrained-minus-backprop gap grows from ~0 at 32px to +0.044 at 224px, from image detail above training resolution.
A standard NeuroAI comparison asks which learning rule produces the most brain-like visual representations: backpropagation, feedback alignment, predictive coding, or STDP. The usual test is RSA. Layer activations over a stimulus set become a representational dissimilarity matrix, then Spearman-correlated with the matching matrix from visual cortex.
Biologically plausible rules still do not scale, so any study that includes them trains a small CNN on 32×32 CIFAR and scores it against brain responses to naturalistic images, typically rendered at 224px. A recurring qualitative result in that setting is that a randomly initialized network already matches or beats backpropagation at V1. The author had reproduced that result. This paper shows the ranking is an evaluation-resolution effect.
One small CNN (three conv blocks with 3×3 kernels, batch norm, ReLU, 2×2 max-pool, 32/64/128 channels, then FC-512 and a 10-way head) is trained on an 8,000-image CIFAR-10 subset at 32×32 for 40 epochs, five seeds. Conditions: untrained, backprop, feedback alignment, predictive coding, STDP. Human data are THINGS-fMRI RDMs over 720 object images from three subjects (V1, V2, LOC, IT). Macaque data are single-seed Brain-Score sets: FreemanZiemba2013 (V1/V2) and MajajHong2015 (V4/IT). The main mapping is Conv1→V1.
The manipulation freezes weights and normalization and resizes the brain stimuli to 32, 64, 96, 128, 160, and 224px. A second arm first downsamples every image to 32px and then upsamples, so content is capped while the number of pooled positions still grows. ImageNet-pretrained ResNet-50 and Swin-Tiny, both trained at 224px, test architecture and train/eval matching. A 2×2 batch-norm calibration holds convolutional weights bit-identical and only changes stored statistics, to test whether the untrained baseline is simply an unnormalized network.
The V1 ranking moves with evaluation resolution. Every trained condition peaks near the 32px training resolution and falls as resolution rises:
| condition | 32px ρ | 224px ρ |
| untrained | 0.065 | 0.076 |
| backprop | 0.065 | 0.031 |
| feedback alignment | 0.020 | 0.012 |
| predictive coding | 0.026 | 0.016 |
| STDP | 0.059 | 0.037 |
The untrained-minus-backprop V1 gap grows from −0.001±0.007 at 32px (3/5 seeds positive, not significant) to +0.044±0.006 at 224px (5/5). Selecting the best V1 layer per condition and resolution leaves the growth intact, +0.014 to +0.060. The claim that the gap is 0 at training resolution is specific to the fixed Conv1 mapping.
Four candidate mechanisms fail. Calibrating untrained batch-norm at the evaluation resolution, with convolutional weights bit-identical, still leaves a +0.041 (CIFAR) or +0.033 (THINGS) advantage at 224px. ResNet-50 (0.045→0.033) and Swin-Tiny (0.080→0.052), both trained at 224px, also peak at low resolution, so this is not a penalty for leaving the training resolution. A Gabor filterbank itself only reaches ρ≈0.018–0.037 against V1; ResNet-50's early stage is about ten times more Gabor-like than the untrained CNN and still aligns worse. A single luminance scalar hits ρ=0.075 against the V1 RDM, matching the untrained network's 0.076, and partialling luminance halves that alignment (0.076→0.038). Luminance similarity even ranks conditions the way their resolution slopes do. It does not carry the effect: CIFAR calibration at evaluation resolution lowers luminance similarity while raising V1 alignment.
A fifth experiment pins the dependence on content. With detail capped at 32px and pooled positions growing twelve-fold (1,024→12,544), the gap opens by only +0.003±0.001 versus +0.030±0.002 when content can change. Backprop's decline disappears (−0.023→−0.000). About 90% of the resolution dependence comes from image detail above the training resolution, not from how many positions are averaged.
The training trajectory is the same knob. At 224px, backprop V1 alignment falls −0.031 from epoch 0 to 40, the familiar "training hurts early visual alignment." The same run at 32px ends at −0.000. At LOC, backprop stays above untrained at every resolution (+0.019 at 32px, +0.018 at 224px, 5/5 seeds). That learning effect survives the sweep.
A separate implementation bug: predictive coding and STDP had overridden eval() as a no-op, so batch-norm stayed in training mode during feature extraction. After the fix, STDP's flat resolution curve vanishes, and predictive coding degrades V1 more than backprop, reversing a claim in the author's training-dynamics preprint. Random and backprop conditions are unaffected.
If you compare learning rules or architectures at early visual cortex, evaluation resolution has to be a reported variable, at the training resolution and at the usual scoring resolution at minimum. Biologically plausible rules are the most exposed, because they are why training stays small. "Untrained or local rules match backprop at V1" is, in this setting, partly an analysis choice. Backprop still leads at LOC, so learning does reshape representations. Early visual cortex is where the knob swamps the signal.
There is a sharper measurement warning. Under global average pooling plus Spearman RSA, a one-dimensional brightness value matches the best model at V1 in this dataset. That bounds what this comparison can resolve, not model–brain alignment in general. A fitted readout on the full feature map could place the models higher. That check is not in the paper.
The macaque analysis is single-seed, so cross-species support is directional. Human fMRI has three subjects, one of them weak, and the core statistics rest on two. Predictive coding moves Conv1 by about 2% of the initialization norm and never updates Conv3, so "across conditions" is thinner than it sounds. Feedback alignment and STDP use a different initialization from the other three, and the single untrained network is a matched control for only three of five. Joint partial RSA against four correlated low-level references drives trained conditions strongly negative (over-subtraction). Every number uses rank correlation on globally pooled features, which discards spatial structure. The mechanism is still open: why detail above 32px helps random filters and hurts trained ones is not explained.