2026-08-13
Aligning human fMRI and macaque recordings to 8,640 images reveals a rich shared high-dimensional object space in IT cortex; deeper DNN layers encode human IT, shallower layers macaque IT.
Inferotemporal cortex (IT) is the primate brain region that handles object recognition, and both humans and macaques have one. How similar are the two species' neural representations of the same objects? Prior work established a few coarse correspondences, such as animacy and shape, but could not tell whether the similarity is confined to a handful of axes or spans a broad high-dimensional geometry. That gap matters twice over: it decides whether invasive macaque recordings can stand in for human vision, and it decides how literally to read claims that deep networks "explain" visual cortex, and which species' cortex they mirror.
The same 8,640 natural object images were shown to macaques (intracranial multi-unit recordings) and to humans (fMRI). The authors used multi-set canonical correlation analysis (MCCA), a multivariate method, to align the two datasets into a shared "object space."
What ties this paper to AI is that they treat seven DNNs, differing in objective, architecture, and training set, as encoding models. They take layer-wise activations from each network and use cross-validated ridge regression to predict each species' brain space. That becomes a concrete question: which DNN layer depth lines up with which species' IT. They also build a "control embedding" from the early layers of those same DNNs and regress out low-level visual features, to check whether an effect survives once pixel-level features are removed.
| Metric | Human IT | Macaque IT |
| Conceptual R² | 0.071 | 0.042 |
| Visual R² | 0.022 | 0.032 |
| Best-layer DNN R² | 0.196 to 0.236 | 0.177 to 0.218 |
| Aligned DNN depth | Deep (60 to 100%) | Shallow (0 to 20%) |
The shared space is far richer than expected. It holds many interpretable visual and conceptual dimensions and mixtures of the two, not just the usual animacy and shape axes.
Human IT is better predicted by high-level conceptual information (conceptual R²=0.071, well above visual 0.022); macaque IT is more balanced (0.042 vs 0.032). DNN layer depth mirrors that split. All seven networks perform similarly at their optimal layer (human R² 0.196 to 0.236, macaque 0.177 to 0.218), but sliced by depth, the shallowest bin (0 to 20%) predicts macaque better (p=0.012) while the deepest bins (60 to 100%) predict human better (p not above 0.002). Human IT aligns with deep DNN layers, macaque IT with shallow ones.
A surprise turned up in the shared space: symbols, icons, and multi-object ensembles are decodable in both species. After regressing out shallow DNN features, decoding dropped (0.08 to 0.20 AUC) but stayed reliably above chance. Macaque IT appears to carry a precursor to orthographic sensitivity, a response to the visual form of letters and symbols, even though macaques do not read. This decoding strengthens from V1 to IT in both species. Each species also over-represents its own kind: human body parts load on the human space, monkey images on the macaque space.
For anyone in representation learning or vision modeling, this is a species-aware calibration of "which DNN layer resembles which part of the brain." The old shorthand "DNNs resemble IT" was too coarse. Different network depths now map to different species and abstraction levels: deep layers resemble the conceptual, human IT; shallow layers resemble the more visual macaque IT. It adds a variable the field routinely ignores, species, to the question of whether DNNs are good brain models. For researchers extrapolating from macaque data to humans, the warning is concrete: the two differ systematically on conceptual abstraction, and one cannot simply port findings across.
The authors' own biggest caveat is modality confounding. Human data is fMRI, macaque data is intracranial multi-unit, with different anatomical coverage and task parameters, so species effects and methodological effects cannot be fully separated. Within-species contrasts deserve extra caution.
A second issue is broader and easier to miss: the conceptual predictors and the DNN embeddings themselves may carry human bias. Later DNN layers, trained on human-curated datasets, naturally favor the human space, which self-reinforces the "deep layers resemble human IT" conclusion. Removing it cleanly would require stimuli that better reflect macaque visual experience, or differences estimated from macaque behavior.