2026-08-12
LLM caption embeddings that predict brain responses show body-selective regions EBA and FBA encode implied motion speed more strongly than headcount or body size.
The extrastriate body area (EBA) is a patch of visual cortex that responds more strongly to human body parts than to faces or objects. Classic experiments established this with simplified stimuli: isolated body-part photos, point-light biological motion. Recent work using large fMRI datasets and language-model-based encoding models found something odd: EBA's response to natural scenes isn't just about whether a body is present. It shifts with the semantic context around that body. The problem is that caption embeddings from language models are a black box. Nobody could say what specific aspect of a scene EBA was actually tracking.
The paper's starting intuition: which objects co-occur in a scene determines how that scene reads. A person at a table with a fork and a plate reads as "about to eat." Swap the fork and plate for a laptop and the same person now reads as "working." The authors turned this into a three-step analysis.
First, they trained encoding models on fMRI data from eight subjects in the Natural Scenes Dataset (NSD), paired with human-written MS-COCO captions for each image. Captions were converted to 768-dimensional embeddings using the language model all-mpnet, then fit with ridge regression to predict EBA responses. Prediction accuracy (Pearson correlation) ranged from 0.44 to 0.72 across the eight subjects.
Second, the trained models predicted EBA responses to all 73,000 NSD captions, which were sorted by predicted response strength and split into 73 groups of 1,000. For each group, the authors built a co-occurrence matrix counting how often pairs of 12 COCO object categories appeared together in captions. Using captions rather than raw image content was deliberate: images often contain small objects unrelated to a scene's overall meaning, while a human-written caption already filters for what matters.
Third, non-negative matrix factorization (NMF) decomposed the 73 co-occurrence matrices into a small number of independent patterns, each linked to a level of predicted EBA response. This step surfaced the specific object pairings driving EBA activity, and a follow-up behavioral experiment tested the hypothesis it suggested.
NMF found three co-occurrence components, tied to high, moderate, and low EBA response. In the high-response component, "person" co-occurred mostly with "sports" objects. In the moderate-response component, it co-occurred with "accessory" and "vehicle." The low-response component barely included "person" at all. Inspecting the images by hand, high-response scenes mostly showed people mid-action, swinging a racket or doing acrobatics, while moderate-response scenes showed people standing still. The pattern suggested EBA cares less about whether a body is present than about how fast that body appears to be moving.
To test this, five participants rated 100 NSD images (containing only people, to isolate human motion from animal or vehicle motion) for implied motion speed on a 1-5 scale. Ratings were highly reliable, both within participants across repeats (Spearman correlation 0.85-0.96) and between participants (0.80-0.90).
Linear mixed-effects regression against fMRI data showed EBA response correlated significantly with implied motion ratings for person images (beta=0.040, p=0.001). The correlation held after excluding EBA vertices that overlap with the motion-processing regions MT and MST (beta=0.033, p=0.022), ruling out the possibility that the effect was just motion-area signal bleeding into EBA. The fusiform body area (FBA), anatomically distant from EBA, showed a similar pattern. Vehicle images, by contrast, showed no significant relationship (beta=-0.006, p=0.426).
| Analysis | Metric | Result |
| Encoding model accuracy (8 subjects) | Pearson r | 0.44–0.72 |
| EBA vs. person implied motion (all vertices) | beta (SE), p | 0.040 (0.012), p=0.001 |
| EBA vs. person implied motion (MT/MST excluded) | beta (SE), p | 0.033 (0.014), p=0.022 |
| EBA vs. vehicle implied motion | beta (SE), p | -0.006 (0.008), p=0.426 |
The authors also tested body size, number of people, distance from image center, and low-level contrast as competing explanations, using variance partitioning to isolate what each feature uniquely explains. Implied motion was the single best predictor for EBA in 7 of 8 subjects and for FBA in 5 of 8; number of people came second for EBA, body size came second for FBA. Distance from center and contrast explained almost nothing.
This gives a repeatable path from "caption embeddings predict brain responses" to "here's specifically what the brain is tracking." Language-model-based encoding models have proven they can predict fMRI responses well, but rarely explain what drives that predictive power. Combining a co-occurrence matrix with NMF turns the black box into a readable set of object pairings, and the follow-up behavioral experiment closes the loop by testing the resulting hypothesis independently. The method itself, co-occurrence analysis plus variance partitioning, should transfer to other category-selective brain regions beyond body perception.
The authors acknowledge that their three-feature model, implied motion, headcount, and body size, doesn't explain every significant vertex. Emotional and social scene content, which prior work has linked to EBA and FBA responses, was explicitly excluded, partly because NSD images skew toward emotionally neutral, non-social scenes, a limitation of the dataset rather than the method. The behavioral sample was small: only five participants rated implied motion, and between-participant reliability for animal images (0.40-0.71) was noticeably lower than for person and vehicle images, so the animal-motion finding carries less confidence than the main result. The analysis also stayed at the single-vertex level and didn't look at distributed response patterns across vertices, which the authors flag as future work.