Parts-of-Speech as Emergent Categories in SAE Latent Space
Alessandro Bondielli, Lucia Passaro, Serena Auriemma, Alessandro Lenci
cs.CL
2026-09-24
Llama-3-8B SAE activations linearly recover PoS, but as groups of latents, not one per tag; 498 features nearly match the full-space classifier.
Sparse autoencoders (SAEs) unpack a language model's dense activations into a high-dimensional sparse code. A common working picture is that each latent lines up with one interpretable concept. Part-of-speech (PoS) tags are a clean test of that picture: they are discrete, independently annotated, and they split into open classes (nouns, verbs) that keep adding words and closed classes (determiners, conjunctions) that barely do. If SAE latents behaved like atomic grammatical features, each PoS tag should collapse onto a single latent.
CoLingLab at the University of Pisa runs that test on Llama-3-8B, taking residual-stream activations at layer 30 and encoding them with EleutherAI's 32x SAE (131,072 dimensions) on the English GUM treebank. GUM has 14,353 sentences, 252,284 tokens, and all 17 Universal Dependencies tags. The question is not whether PoS information is present. It is how that information is arranged once the representation is sparse.
Three stages.
Controls include a random-label probe that keeps word identity but shuffles PoS, probes on layer-0 embeddings and raw layer-30 residuals, and a swap test that replaces a growing fraction of the compact set with random latents.
PoS is linearly readable from SAE activations. The random-label control scores 0.54 accuracy and 0.42 macro-F1; the real-label probe is reported as 0.88 / 0.97, a 34 to 37 point gap, so this is not just lexical memorization.
The SAE probe itself scores 0.88 accuracy and 0.78 macro-F1, matching layer-0 embeddings and trailing raw layer-30 residuals at 0.92 / 0.84. The authors say the SAE probe uses about 8 times fewer features; 498 versus the 4,096-d residual is roughly that ratio. The SAE's contribution is decomposition and localization, not a higher probe score.
| Probe | Accuracy | Macro-F1 |
| SAE | 0.88 | 0.78 |
| Layer-30 residual | 0.92 | 0.84 |
| Layer-0 embedding | 0.88 | 0.78 |
The organization is not one latent per tag. Open-class one-vs-rest classifiers keep 7,927 non-zero coefficients on average, closed-class 3,282, other 2,094. Nouns use 10,043 latents; coordinating conjunctions use 645. The 95% coverage groups union to 498 latents, 12% of them shared by two or more tags. A multi-class classifier on those 498 scores 0.87 accuracy and 0.76 macro-F1 in cross-validation; at 0% overlap with the selected set the same protocol falls to 0.24 / 0.16.
On the held-out treebank, within-group recall is usually at least 0.95, but mean Distinctiveness is only 0.27 (chance 0.06, exclusive 1.0). On controlled templates, pseudo-multilabel precision is 0.192, recall 0.937, F1 0.318: the groups fire, they do not stay exclusive. A cluster of "first-token" latents also contaminates the templates.
Closed classes are tighter and more stable. Open classes are broader and pick up form, position, and annotation habits. Adjectives co-activate with nouns; verbs collide with nouns on homographs such as drink. That looks like distributional neighborhood, not a pure grammatical atom.
For people who treat SAE latents as concept atoms, this is a negative result on a standard linguistic inventory. PoS shows up as compact, category-dependent groups whose size tracks lexical variability: determiners in the controlled set collapse toward a single activation, because the determiner is almost always the.
498 latents out of 130,246 attested ones is a sharp cut. Morpho-syntax can be localized. Probe coefficients are still not causal. Using these groups as circuit handles needs intervention experiments that this paper does not run.
The authors list the obvious cuts: PoS only; one base model, one public SAE, one layer; linear probes rather than causal edits; a small controlled set built around nouns and verbs.
Two further issues. The control paragraph quotes 0.97 macro-F1 for the real probe, while Table 2 gives 0.78 for the SAE probe; the two figures are not aligned in the text. The compactness gradient is also walked back in the linguistic analysis: it tracks type-inventory size more than graded abstraction. English adpositions and subordinators share forms (by, after, since), so some cross-tag firing is an annotation artifact this study cannot separate.