Berkeley and DeepMind paper: LLM representational geometry comes from symmetry in language statistics

udmrzn · x · 2026-11-20

UC Berkeley and Google DeepMind researchers show that the striking geometries inside LLM representations — circular months, rippled timelines of years, linearly decodable city maps — don't emerge from transformers inventing world models. Even word2vec produces the same shapes. The geometry derives from translation symmetry in pairwise word co-occurrence statistics, which the authors formalize to analytically predict representational manifolds that match empirical results and remain robust to perturbations.

Original post →

More from Research

Research channel →