Paper proves translation symmetry in language statistics shapes model representation geometry
burny_tech · x · 2026-10-27
An arXiv paper (Karkada, Korchinski, Nava, Wyart, Bahri) explains why LLM representations form geometric structures—months as circles, years as 1D manifolds, decodable city coordinates:
- Language co-occurrence statistics exhibit translation symmetry (e.g., two months' co-occurrence frequency depends only on their interval).
- The authors prove this symmetry governs the geometry in high-dimensional word embeddings and analytically derive manifold shapes; predictions match large embedding models and Gemma 2 2B internal activations.
- Representational geometry is robust to perturbations (e.g., deleting all sentences where two months co-occur), emerging naturally when statistics are controlled by a latent variable.
- Conclusion: representational manifolds originate in statistical symmetries of natural language.
Related event: Paper: Language Symmetry Shapes Representation Geometry(2 posts)→
More from Research
- UC Berkeley paper: RL reweights LLM reasoning strategies rather than expanding them — burny_tech · 2026-10-27
- Dehaene lab finds a cortical map integrating integers and fractions via 7T fMRI — burny_tech · 2026-10-27
- LLM rewrites a physics paper into a clearer 12-page explainer — correctness unverified — thomasahle · 2026-10-27
- OpenAI's new math results accused of scooping community mathematician's Yang-Mills work — burny_tech · 2026-10-27
- NYC Reliable AI Meetup on Synthetic Data Set for Nov 16 with NYU's Gautam Kamath — karen_ullrich · 2026-10-27
- Meta's InteractiveWorld Benchmark Shows MLLMs Still Far from Reliable Physical Assistants — meta · 2026-10-27