Word Vectors Without Transformers Recover Space, Time, Pain and Emotion, Challenging LLM World Model Evidence
gerardsans · x · 2026-10-08
AI researcher Gerard Sans comments on ebarenholtz's V2 preprint World Properties without World Models.
Key findings:
- Plain word vectors (no transformer, no context) recover much of what decoding studies cite as evidence of internal world models in LLMs: space, time, pain, emotion
- This challenges the growing LLM "decoding" literature — if activations can be linearly decoded to yield a variable, researchers typically infer the model represents that variable
- The paper invokes J.R. Firth's 1957 dictum "knowing a word by the company it keeps," suggesting decodability may stem purely from word co-occurrence distributions
Gerard Sans adds: corpora carry patterns, observers project meaning, and meaning lives in lived experience rather than in the corpus. AI is a text sampler whose outputs reflect structures embedded in training data, not inherent understanding.
Related event: Word Vectors Without Transformers Question LLM World Models(2 posts)→
More from AGI Musings
- Meta researcher Dimitris Papail: this is the best time ever to be a researcher — DimitrisPapail · 2026-10-08
- GPT's math results show credit diffusing across thousands: the age of heroes is ending — sandersted · 2026-10-08
- Anthropic Study: AI and Robots Can Already Cover 81% of U.S. Employment Tasks — Polymarket · 2026-10-08
- GPT's Math Wins Credit Thousands of Collaborators, Not One Mind — sandersted · 2026-10-08
- Reddit proposal: keep AGI from learning to code and use humans as the safety bottleneck — meira_xxx · 2026-10-08
- repligate mocks Paul Christiano's post-AGI vision: neither possible nor good — repligate · 2026-10-08