8-Year Study Finds LLMs Have Implicit Symbolic Structure, Multiplicative Codes Enable Generalization
jm_alexia · x · 2026-09-06
A new paper from RTomMcCoy's group, the result of an 8-year project, shows that LLM representations harbor implicit symbolic structure—explaining why neural models excel in symbolic domains like language, code, and math.
- Key finding: models that generalize over combinatorial binding (10^10 combinations) exhibit multiplicative structure in their representations;
- Models that fail to generalize instead "memorize" objects as linear directions, which blocks generalization;
- The work also contrasts next-token-prediction and embedding paradigms, arguing LLMs are not fundamentally opposed to symbolic systems but implicitly realize one.
More from Research
- New Paper Maps When Understanding and Generation Actually Synergize in Native Unified Multimodal Models — liuziwei7 · 2026-09-06
- Google engineers argue alignment requires cooperative multi-agent learning, not solipsistic AI — sebkrier · 2026-09-06
- Langford argues gradient descent pushes transformers to discard deep-layer information — JohnCLangford · 2026-09-06
- Graph memory loses to flat vector baseline on LongMemEval: F1 0.42 vs 0.47 — CShorten30 · 2026-09-06
- 2.6B LFM model trained in Liquid AI's MMAIGym sets new SOTA in chemical synthesis — JosephJacks_ · 2026-09-06
- New preprint gives a unified theory of H-duality, explaining why reversing stepsizes yields new optimizers — prof_grimmer · 2026-09-06