8-Year Study Finds LLMs Have Implicit Symbolic Structure, Multiplicative Codes Enable Generalization
jm_alexia · x · 2026-09-06
A new paper from RTomMcCoy's group, the result of an 8-year project, shows that LLM representations harbor implicit symbolic structure—explaining why neural models excel in symbolic domains like language, code, and math.
- Key finding: models that generalize over combinatorial binding (10^10 combinations) exhibit multiplicative structure in their representations;
- Models that fail to generalize instead "memorize" objects as linear directions, which blocks generalization;
- The work also contrasts next-token-prediction and embedding paradigms, arguing LLMs are not fundamentally opposed to symbolic systems but implicitly realize one.
More from Research
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- GAFU Uses Generation-Side Representations to Boost Fine-Grained Visual Understanding in Unified Multimodal Models — Tengju Ru · 2026-10-23
- NTU Study: Learned Unmasking Policies Improve Speed-Accuracy Trade-offs in Diffusion Language Models — nanyang-technological-university-singapore · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23