Princeton NeurIPS paper: data diversity, not architecture, drives symbolic mechanisms in LLMs
burny_tech · x · 2026-11-19
A NeurIPS 2026 paper by Melody Zixuan Li and Taylor Webb (Princeton) trains neural networks from scratch on abstract sequence tasks to find what drives emergent symbolic mechanisms in LLMs. Key findings: - Transformers trained from scratch do develop symbolic mechanisms, confirmed via representational, attentional, and causal mediation analyses. - Across multiple axes, greater data diversity yields stronger symbolic signatures. - Surprisingly, the same signatures emerge even in MLPs, showing architectural inductive biases aren't necessary. - Consistent patterns also appear in entity-attribute binding tasks. Conclusion: data diversity, not architecture, is the primary driver of emergent symbolic computation.
More from Research
- SE3 Labs Releases First Open YAM Real-Robot Teleop Dataset: 500 Bimanual Episodes — paigeinsf · 2026-11-19
- 1.5 years under review: researchers urge releasing AI papers immediately after submission — carlosdponx · 2026-11-19
- Berkeley's StreamEmbed indexes your screen history on-device, 3.7x faster at matched accuracy — berkeley_ai · 2026-11-19
- NAACL RAF opens 2026-2027 grant proposals for NLP initiatives, due May 6 — keviv9 · 2026-11-19
- Open-sourced Indic profanity detector covers 12 Indian languages, runs at 4ms per comment on CPU — sudalairajkumar · 2026-11-19
- MAVE reconstructs editable 3D blockouts from film shots, works inside Blender — bdsqlsz · 2026-11-19