teortaxes: info-theoretic limits block models from spontaneously forming von Neumann architectures
teortaxesTex · x · 2026-10-28
In a thread about whether training methods could encourage modular "computers" to form inside neural networks, teortaxesTex pushes back on Yacine's suggestion of fast architecture sweeps with pufferlib: he argues it's a fundamental information-theoretic problem, and slopping RL tasks or procedural data won't make a model build von Neumann architectures out of circuits.
Related event: Debate: Can Neural Nets Spontaneously Emerge Modular Structures?(3 posts)→
More from Research
- Debate set: Can we make AIs sentient — and should we? Solms vs. Humphrey — cccalum · 2026-10-29
- Samaya post-trains a 30B model to beat frontier AI on financial research at 11x lower cost — maithra_raghu · 2026-10-29
- Samaya's 30B post-trained model beats frontier on financial research at 11x lower cost — maithra_raghu · 2026-10-29
- Scaling cross-situational word learning: 2,441 hours of child-perspective video from Stanford team — RichmanRonald · 2026-10-29
- Curiosity algorithms need world models plus locomotor primitives to replicate animal behavior — aran_nayebi · 2026-10-29
- Virtual fly study finds autonomous exploration maximizes population-level state coverage — aran_nayebi · 2026-10-29