Neural net signal propagation debate: it's about shaping activations, not the mean
roydanroy · x · 2026-11-01
A signal propagation expert pushes back on a thread about large neural networks: the issue isn't the mean of post-activations, since preactivations end up roughly zero-mean after weight multiplication. What's actually happening is "shaping" activations for better signal propagation through the network — a technical nuance thread on initialization and training dynamics.
More from Research
- Two worlds: France's star mathematician shuns AI while anonymous peers push further with it — IgorCarron · 2026-11-01
- Stanford-Elsevier top 2%: Swiss AI researchers lead with 12.0 citations per paper — NegotiationOk7535 · 2026-11-01
- UIUC, Meta, Amazon & DeepMind release 135-page survey charting agentic reasoning for LLMs — mdancho84 · 2026-11-01
- Andrew Ng: work on mechanistic understanding of generalization isn't mech interp as narrowly construed — andrewgwils · 2026-11-01
- Kornia, the geometric computer vision library for Spatial AI, holds 11.6k stars — kornia · 2026-11-01
- Google Research's sleep framework lets LLMs consolidate memory offline, retaining up to 10M tokens — iruletheworldmo · 2026-11-01