Identical weights can hide wildly different neuron utility

TheGradient · x · 2026-07-24

A thread argues that raw weight size can badly mislead when judging a neuron's usefulness.

• Two neurons can have identical weight norms and still behave very differently.
• The example shows a ReLU neuron that is highly active for one input distribution but mostly dead for another, despite identical weights.
• The key point is that context and data distribution matter as much as parameter magnitude.

Original post →

More from Models

• MiniMax says AMD MI355X is now close to Nvidia B200 in model serving — hongyangzh · 2026-07-24
• NVIDIA’s OO Agents make an LLM agent look like a Python object — nvidia · 2026-07-24
• A pelican-on-a-bike SVG turned into a tiny open-model bake-off — Atretador · 2026-07-24
• Codex gets a usage-limit reset calendar for June–July 2026 — haider1 · 2026-07-24
• GLM-5.2 nearly doubles Kimi K3 on a long-horizon browser benchmark — zainhas · 2026-07-24
• KAT-Coder-V2.5-Dev goes open-weight with 35B total and 3B active parameters — AdinaYakup · 2026-07-24

Models channel →