Poking into Qwen3-8B weights reveals simpler neuron activation in early and late layers
maxsloef · x · 2026-10-21
Developer maxsloef noticed an odd pattern of redundancy while poking around in Qwen3-8B's weights: neurons in early and late layers learn much simpler activation patterns than those in middle layers. An accompanying chart of mean cos² per layer shows activation correlation peaking near 10% around layer 1, hovering at 1-2% through the middle, and rising again to 4-5% at the final layer — well above the shuffled-rows null baseline.
The author adds technical context: a neuron in this model is three vectors (gate, up, down). Gate and up act as detectors for concepts in the residual stream, controlling how much of the down vector gets written back; the gate has a nonlinearity (SwiGLU) while up does not. The finding suggests first/last-layer FFNs may serve simpler, more redundant roles.
Related event: Developer Teases Apart Qwen3-8B Weights, Finds Layered Activation Patterns(2 posts)→
More from Models
- NVIDIA Nemotron demos go live — mervenoyann · 2026-10-22
- Recursive's Flint: open local coding model hits 125+ tokens/s on a MacBook Pro — HeyAmit_ · 2026-10-22
- vLLM adds day-0 support for Flint: 500 TPS on a single GB200 — _rockt · 2026-10-22
- WIRED: Recursive's Flint shows AI-built AI is getting closer to real recursive self-improvement — cong_ml · 2026-10-22
- Dev Quips Argon Is Good Because It Adds Things You Never Asked For — andrew_n_carr · 2026-10-22
- XBOW built a full-chain Chrome exploit in 6 days with gpt-5.6-sol, earning Google's $250K first VRP bounty of 2026 — moyix · 2026-10-22