Developer Teases Apart Qwen3-8B Weights, Finds Layered Activation Patterns

A developer dissecting Qwen3-8B weights found that neurons in shallow and deep layers learn notably simpler activation patterns than middle layers, measured via per-layer mean cos². Each neuron comprises gate, up, and down vectors, with gate/up detecting concepts and down writing back to the residual stream.

2026-10-21 ~ 2026-10-21 · 2 related posts

• Poking into Qwen3-8B weights reveals simpler neuron activation in early and late layers — maxsloef · 2026-10-21
• How Qwen3-8B neurons work: gate/up detect concepts, down writes back to the stream — maxsloef · 2026-10-21