How Qwen3-8B neurons work: gate/up detect concepts, down writes back to the stream
maxsloef · x · 2026-10-21
A technical follow-up to maxsloef's observation about Qwen3-8B's weights, explaining the neuron definition used in the analysis.
In this model, a neuron consists of three vectors: gate, up, and down. Gate and up function as detectors for concepts in the residual stream, controlling how much of the down vector gets written back to the stream. The gate has a nonlinearity (SwiGLU) while up does not. This structure gives the earlier finding — simpler activation in first/last layers — a concrete mechanistic footing.
Related event: Developer Teases Apart Qwen3-8B Weights, Finds Layered Activation Patterns(2 posts)→
More from Models
- Baseten releases open-source decision models mammoth-1 (27B) and elephant-1 (9B), free for a week — baseten · 2026-10-22
- Wiz uses Gemini to uncover CVSS 9.2 PHP core flaws affecting 70% of the web — ralucaadapopa · 2026-10-22
- NVIDIA Nemotron demos go live — mervenoyann · 2026-10-22
- vLLM adds day-0 support for Flint: 500 TPS on a single GB200 — _rockt · 2026-10-22
- WIRED: Recursive's Flint shows AI-built AI is getting closer to real recursive self-improvement — cong_ml · 2026-10-22
- Dev Quips Argon Is Good Because It Adds Things You Never Asked For — andrew_n_carr · 2026-10-22