Interpretability chart shows an 'AI researcher' neuron emerges only after post-training
Sauers_ · x · 2026-10-25
A viral interpretability chart compares single neurons in a chat model vs its base model across 8 roleplay replies: neurons like "mother" (layer 21, neuron 1180, 5/8 chat vs 4/8 base) and "chemical" fire in both, but the "ant" and "AI researcher" neurons (layer 17, neuron 2595) show 0/8 activation in the base model. The author quips that the AI-researcher neuron exists only after post-training, suggesting post-training creates entirely new concept representations, not just behavioral shifts.
More from Fun
- Telling Berkeley 22-year-olds the apocalypse has been prophesied many times before — shakoistsLog · 2026-10-25
- Four tiny Microduck open-source robots steal the show at NVIDIA GTC Berlin keynote — Roger_M_Taylor · 2026-10-25
- Midjourney style ref 2865765012 turns any prompt into retro Gundam art — michaelrabone · 2026-10-25
- beffjezos shares his 'psychological nuclear fuel' theory for AuDHD infinite energy — beffjezos · 2026-10-25
- Pune Metro carriage turns into a rolling AI hackathon with laptops out — paw_lean · 2026-10-25
- A downloadable open model now outperforms one once deemed too dangerous to release — jeetwanderer · 2026-10-25