Interpretability debate: under multi-hop reasoning, entities may not exist as representations
livgorton · x · 2026-10-19
A technical interpretability exchange: luciaquirke explains that "What is an attraction in Paris?" could activate a (France, Paris, Eiffel, Country) neuron and an (attraction, Eiffel, Golden Gate Bridge) neuron, whose boosts hit the "Eiff" token hardest — in this ontology, entities don't exist as such. livgorton pushes back: for multi-hop reasoning, wouldn't you still end up with entity-like representations? Touches on polysemanticity and compositional representations.
Related event: Monosemanticity debate: Does neuron count cap what a model can know?(8 posts)→
More from Research
- MathEconLib open-sourced: Axiom Math and top economists bring machine-checked proofs to economics in Lean 4 — soumitrashukla9 · 2026-10-23
- Ken Ono invites the community to build MathEcon on the new Lean 4 economics library — soumitrashukla9 · 2026-10-23
- IFM open-sources RL360: 8,192 concurrent sandboxes and ~1.3M rollouts for agentic RL — rupspace · 2026-10-23
- Science lifts expression of concern on 2024 AI debunking-conspiracies paper; conclusions unchanged — DG_Rand · 2026-10-23
- Embodied First Exam benchmarks coding agents on 1,090 tasks across 44 robots — ManlingLi_ · 2026-10-23
- Anima Anandkumar on neural operators: AI weather models beat supercomputers — AnimaAnandkumar · 2026-10-23