Interpretability discussion: experiments suggest ~2 bits/parameter for factual recall
OrionJohnston · x · 2026-10-16
Responding to Neel Nanda's point that language models contain far more facts and memorized items than they have neurons (computational superposition), Orion Johnston notes that experiments suggest the density for factual recall is around 2 bits per parameter — a concrete data point in the interpretability discussion on model capacity and superposition, joined by researchers including Neel Nanda.