LLMs compute addition via Fourier features: 28 LLaMA neurons implement the mechanism
banburismus_ · x · 2026-10-18
Citing @dronh's paper: language models use Fourier features to compute addition, and a set of 28 MLP neurons in LLaMA implements this general addition mechanism, with subsets corresponding to Fourier features of different periods. @banburismus uses this to push back on @JacobHHilton's claim that circuits get harder to interpret at scale.
Related event: Study Finds LLMs Use Fourier Features for Addition with Just 28 Neurons(2 posts)→
More from Research
- Open-source Engram memory engine, modeled on databases, matches Mem0 on BEAM — phecdaYm · 2026-10-18
- Mathematician: Pages of Routine Analysis Work Well, and AI Makes Writing Them Easier — jessi_cata · 2026-10-18
- Chinese researchers argue harness engineering is over: the eval set becomes the job — blaizedsouza · 2026-10-18
- Andrew Ng: 'AI for Science' will become meaningless as AI becomes every scientist's tool — andrewgwils · 2026-10-18
- Open-source Intro to Semiconductor Manufacturing book follows a wafer from sand to chip in 25 chapters — CatAstro_Piyush · 2026-10-18
- Linus Lee warns embedding models shouldn't be narrowly optimized for retrieval — thesephist · 2026-10-18