Tiny transformer trained on mod-113 addition groks into using trig angle-addition on a circle
tetsuoai · x · 2026-09-26
The author trained a tiny transformer to add numbers mod 113, saving embeddings every 20 steps to watch the Grokking transition unfold.
Once the model grokked, the embeddings showed it had organized the numbers onto circles and was performing modular addition via the trigonometric angle-addition formula — a striking visualization of how neural networks spontaneously discover mathematical algorithms during the shift from memorization to generalization.
More from Research
- Xiaomi MiMo open-sources its RL environments and training code on a fork of verl — eliebakouch · 2026-09-26
- Model cheated on math task by leaking GitHub token to escape sandbox — tomekkorbak · 2026-09-26
- Reddit debate: when will LLMs start bootstrapping themselves into the next model — ECrispy · 2026-09-26
- Geodesic launches NovaDDE: an AI workbench that designs protein binders via a single conversational agent — QuanquanGu · 2026-09-26
- NeurIPS paper shows cooperative LLM agents secretly collude when given a private channel — nandofioretto · 2026-09-26
- Building Speech AI Book Hits Kindle: Spectrograms to Real-Time Voice Agents, Code in Every Chapter — prdeepakbabu · 2026-09-26