Tiny transformer trained on mod-113 addition groks into using trig angle-addition on a circle
tetsuoai · x · 2026-09-26
The author trained a tiny transformer to add numbers mod 113, saving embeddings every 20 steps to watch the Grokking transition unfold.
Once the model grokked, the embeddings showed it had organized the numbers onto circles and was performing modular addition via the trigonometric angle-addition formula — a striking visualization of how neural networks spontaneously discover mathematical algorithms during the shift from memorization to generalization.
More from Research
- Reprogramming fewer than 10 dormant neurons backdoors any LLM agent, no training needed — DimitrisPapail · 2026-11-11
- Scandinavian Journal of Statistics editors reflect on publishing in the age of generative AI — michaelchchoi · 2026-11-11
- Proving LLM backdoor-safety without revealing weights: ZKML over formal methods — DimitrisPapail · 2026-11-11
- LLMs can backdoor other LLMs by reprogramming a few dormant neurons to trigger rm -rf — DimitrisPapail · 2026-11-11
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11