USC paper finds 9 LLMs converge on Fourier number features at periods 2, 5, 10

BrihiJ · x · 2026-09-29

A paper from USC and UC San Diego, Convergent Evolution, to be presented at COLM 2025, examines how different language models learn strikingly similar number representations.

• Universal Fourier spikes: 9 pretrained LLMs (GPT-2, Llama-3/4, DeepSeek-V3, Mamba family, Kimi-Linear, etc.), classical embeddings (GloVe, FastText), and even raw number-token frequency distributions all show Fourier spikes at periods T = 2, 5, 10.
• Two-tier hierarchy: spectral spikes are universal, but not all architectures learn geometrically separable features usable for linear mod-T classification. Of four 300M-param architectures trained on 10B FineWeb-Edu, Transformer, Gated DeltaNet, and Mamba-2 develop linearly separable mod-T classes while an LSTM stays at chance despite a larger spike.
• Theory: Theorem 1 shows Fourier sparsity is necessary but not sufficient for mod-T linear separability; within-class scatter controls the gap.
• Two routes to geometric convergence: complementary text co-occurrence signals, and multi-token addition where per-digit carries turn each position into a modular subproblem.

The author, on the job market, is also presenting posters on token-level off-policy learning and SAEs for reasoning.

Original post →

More from Research

• RL Gains May Hit a Wall: Verifiable Data Limits and Stalling nanochart Benchmarks — QuintinPope5 · 2026-09-29
• TALES Benchmark on LLM Game-Playing Accepted to NeurIPS 2026, New Results Coming — tw_killian · 2026-09-29
• UBC's Schmidt wins ECML PKDD Test of Time Award for 2016 gradient convergence paper cited 2,000+ times — MarkSchmidtUBC · 2026-09-29
• Amazon AGI's AutoGym auto-generates tasks, environments, and verifiers for agent RL training — omarsar0 · 2026-09-29
• RECLAIM preprint: best AI agent reproduces only 41% of papers with code, 15% without — VraserX · 2026-09-29
• Diffusion Schrödinger Bridge paper solves entropy-regularized optimal transport to speed up generative modeling — skoularidou · 2026-09-29

Research channel →