Paper models Transformer components as stochastic geometry and tests five architectures
Zhihua Liang · hf · 2026-07-21
Continuous geometric framework for Transformers
This paper recasts core Transformer components — RMSNorm, RoPE, softmax attention, FFN, residual streams, SGD, and weight decay — as parts of an integro-differential equation on a semantic fiber bundle.
What it claims
- A single geometric axiom: token sequences form a discrete 1-manifold with a canonical measure lattice.
- Attention is interpreted as entropic optimal transport / a Schrödinger bridge.
- SGD is modeled as Itô diffusion that breaks detailed balance.
Experiments
The authors run a six-part campaign across five architectures — Qwen3, LLaMA-3.1, Gemma-3, GPT-2, and Mistral — covering 124M to 8B parameters.
They report quantitative agreement with several geometric predictions, including:
- ε^{-1/2} Lipschitz scaling calibration with R² = 1.000
- Lie–Trotter operator-splitting torsion
- symmetric ablation instability
- O(1/k) suppression of Poincaré recurrence on the RoPE torus
- a thermodynamic context-limit phase transition
- a non-equilibrium steady-state parameter vortex
Takeaway
The paper argues that continuous stochastic differential geometry can provide a predictive vocabulary for Transformer stability limits, context bounds, and optimization dynamics.
More from Research
- Microsoft open-sources Resource2Skill to turn videos and articles into executable agent skills — aigclink · 2026-07-21
- Microsoft open-sources Resource2Skill to turn tutorials into executable agent skills — aigclink · 2026-07-21
- Soofi S 30B-A3B releases a full pretraining report and claims open-model leads in English and German — abursuc · 2026-07-21
- Shared agent workspaces fail in a fixed order, from stale reads to zombie writes — mrvladp · 2026-07-21
- A solo founder built an enterprise knowledge graph without a graph database — TheRedfather · 2026-07-21
- Practical rolling-shutter pose estimation uses affine correspondences — ducha_aiki · 2026-07-21