Near-zero token embeddings silently stall gated models, hitting 52.6% of open checkpoints
Xin Liu · hf · 2026-11-04
A new study exposes a hidden training pitfall in gated sequence models (Mamba, Gated DeltaNet, xLSTM): 52.6% of untied gated/hybrid open-weight checkpoints (20 of 38) contain reachable input embeddings below 10⁻⁶, versus 0% of weight-tied models. Near-zero rows arise when rare or template tokens get zero input gradients during pretraining while AdamW weight decay shrinks untied embeddings.
In gated models, a single near-zero token amplifies input-embedding gradients by over 30 (up to 35) orders of magnitude, causing extreme gradient amplification and training instability; Transformers remain unaffected because non-positional embeddings or the BOS token are protected.
The authors propose GroupClip and GateFloor to constrain embedding norms and stabilize gated behavior.
More from Research
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04
- PixelCraft: A Multi-Agent Framework Targets Multimodal AI's Chart-Reading Failures — burkov · 2026-11-04