SpectralShift extends Gated DeltaNet context windows via spectral reparameterization
RUC-AIBOX · hf · 2026-09-17
Linear attention layers are increasingly replacing softmax attention for long-context modeling, yet existing context-extension methods keep continued pretraining without modifying these layers, ignoring the spectral properties of linear attention state dynamics. The RUC team studies long-context extension of Gated DeltaNet (GDN) through the spectrum of its transition matrix, identifying two key factors: a sufficiently broad slow spectral band aligned with target dependency length, and preservation of fast-decaying modes for state clearing and context switching.
They propose SpectralShift, which reparameterizes alpha-projection initialization to reshape the decay spectrum and enhance slow propagation, plus a learning-rate scaling for alpha projections to ease long-context training. Experiments show consistent long-context gains over training, offering an effective, efficient way to extend context windows of linear attention models. Code is open-sourced on GitHub.
More from Research
- XConf Estimates LLM Confidence From the Model's Own Track Record, Beating Self-Consistency at 1/10 Cost — CambUni · 2026-09-17
- Researchers Uncover 'Value Flattening' in PPO Critics — Sparse Supervision on 3 States Fixes It — Shanghai-AI-Laboratory · 2026-09-17
- PANORAMA Grounds Every Caption Phrase to Pixel-Level Masks, Tops New PanoCaps Benchmark — Panorama-grounding · 2026-09-17
- Four Scheduling Techniques Flatten MoE Training Memory Peaks, Enabling 1M Context at 10.4x Throughput — Shrey Pandit · 2026-09-17
- Extending SGD diffusion approximations to optimization over probability distributions — burkov · 2026-09-17
- Mallat's one-slide test: are generative models generalizing or just memorizing? — prof_kamilov · 2026-09-17