DAMI: dual-axis memory initialization improves long-context retrieval in gated linear attention
SpeedAI-Lab · hf · 2026-11-03
SpeedAI-Lab and Beihang University researchers introduce DAMI (Dual-Axis Memory Initialization), addressing long-context retrieval in gated linear recurrent models.
- Problem: linear attention scales linearly with sequence length, but finite-state memory limits retrieval; memory retention depends on state decay shaped jointly by gate parameters (Alog, dtbias) and input activations.
- Method: jointly configure decay and sensitivity parameters before training.
- Validation: controlled Gated Delta-Net (GDN) experiments compare initialization strategies for memory timescales and input sensitivity across near/mid/far evidence retrieval.
More from Research
- RaiSim 2.13.0 lets you run robotics examples online and adds a fast fluid dynamics model — ChongZzZhang · 2026-11-03
- Four low-cost coding agents survive Minecraft for 12 days, with server-log audit of every milestone — Unusual-Spot-5846 · 2026-11-03
- DR Pixel: open-source page-native deep research agent beats OCR-based pipelines — Berkeley-NLP · 2026-11-03
- What OpenAI left out of its Navier-Stokes announcement — elsleightholm · 2026-11-03
- Paper rejected with two reports: the AI referee report was excellent, the human one cited themselves — birchlse · 2026-11-03
- Skyfall AI's SKY-001 world model matches GPT-6 Astra CAD accuracy at 150x lower cost — rohanpaul_ai · 2026-11-03