Linear attention suffers poor recall on million-token DNA sequences
No-Coffee-8227 · reddit · 2026-08-16
A researcher modeling DNA sequences (up to 1M tokens) found that linear attention, while efficient, fails at long-range recall, achieving only 25% on Needle-in-a-Haystack tests (random chance). This issue persists even with established models like HyenaDNA. Performance drops as context length increases, despite decent results at 16K context. The author seeks architectural solutions that avoid expensive softmax or external memory.
Related event: Linear Attention Struggles with Recall in Long DNA Sequences(2 posts)→
More from Research
- GAE paper learns a geometry-native latent space for 3D-consistent world generation — yshan2u · 2026-10-16
- Agents turn conservative as context grows: edits shrink from 150 to 30 lines — dbreunig · 2026-10-16
- LMR-BENCH at EMNLP 2025 tests whether LLM agents can reproduce real LLM research — ZhiyuChen4 · 2026-10-16
- Ex-OpenAI researcher claims most neurons are legible now; critics call it hype — anshulkundaje · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Survey: AI agent evaluation is getting realistic but still fails to measure safety and reliability — burkov · 2026-10-16