Solving long-range recall in linear attention for DNA sequences
No-Coffee-8227 · reddit · 2026-08-16
- Issue: When using linear attention for DNA sequences up to 1M tokens, long-range recall drops to 25% (near-random chance) in Needle-in-a-Haystack tests.
- Observations: Existing models like HyenaDNA show similar poor performance, while smaller models perform well (50-60%) at 16K context, suggesting a length-related issue.
- Limitations: Current solutions often rely on external memory, sliding windows, or hybrid architectures, which the author wants to avoid.
- Question: Is this a fundamental limitation of linear attention's compressed state, or are there architectural approaches that can preserve retrieval?
Related event: Linear Attention Struggles with Recall in Long DNA Sequences(2 posts)→
More from Research
- Researcher slams papers whose stacked changes beat even weak baselines barely — retr0jirachi · 2026-10-16
- FrankenSim builds a physics-based virtual orchestra, piano and drum demos live — doodlestein · 2026-10-16
- FoveaVLA replaces wrist cameras with learned foveation, hitting 70.8% success — Haoyu_Xiong_ · 2026-10-16
- ApolloPFN: a zero-shot forecasting network that natively handles exogenous covariates — burkov · 2026-10-16
- ICLR reviewer says 4/5 papers in review pile read like AI slop — berkustun · 2026-10-16
- Tsinghua's DBS learns content-adaptive frequency band splits for audio source separation — thu-ml · 2026-10-16