First Passage Attention: path extrema enable perfect long-range retrieval with 25% cache
Yuxuan Liu · hf · 2026-10-21
A paper by Yuxuan Liu (University of Bergen) proposes First Passage Attention (FPA), replacing monotonic decay accumulation in attention with path extrema: a key's logit is lowered proportionally to the deepest dip of a learned scalar field below its level over the intervening span, enabling selective erasure of intermediate content while preserving structurally relevant older tokens.
- FPA generalizes the Forgetting Transformer (FoX) as a special case; with a shared learned walk, unpredicted logits of alive keys naturally form a stack.
- A constant-dimension FPA head realizes any subset of keys staying alive, whereas softmax heads require dimension linear in the number of keys.
- In synthetic experiments, one-layer FPA heads converge to target sets while softmax and FoX baselines fail on every seed.
- Scaled to a 110M-parameter LM, FPA matches FoX in perplexity and uniquely achieves perfect passkey retrieval at 8× training length, even with its KV cache compressed to 25%.
More from Research
- Seqlet caller follow-up: credits for DeepLIFT contributors — jmschreiber91 · 2026-10-21
- Center for AI Safety Publishes Full Syllabus for Intro to ML Safety Course — ArtificialOther · 2026-10-21
- Seqlet caller gets 22x speedup: 167,750 loci drop from 7.8s to 0.35s — jmschreiber91 · 2026-10-21
- X trends: SophontAI's Holon-S1 predicts gene activity from cancer slides; decision models rise — iScienceLuvr · 2026-10-21
- NEJM AI study: LLM-recommended care costs up to 10x more than physician orders — dvir_a · 2026-10-21
- Nucleotide dependency analysis turns genomic language models into functional element detectors — jmschreiber91 · 2026-10-21