Schmidhuber explains his 1991 positional encoding: hyperbolic 1/t time-decay still widely used
SchmidhuberAI · x · 2026-10-06
- Schmidhuber revisits his 1991 positional encoding, which was based on a 1/t (or 1/Δt) relationship: a hyperbolic time-decay bias to encode sequence order, still in wide use today.
- His rationale: human memory and many real-world physical systems forget information following a power law, and a 1/t relationship forces the system to prioritize immediate local context over long-range dependencies.
- He notes Google has made a similar observation, framing this as context for current debates on positional encoding design.
More from Research
- New algorithm makes superword tokens practical without slow context overhead, COLM 2026 poster claims — yuvalpi · 2026-10-06
- Stanford's Agent0 evolves agents from zero data, beats self-play baselines — yuyinzhou_cs · 2026-10-06
- UW PhD Shangbin Feng enters 2026-27 faculty market with participatory AI research agenda — shangbinfeng · 2026-10-06
- Babies' brains sync with strangers via dad's scent, Science Advances study finds — rickasaurus · 2026-10-06
- Decision grader replaces LLM judge: 32x cheaper, 8x faster, 94% agreement on evals — rhythmrg · 2026-10-06
- Raghunathan lab to present pretraining safety and adaptation papers at COLM 2026 — AdtRaghunathan · 2026-10-06