SemiAnalysis probes positional embeddings: what lies beyond common mathematical constraints?
burny_tech · x · 2026-09-13
- Attention has no innate sense of time; positional embeddings are the sole mechanism by which LLMs gauge token distance.
- A recent Jane Street × 3Blue1Brown YouTube video maps the space of embeddings satisfying common-sense mathematical restrictions, a space already well exploited in the ML literature.
- SemiAnalysis kicks off a 7-part thread asking: relax those constraints and is there a more general space of possible position functions? Experiments to follow.
More from Models
- Char-level models plus ngram or canon layers 'are the future', dev claims — cephaloform · 2026-09-13
- ~$11 per billion tokens: decentralized-trained model's inference optimizations revealed — bittingthembits · 2026-09-13
- Dev touts char-level models: no retokenization bugs, dead-simple string munging — cephaloform · 2026-09-13
- User Says Model Wrote Chinese in a Chat Title, Then Insisted It Was Armenian — hydralisk_hydrawife · 2026-09-13
- OpenAI open problem #2 cracked by human theorists; paper unifies and improves MRRW bounds — abeirami · 2026-09-13
- Burkov: Models Inventing Fake Issues Is the Same Sycophancy We Saw in 2024 — burkov · 2026-09-13