SemiAnalysis probes positional embeddings: what lies beyond common mathematical constraints?
burny_tech · x · 2026-09-13
- Attention has no innate sense of time; positional embeddings are the sole mechanism by which LLMs gauge token distance.
- A recent Jane Street × 3Blue1Brown YouTube video maps the space of embeddings satisfying common-sense mathematical restrictions, a space already well exploited in the ML literature.
- SemiAnalysis kicks off a 7-part thread asking: relax those constraints and is there a more general space of possible position functions? Experiments to follow.
More from Models
- Leak: Google Reportedly Has Another 'Hella Chonky' Model After Carbon — iruletheworldmo · 2026-10-30
- Users report across-the-board GPT quality drop: weaker writing and broken instruction following — Kyrannio · 2026-10-30
- Leak claims Gemini Argon hits Pro trial from November 2nd (unverified) — Emotional-Company847 · 2026-10-30
- OpenAI retiring waves of legacy models: GPT-5 family dies Dec 11, GPT-4.1 by Jan 2027 — ThePeterMick · 2026-10-30
- DeepMind CEO Demis Hassabis tells Musk: testing with trusted partners, coming soon — demishassabis · 2026-10-30
- Adobe stands firm on not training on customer data, using licensed stock instead — omooretweets · 2026-10-30