Why You Can't Just Remove Positional Embeddings: Self-Attention Is Permutation Equivariant
cneuralnetwork · x · 2026-10-29
A concise Q&A from @cneuralnetwork on why transformers need positional information.
- The bad answer: "because transformers need to know word order" — restates the conclusion without explaining the mechanism.
- The good answer: self-attention without positional information is permutation equivariant — rearranging input tokens just rearranges the outputs correspondingly, and the model cannot tell which token came first.
The takeaway is a learning principle: understand the math behind the mechanism rather than memorizing architectural facts.
More from Research
- Reuters Institute 9-country survey: weekly AI use jumps from 34% to 51% in a year — _FelixSimon_ · 2026-10-29
- Next neural twins will let models control what their partner can read — Sauers_ · 2026-10-29
- Interpretability find: a Talkie neuron makes the model believe it's Walter Scott trapped in his own novel — repligate · 2026-10-29
- Linked AI twins copy each other's test answers — even when the partner is wrong — Sauers_ · 2026-10-29
- Neural twins write with interleaved tokens as one voice while chatting with each other — Sauers_ · 2026-10-29
- Removing one training dataset made the linked 'neural twins' score higher on happiness probes — Sauers_ · 2026-10-29