François Fleuret dissects a fundamental transformer limitation: sequence and token indexing
francoisfleuret · x · 2026-11-17
François Fleuret started a technical thread on a fundamental trick — and source of limitation — of transformers that many miss.
- A transformer identifies both the sequence index and the token index in its "internal state".
- Due to causal attention, when processing "The cat hunts the mouse", <mouse>'s embedding can be enriched with "is hunted by a cat", but <cat>'s embedding cannot symmetrically be enriched with "is hunting a mouse".
- He then asks why this asymmetry is not a problem in practice, with the answer developed in the linked thread.
Worth following for anyone interested in attention mechanics and expressivity limits.
More from Research
- 8th NeurIPS Robot Learning Workshop to debate: Is Physical AI truly going zero-shot? — m_wulfmeier · 2026-11-17
- François Fleuret explains a memory trick: pay O(time steps) instead of O(steps × components) — francoisfleuret · 2026-11-17
- Fudan & OpenMOSS map the trade-off: long-context hybrids learn more, extrapolate less reliably — jiqizhixin · 2026-11-17
- Models favor their own APIs in code: Opus calls Claude API in 97% of implementations — santiviquez · 2026-11-17
- Search behavior gap: Opus searched in 10/60 answers, GPT models searched every time — santiviquez · 2026-11-17
- 79% of 2,880 LLM API recommendations went to just four big-cloud services — santiviquez · 2026-11-17