François Fleuret: why doesn't causal attention's asymmetric information flow hurt?

francoisfleuret · x · 2026-11-17

Noted researcher François Fleuret poses a fundamental architecture question: due to the causal structure of Transformers, processing "The cat hunts the mouse" can enrich <mouse>'s embedding with "is hunted by a cat," but cannot symmetrically enrich <cat>'s embedding with "is hunting a mouse."

He asks why this asymmetry doesn't seem to be a problem in practice—an open discussion on the one-directional information flow of causal attention.

Original post →

More from Research

Research channel →