Ex-Meta researcher sparks discussion on modern SOTA Transformer features

Former Meta researcher François Fleuret initiated a discussion on X about key modern features of SOTA Transformers, citing SwiGLU, short-range convolutions, and hybrid attention patterns, and asking the community what else belongs on the list.

2026-11-10 ~ 2026-11-10 · 2 related posts

• François Fleuret asks: what are the key modern features of SOTA transformers? — francoisfleuret · 2026-11-10
• What defines modern SOTA Transformers? SwiGLU and short convs join the list — giffmana · 2026-11-10