François Fleuret asks: what are the key modern features of SOTA transformers?
francoisfleuret · x · 2026-11-10
Former Meta AI researcher François Fleuret kicked off a discussion on X about the key "modern features" of SOTA transformer models. His list: hybrid attention stacks (e.g. repeating 3x sliding-window attention + 1x full-context attention), MoE FFNs, and fancy long-context strategies — though he notes most of those are mainly constant-factor KV cache size reductions. He asks the community what else belongs on the list.
More from Research
- Alexander Mattick's masterclass unifies energy-based models, diffusion, flows and flow matching — burny_tech · 2026-11-10
- Science Advances: 40.68 MHz waves flush brain waste in mice within 10 minutes — OwariDa · 2026-11-10
- Mayo Clinic explores biological avatars plus digital twins for clinical trials — HealthcareLdr · 2026-11-10
- JoVox: image-conditioned hierarchical flow matching generates 64³ voxels in one model — pmttyji · 2026-11-10
- HALO: an MCP plugin that audits personal agents' self-improvement loops on an L0-L5 scale — liuziwei7 · 2026-11-10
- LMMS Lab unveils Action-Sensing-Memory framework for personal multimodal agents — liuziwei7 · 2026-11-10