What defines modern SOTA Transformers? SwiGLU and short convs join the list
giffmana · x · 2026-11-10
- François Fleuret asks what the key "modern features" of SOTA Transformer models are, listing hybrid attention (e.g. 3x SWA + 1x FCA), MoE FFNs, and fancy long-context strategies (mostly constant-factor KV cache reductions).
- Lucas Beyer adds SwiGLU as a definite non-obvious departure from the original, and notes short convs keep showing up though they may not be fully established yet.
Related event: Ex-Meta researcher sparks discussion on modern SOTA Transformer features(2 posts)→
More from Research
- VLK generates large-scale synthetic data for humanoid loco-manipulation without teleoperation — pabbeel · 2026-11-11
- MacroData Labs launches MacroHands, SOTA 3D hand models from egocentric video — gui_penedo · 2026-11-11
- Representation-readout decomposition unifies grokking and double descent — s_y_chung · 2026-11-11
- Research World Models: predict which AI research experiments are worth the compute — yuyinzhou_cs · 2026-11-11
- AI Trials Observatory launches with 93 randomized trials on how AI affects people — DG_Rand · 2026-11-11
- Newton physics engine ships FBA solver, enabling cloth sim2real in minutes — ChongZzZhang · 2026-11-11