DiscipleOfZara: RoPE/GQA variants are all incremental — no attention-free revolution coming
DiscipleOfZara · x · 2026-11-03
DiscipleOfZara replies to kalomaze: 0→1 and 1→N are categorically different; only the CNN→Transformer shift was a real paradigm change. The original Transformer and all its variants (RoPE, SwiGLU, GQA) are recognisably Transformers — not an enormous shift — so there's no revolution requiring dropping attention.