kalomaze vs DiscipleOfZara: Debate Over Whether Transformer Architecture Is Still Evolving
On November 3, kalomaze and DiscipleOfZara engaged in multiple rounds of debate on social media over whether the Transformer architecture is still undergoing substantive change. At the heart of the dispute: whether improvements like RoPE, SwiGLU, and GQA count as paradigm shifts or incremental tweaks, as well as
2026-11-03 ~ 2026-11-03 · 5 related posts
Primary sources
- DiscipleOfZara: the core insight is still Attention Is All You Need — DiscipleOfZara · 2026-11-03
- kalomaze on the architecture debate: incremental-looking changes hide big underlying shifts — kalomaze · 2026-11-03
- [source] DiscipleOfZara: RoPE/GQA variants are all incremental — no attention-free revolution coming — DiscipleOfZara · 2026-11-03
- [source] kalomaze: attention-free step-change improvements are hopium, not reality — kalomaze · 2026-11-03
- kalomaze: assuming a step-change that requires dropping attention is even more exotic — kalomaze · 2026-11-03