DiscipleOfZara: the core insight is still Attention Is All You Need
DiscipleOfZara · x · 2026-11-03
DiscipleOfZara restates his point: the core insight remains Attention Is All You Need, and everything kalomaze listed — RoPE, SwiGLU, GQA — is an incremental improvement on the Transformer; nobody looks at such a model and says it's not a Transformer. Part of the ongoing architecture debate with kalomaze.