kalomaze on the architecture debate: incremental-looking changes hide big underlying shifts
kalomaze · x · 2026-11-03
Responding to the claim that RoPE/SwiGLU/GQA are mere incremental tweaks and only the CNN→Transformer jump was a real paradigm shift, kalomaze argues that's precisely the point: improvements can look incremental locally while the underlying architecture itself shifts substantially over time — nobody denies a model with RoPE, SwiGLU and GQA is a Transformer, but that doesn't mean the architecture hasn't meaningfully evolved.
More from Models
- NorthCUA-Pro hits 93.3 on ScreenSpot-Pro, edging out GPT-6 Astra for new SOTA — NielsRogge · 2026-11-03
- Mystery 'Pulse Alpha Free' model with 1M context appears stealth-free on OpenCode — mark_k · 2026-11-03
- Anthropic blocks Claude Code from cloning existing video games after knockoff wave — The Decoder · 2026-11-03
- Researcher flags BioReason benchmark flaws: model unresponsive to randomized inputs — anshulkundaje · 2026-11-03
- AgentTime Update: GPT Models Lead Time-Awareness, Best Forecast Miss Is 2.26× — maksym_andr · 2026-11-03
- paul_cal catalogs the untold failures of LLM tuning — paul_cal · 2026-11-03