kalomaze: architectures better than the transformer exist, you just call them modified transformers
cephaloform · x · 2026-11-03
Replying to a question about whether OpenAI should search for architectures beyond the transformer, kalomaze argues better architectures appear all the time — we just file them under "modified transformers." GPT-3's architecture loses to baselines with RoPE, then RoPE+SwiGLU, then +GQA, and so on: architectural progress is incremental stacking, not clean-slate redesigns.
More from Research
- CUA-Master: Nested Dual-Loop Framework Scales Computer-Use Agents via Open-World Recursive Improvements — Xinlei Yu · 2026-11-03
- CrossSWE benchmark shows coding agents struggle with cross-repository development — Silin Chen · 2026-11-03
- kalomaze: assuming a step-change that requires dropping attention is even more exotic — kalomaze · 2026-11-03
- LLMs forecast economics RCT effects with 0.054 SD median error, beating naive benchmarks — RobbWiller · 2026-11-03
- Same policy can fail outside its training engine: Manda tests sim-to-sim transfer across MuJoCo, PhysX, Newton and Genesis — facontidavide · 2026-11-03
- Repeated cheap-model sampling beats frontier LLMs in vuln detection at 13% cost — andreamichi · 2026-11-03