Ablation study: fine-grained MoE is the best LLM architecture change since GPT-3
stochasticchasm · x · 2026-10-31
nvpavlichenko released ablation results run with bux, written up as "I Tried Every LLM Architecture Change since 2020."
Key takeaways:
- Fine-grained MoE is the best architecture change since GPT-3
- Linear attention and hyper-connections also genuinely work
The author notes the coverage isn't literally exhaustive but close enough — a rare public head-to-head ablation of post-2020 architecture ideas.
More from Models
- SenseNova-U1.5-8B hits 125 tok/s on M5 Ultra in llama.cpp and spots a $1 receipt error — Binary_orchid · 2026-10-31
- Pundit: Anthropic killing 5-hour limit plus Fable launch makes it unstoppable — iruletheworldmo · 2026-10-31
- ChatGPT Daybreak Blue sessions may train OpenAI models even with training opt-out off — littmath · 2026-10-31
- Claude refuses to draw its own logo over copyright concerns — 141_1337 · 2026-10-31
- AI cracks a 27-year-old pillar of computer science, OpenAI and Anthropic claim — Fcking_Chuck · 2026-10-31
- Perplexity early evals: Gemini 4 Carbon tops WANDR benchmark for long-form research — AravSrinivas · 2026-10-31