Nestor outperforms matched-sparsity MoE baselines in pretraining and long-context finetuning
teortaxesTex · x · 2026-10-27
TianzheC introduces Nestor, a new architecture that is more compute-efficient than a matched-sparsity MoE baseline in both pretraining and long-context continued pretraining. The reposter, Xinyu2ML, praises the elegance of its sparsity design across both token and channel dimensions for attention, while expressing doubts about its practicality on real hardware.
Related event: HKU and Berkeley Propose Nestor, Outperforming MoE in Efficiency(2 posts)→
More from Research
- OpenAI claims 100+ longstanding math problems solved, 700+ results on GitHub in two months — haider1 · 2026-10-27
- Study: just 10 minutes of AI assistance can erode motivation to solve problems alone — Polymarket · 2026-10-27
- Murea Labs' InWorld: AI-Generated World That Remembers Characters and Objects Turns Later — jiqizhixin · 2026-10-27
- GPT-6 moves a camera to insert a flower, yet burns an egg: reflections on the Astra report — liuziwei7 · 2026-10-27
- Eric Topol and colleagues in Nature: measuring immune system status in the clinic is about to change — EricTopol · 2026-10-27
- Google's free AI Harness Crash Course interactively breaks down agent engineering — datawithsuman · 2026-10-27