Nestor: nested attention sparsity from HKU/Berkeley challenges MoE for pretraining
hkuhk · hf · 2026-10-27
Nestor, by Tianzhe Chu (HKU), Druv Pai and Yi Ma (UC Berkeley/HKU), explores conditional computation that generalizes the attention operator instead of the MLP. The induced sparsity naturally nests (hierarchically composes), and the paper reports improvements over standard MoE transformers in pretraining, long-context adaptation, and continual learning—offering an alternative route to scalable training.
Related event: HKU and Berkeley Propose Nestor, Outperforming MoE in Efficiency(2 posts)→
More from Research
- A forgotten 1953 computer run at Los Alamos launched nonlinear wave science — and Mary Tsingou was written out of its name — creatoroff · 2026-10-28
- Nomagic details using synthetic sim data to train rare-failure robot anomaly detection — m_wulfmeier · 2026-10-28
- Wayve researcher: there is no good sim2real — it's always real2sim2real — m_wulfmeier · 2026-10-28
- Late submission with homemade C++ GNSS library beats Kaggle Smartphone Decimeter Challenge winner — rsasaki0109 · 2026-10-28
- Math is not empirical; intelligence should target cheaper experimental signals — shlomifruchter · 2026-10-28
- ArabicNLP 2026 kicks off in Budapest alongside EMNLP with a two-day program — mageed · 2026-10-28