HKU and Berkeley Propose Nestor, Outperforming MoE in Efficiency

Nestor from HKU and UC Berkeley replaces MoE's MLP sparsification with nested attention sparsity, outperforming MoE baselines of matched sparsity in both pretraining and long-context continued pretraining efficiency.

2026-10-27 ~ 2026-10-27 · 2 related posts

• Nestor: nested attention sparsity from HKU/Berkeley challenges MoE for pretraining — hkuhk · 2026-10-27
• Nestor outperforms matched-sparsity MoE baselines in pretraining and long-context finetuning — teortaxesTex · 2026-10-27