2.3B MoE hybrid Mamba model matches Llama-3.2-3B with <1% of its pretraining FLOPs
tri_dao · x · 2026-09-23
A developer pretrained Rigel, a 2.3B MoE (360M active) hybrid Mamba-2 model that lands within a few points of Llama-3.2-3B using less than 1% of its pretraining FLOPs.
- No dedicated cluster: the single run hopped between H100s, A100s, V100s and TPU v5p/v6e on one codebase.
- Tri Dao amplified the thread, praising the effort of getting pretraining to work across three generations of Nvidia GPUs and two generations of TPUs, calling it a very good model for its size.
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12