Berkeley team trains 2.3B MoE Hybrid Mamba matching Llama-3.2-3B with <1% of pretraining FLOPs
berkeley_ai · x · 2026-09-24
A UC Berkeley academic team unveiled Rigel, a 2.3B-parameter MoE (360M active) Hybrid Mamba-2 model that lands within a few points of Llama-3.2-3B dense using less than 1% of its pretraining FLOPs.
Key points:
- No dedicated cluster: the run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase.
- The team argues academia can compensate for scarce compute with creative students training novel architectures on whatever hardware they can find.
- They are seeking resources to scale the research to the next training rung.
A notable demonstration that new architectures can be validated on a shoestring, heterogeneous compute budget.
More from Research
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12