Olala mini: 7B total / 1B active model trained on 13T tokens mixes Mamba-3 with differential attention
LucaAmb · x · 2026-11-20
A new open model, Olala mini, runs 7B total parameters with only 1B active per token, trained on 13T tokens. Its custom architecture interleaves Mamba-3 SSM layers with Grouped Differential Attention (4:1 ratio), plus LatentMoE with 256 latent experts (2 activated), Geodesic Norm on the residual hypersphere, and a 152k vocabulary — a detailed exploration beyond the standard Transformer recipe.
More from Models
- Even with OpenAI's Daybreak access, Codex still blocks cybersecurity requests — AIFlow_ML · 2026-11-20
- DeepSeek v4.1 flash reportedly served at higher quality for 10% of the price — _xjdr · 2026-11-20
- Open-source decision model Laya hits 35k stars, outputs calibrated probabilities in 33ms — kingslyj · 2026-11-20
- Google's Logan Kilpatrick teases Gemini subscription changes starting December 2 — teortaxesTex · 2026-11-20
- Analyst: Opus 5.5 jump wasn't compute-limited — China's real gap is data engineering — teortaxesTex · 2026-11-20
- Kelsey Tuoc: Gemini Carbon+Neon is 'currently unbeatable' at running experiments and analyzing results — ZeroStateReflex · 2026-11-20