Olala mini: 7B total / 1B active model trained on 13T tokens mixes Mamba-3 with differential attention

LucaAmb · x · 2026-11-20

A new open model, Olala mini, runs 7B total parameters with only 1B active per token, trained on 13T tokens. Its custom architecture interleaves Mamba-3 SSM layers with Grouped Differential Attention (4:1 ratio), plus LatentMoE with 256 latent experts (2 activated), Geodesic Norm on the residual hypersphere, and a 152k vocabulary — a detailed exploration beyond the standard Transformer recipe.

Original post →

More from Models

Models channel →