Flint MoE model lands in llama.cpp: 39.8B total params, only ~2.5B active per token
jacek2023 · reddit · 2026-10-22
llama.cpp PR #30928 adds support for Flint (FlintForCausalLM), a decoder-only MoE Transformer by Recursive. Key specs: 39.8B total parameters with 2.5B active per token (including output head); 49 layers (2 dense + 47 MoE) with 256 routed experts per MoE layer, 6 selected per token plus an always-active shared expert; attention uses 35 sliding-window layers (513-token window) and 14 full-attention layers, 32 query heads / 4 KV heads, per-head QK RMSNorm and learned attention sinks; RoPE only on sliding-window layers. Context length 262,144 tokens, vocab 150,144, stated knowledge cutoff June 2026.
More from Infra
- Image-to-video in ComfyUI without a local GPU: Forma custom nodes, pay-as-you-go from $5 — no3us · 2026-10-22
- Meta's Muse Spark Open: 816B-parameter open-weight model, 1M context, live on Together — togethercompute · 2026-10-22
- Speculative data center debt repackaged and sold to pension funds, echoing subprime bubble — churchkey · 2026-10-22
- Running Qwen3.8 Flash Next on 4x RTX 3090s: Which Inference Engine? — Civil_Fee_7862 · 2026-10-22
- Microsoft details Azure Maia 300: co-designed silicon-to-datacenter AI system — thoefler · 2026-10-22
- Vantage DC reserves 1GW+ of Heron Link capacity for 800VDC AI datacenters — espricewright · 2026-10-22