Indie dev open-sources an 11M sparse MoE SLM pretrained on 5.7B tokens for fine-tuning
Nameis19letterslong · reddit · 2026-10-23
A developer pretrained LS-Micro-Base on 5.7B tokens: an 11M-parameter sparse MoE with a 7.28M backbone (5.52M active, top-1 routing over 2 experts) plus 4.21M ngrams, and a 2048 context window vs the 256/512 typical at this size.
On their Open SLM Leaderboard it would rank around the top 64% overall, top 47.4% among sub-50M models and top 37.6% under 10M.
It's meant as a base for specialist fine-tunes — the pretraining mix covers general knowledge, chat, math, tool calling, instruct and code. The full training pipeline (cudalm-c, CUDA with Blackwell optimization and full CPU parity) is open-sourced, with a 5-minute quickstart on the HuggingFace model card.
More from Models
- Claim: Anthropic kills the REA, Claude now refuses to do it — kingslyj · 2026-10-23
- Google reportedly building a Decision model with new createDecision method in Gemini API — lyraxana · 2026-10-23
- DeepSeek V4.1 Flash test: official API hits 99% cache vs 92-96% on OpenCode Go — teortaxesTex · 2026-10-23
- 1017 Reportedly Destroys Opus 5.5, Astra and Sol at Building a Complex Machine — teortaxesTex · 2026-10-23
- Professor: Keep Your Unique Voice — Experts Spot LLM Writing Instantly — plopesresearch · 2026-10-23
- Kimi K3.1 rumored to drop imminently, screenshot shows it live in Kimi Code — 141_1337 · 2026-10-23