Sebastian Raschka Maps October's 'Open October': 10 Open LLM Architectures in One Month
rasbt · x · 2026-11-03
Sebastian Raschka (rasbt) rounds up 10 open-weight LLM architecture releases from October — "Open October" — with an illustrated gallery (llm-gallery.com):
- Kolibri-1 (78B, Aleph Alpha): classic GQA + sliding window attention, 3.4B active
- Ling 3.1 Flash (560B, InclusionAI): Kimi Delta Attention plus gated MLA inspired by DeepSeek, 4.5% active
- Muse Spark Open (816B, Meta): Meta's first major open-weight release since Llama, GQA+SWA + LatentMoE
- Nemotron 3.5 Super VL (120B, NVIDIA): vision-language, mostly Mamba-2 layers with just 8 attention layers
- Dory1.0 (4B, NVIDIA): looped transformer design
- Reflection Beam (501B): GQA+SWA, 49 MoE layers with 256 experts
- Recursive Flint (40B): standard MoE with GQA and SWA
- MiniMax M3.1-Flash (428B): proprietary MiniMax Sparse Attention, quantization-aware training
- LFM3 (1.7B, Liquid AI): hybrid short convolutions + attention
- Mistral Large 4 (1T): multimodal MoE combining MLA with sparse attention
Common trends: GQA+SWA remains the baseline, while MLA, sparse attention, Mamba hybrids and latent MoE are the main innovation vectors. More deep dives promised.
Related event: Raschka Rounds Up 'Open October': 10 New Open LLM Architectures(2 posts)→
More from Models
- JFPuget's team takes 3rd on both Kaggle ARC-AGI competitions ending the same day — JFPuget · 2026-11-04
- Tencent's WeChat Vision team open-sources WeVisOne-VL, a unified 2B/4B perception VLM — bdsqlsz · 2026-11-04
- DeepMind-Stanford paper: a red heart in your prompt flips code from 0% to 100% recursive — rohanpaul_ai · 2026-11-04
- A Red Heart Can Steer Code: DeepMind-Stanford Paper Probes Visual Meaning in VLMs — rohanpaul_ai · 2026-11-04
- DeepSeek open-sources DeepInfer inference engine delivering 26,000+ tokens/sec per GPU — bycloud · 2026-11-04
- DeepMind RL-trained Gemini doubles correct psychiatric differential diagnoses in blind study — SRSchmidgall · 2026-11-04