Leaked ~30B Parameter Dense Multimodal Model: Architecture and Experimental Advantages
A_K_Nain · x · 2026-08-10
A developer shared architectural details of a newly released 30B parameter model on social media, expressing optimism about its potential.
Key Architecture Details:
- Dense model with a separate vision encoder.
- Utilizes SSSL, SwiGLU, and RoPE.
- Features 52 layers and a vocabulary size of 202,048.
- Multimodal capabilities.
Evaluation: The author notes that being a dense model is the release's strongest advantage, as dense architectures are significantly easier to experiment with than MoE. If its performance holds up, it could become the go-to model for small-scale experiments.
Related event: Leaked Architectures of New 30B to 400B Parameter Models Spark Discussion(2 posts)→
More from Models
- Google Releases DiffusionGemma Technical Report on Text Diffusion — rseroter · 2026-08-11
- Mistral Releases 3B Open-Weights Content Moderation Model — thione · 2026-08-11
- OpenAI Upgrades ChatGPT with GPT-5.6 Sol, Reducing Factual Errors by 68% — thione · 2026-08-11
- Ex-Reddit CEO Yishan Wong Complains About Claude's Gratingly Awful Speaking Style — ZeroStateReflex · 2026-08-10
- Open Source Catches Up: WebDev Code Arena Gap Narrows to 10 Points — arena · 2026-08-10
- Gemma RL Finetune for News Updated: Shorter Headlines but Sparse Bodies — ivan_bezdomny · 2026-08-10