Laurence Moroney on 2026 On-Device Small AI: Gemma 4 & Qwen 3.5 Top Picks
lmoroney · x · 2026-08-22
Laurence Moroney analyzes the state of small models (2-12B parameters) suitable for on-device deployment in 2026, focusing on permissive licenses and performance for phones/laptops/offline use.
- Gemma 4:
- License: Switched to Apache 2.0, making it ideal for education and commercial use.
- Specs: Ranges from 2.3B (E2B) to 31B. The new 12B Unified is an encoder-free multimodal model running on 16GB VRAM with native audio.
- Performance: E2B achieves 133 prefill/7.6 decode tok/s on Raspberry Pi 5 CPU, higher on NPUs.
- Qwen 3.5: Also Apache 2.0, another pillar of the on-device ecosystem.
- Others: SmolLM3 publishes a full recipe; Llama 4 is open-weight but requires license review.
Moroney recommends prioritizing Gemma 4 and Qwen 3.5 for local stacks, emphasizing memory bandwidth and reproducibility.
Related event: Gemma 4 and Qwen3.5 Top Picks for On-Device Models in 2026(2 posts)→
More from Infra
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16
- Meta launches Muse Spark Open, says it runs on two M5 Ultra Mac Studios with Apple's help — The Verge AI · 2026-10-16
- vLLM ships Day-0 LoRA RL for Muse Spark Open, lifting AIME from 0.56 to 0.85 — vllm_project · 2026-10-16
- DSpark draft model for Muse Spark Open proposes 7 tokens per pass with 23-42% acceptance — vllm_project · 2026-10-16