Liquid AI explains on-device small model design: antidoom training cuts doom loops from ~10% to 1% in LFM2.5
helloiamleonie · x · 2026-09-25
A detailed thread by a Liquid AI researcher on how the LFM2 family of on-device models (LFM2.5-350M to 2.6B) is built.
- Three constraints: memory (4GB usable on an iPhone 8GB device, must fit weights + KV cache), speed (agentic workflows involve many tool calls and reasoning steps), and quality.
- Architecture: efficiency must be designed from the ground up. Small models mix full-attention layers with efficient layers; since theoretical layer gains differ from real-world ones, they profile on target hardware and landed on grouped query attention combined with gated short conv layers.
- Embedding problem: large-vocab tokenizers bloat small models' embedding layers, shrinking effective parameter count, so they minimize the embedding layer.
- Post-training: modern recipe = SFT → RL plus on-policy distillation and agentic RL.
- Antidoom training: to fix doom loops (10%) in small reasoning models, they detect the loop, find the triggering token, reject/downsample it, and upsample alternatives — reducing doom loops to 1% in LFM2.5-2.6B. Also a nice complement to the sparse-reward problem in RLVR.
More from Models
- Gemini 4 Carbon matches predecessor's score but explores less, gets distracted — patience_cave · 2026-11-11
- Fable 5.2 takes MazeBench frontier, matches GPT-6 Astra puzzle-solving — patience_cave · 2026-11-11
- MazeBench: Fable 5.2 scores 15%, near-frontier at half price; Gemini 4 Carbon 11% — patience_cave · 2026-11-11
- ERNIE 5.5 Flash Preview enters LMArena Battle Mode for crowdsourced ranking — arena · 2026-11-11
- Indie dev flips from $20 Pro back to $200 Gemini Ultra after two days of deep use — yihui_indie · 2026-11-11
- Leak: Google's upcoming model lineup includes Hydrogen, Krypton, Helix and a video model — koltregaskes · 2026-11-11