AirLLM streams model layers one at a time: 70B LLM on a 4GB GPU, 2.8T Kimi K3 under 4GB VRAM
alex_verem · x · 2026-09-14
Gavin Li's open-source AirLLM (26.9k stars, actively maintained) runs massive open models on tiny GPUs: 70B Llama on a single 4GB card, DeepSeek-V3 (671B) on 12GB, and Kimi K3 (2.8T) under 4GB — no quantization, distillation, or pruning.
The trick is layer-by-layer streaming: only one layer sits on the GPU at a time, so VRAM depends on layer size, not model size. Sparse MoE models stream one expert at a time, which is how a 2.8T model fits in less memory than a 70B one. One line of code supports Llama, Qwen, DeepSeek, Mistral, Phi, and Gemma, with new support for training on small VRAM (125B under 6GB).
The trade-off is speed: streaming layers from disk is slow, so it won't replace real inference setups — but it lets anyone with a gaming laptop poke at frontier-size open models.
More from Infra
- agi-memory: SQLite-only persistent memory MCP server for coding assistants, 32MB RAM — Rude_Gate7599 · 2026-09-14
- $3000 home server with 128GB VRAM runs Qwen3.8-next at 1.3k tps prefill, 70 tps code — Thin_Pollution8843 · 2026-09-14
- Top 10 foundry revenue hits $53.5B in Q2, TSMC holds 72.5% share — Beth_Kindig · 2026-09-14
- Qwen3.8 27B INT4 With 144K Context Runs on a Single RTX 3090 via vLLM — Altruistic_Heat_9531 · 2026-09-14
- Nvidia's $59.7B Quarterly Net Income Works Out to ~$656M Profit Per Day — himanshustwts · 2026-09-14
- Tiny Neural Nets Revival: Transformer Hits ~1500 tok/s on M4 CPU via SME2 Instructions — GregoryDiamos · 2026-09-14