AirLLM streams model layers one at a time: 70B LLM on a 4GB GPU, 2.8T Kimi K3 under 4GB VRAM

alex_verem · x · 2026-09-14

Gavin Li's open-source AirLLM (26.9k stars, actively maintained) runs massive open models on tiny GPUs: 70B Llama on a single 4GB card, DeepSeek-V3 (671B) on 12GB, and Kimi K3 (2.8T) under 4GB — no quantization, distillation, or pruning.

The trick is layer-by-layer streaming: only one layer sits on the GPU at a time, so VRAM depends on layer size, not model size. Sparse MoE models stream one expert at a time, which is how a 2.8T model fits in less memory than a 70B one. One line of code supports Llama, Qwen, DeepSeek, Mistral, Phi, and Gemma, with new support for training on small VRAM (125B under 6GB).

The trade-off is speed: streaming layers from disk is slow, so it won't replace real inference setups — but it lets anyone with a gaming laptop poke at frontier-size open models.

Original post →

More from Infra

• Claymation body horror on 8GB: three wrong settings, not the GPU, made local renders slow — roro9- · 2026-10-30
• durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
• An unnamed firm already runs a fairly scaled dedicated vehicle for compute lending, says analyst — matt_slotnick · 2026-10-30
• GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
• Sam Altman says the chip supply chain's invisible hand is failing — and DRAM is why — SnoozeDoggyDog · 2026-10-30
• MoEShift runs a 125B MoE on a 24GB Mac: 7.5GB peak memory at ~10 tok/s — dave-tro · 2026-10-30

Infra channel →