AirLLM runs a 70B LLM on a single 4GB GPU by streaming layer weights from disk
techNmak · x · 2026-10-03
The 35K-star open-source project AirLLM runs a 70B LLM on a single 4GB GPU without quantization, distillation, or pruning.
The trick is changing when weights enter VRAM:
- The checkpoint is split into per-layer shards on disk, and the full Hugging Face model is created on PyTorch's meta device, so parameters take no real memory
- Hooks load each layer's weights onto the GPU right before it runs and drop them immediately after, while prefetching the next layer in a pipelined fashion
- The full 70B checkpoint never needs to sit in VRAM at once
For MoE models it goes further: experts are streamed individually, loading only the ones a token actually routes to. The repo claims it can run the 2.8T-parameter Kimi K3 this way.
More from Infra
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19
- Leak Hints at Classified 1M-GPU Training Cluster in China That Skips HBM — teortaxesTex · 2026-11-19
- Tencent leaned on Japanese neocloud for AI compute; DeepSeek, Zhipu best positioned — teortaxesTex · 2026-11-19
- Open-source prefill engine for GLM-5.3-Flash beats vLLM by 3.2x on 4x B200 — CShorten30 · 2026-11-19