ktransformers: Run LLMs on 24GB VRAM
dosco · x · 2026-07-18
The ktransformers team from Tsinghua optimizes MoE model deployment with a straightforward approach:
- Keeping frequently used experts on the GPU while offloading less common ones to the CPU, enabling larger models to run on smaller VRAM.
- The post claims this allows DeepSeek-V3 / R1 to support 139K context on 24GB VRAM, achieving up to 28x speedups over standard setups. It also makes fine-tuning DeepSeek-V3 on 4x RTX 4090 GPUs possible.
- Developed by the Tsinghua MADSYS Lab, the project is licensed under Apache 2.0 and has surpassed 17,000 stars on GitHub.
Related event: ktransformers Enables Local Inference of Massive Models on 24GB VRAM(4 posts)→
More from Infra
- xAI's Colossus 1 hits ~14.6 exaflops FP64, ~5x Top500 leader — cis_female · 2026-10-27
- Lambda runs 19 B200 nodes in a 16-node power budget, boosting throughput 24% — TheZachMueller · 2026-10-27
- Podcast: Micron's HBM roadmap, Meta's AI infra, and NVIDIA's supply chain lock-in — BenBajarin · 2026-10-27
- SGLang MegaKernel + PD Pushes the TCO-Latency Frontier on GB300, SemiAnalysis Benchmarks Show — AccBalanced · 2026-10-27
- Scaling self-hosted Convex: compiling TypeScript Convex functions to WASM — leventov · 2026-10-27
- Anthropic hits $16.3B H1 revenue as compute efficiency jumps 5.2x in 18 months — daniel_mac8 · 2026-10-27