Tsinghua's KT-FT fine-tunes 700B MoE on RTX 4090s, up to 21.2x faster than offloading
机器之心 · wechat · 2026-10-16
KTransformers-FineTune (KT-FT), led by Tsinghua's MADSys lab and Qujing Tech with the LlamaFactory team, is an open-source CPU/GPU co-computation system for fine-tuning huge MoE models on limited VRAM. Key insight: the offload paradigm fails on sparse MoE loads — PCIe weight transfers (62.5ms per 1B BF16 params at PCIe 5.0 peak) take 22x longer than GPU compute (2.8ms on RTX 5090), so compute can't hide transfers. KT-FT instead runs MoE experts directly on CPU via AVX-512/AMX while the GPU handles attention, adding a semantic-aware proactive layout engine and work-stealing thread scheduling to handle routing skew. On DeepSeek-V3.1-671B, Qwen3.5-397B and Qwen3-235B, LoRA throughput reaches 2.5–21.2x offload baselines (166.8 vs 12.1 tokens/s in a 2K-sequence comparison). Optional LoRAExpert shared adapter path further improves Time-to-Quality. Deployments: Hisense scaled from 30B to 397B on eight RTX 4090s, cutting iteration cost 81.5%; JoyAI fine-tunes 700B MoE with 65K context on one RTX 4090. Fully open-source, integrated into LlamaFactory.
More from Infra
- SGLang Summit sponsors grow from 11 to 36 in one week; Lip-Bu Tan, Lilian Weng to speak — BanghuaZ · 2026-10-18
- Yacine jokes his datacenter KPI is gigawatt-hours: get paged if not burning enough power — yacineMTB · 2026-10-18
- OpenAI reportedly handled ~15T tokens daily in April, likely higher now — JoshPurtell · 2026-10-18
- 1.68x inference speedup drops, directly accelerating GRPO post-training — yacineMTB · 2026-10-18
- OpenRouter is only a fraction of the total addressable market, poster argues — sloppenheimer · 2026-10-18
- Developer builds change alerts for LLM prices and retirements across Bedrock, Azure and Vertex — jsandhu0780 · 2026-10-18