SlideDP fine-tunes Qwen2.5-72B on four RTX 4090s, beating FSDP2 throughput by 11.2%
Ruijia Yang · hf · 2026-10-01
- Problem: Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources; replicated transfers amplify traffic and strong scaling exposes host work.
- Approach: SlideDP is a synchronous data-parallel runtime for shared-host multi-GPU systems: one authoritative host state, communication routes decoupled from state layout, and pipelining of parameter delivery, gradient aggregation, and CPU updates across ranks and chunks, guided by an analytical step-time model.
- Results: Geometric-mean throughput ratios of 1.46–2.64x over SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s it approaches GPU-resident FSDP2 throughput for Qwen3-14B; with larger batches it processes over 1M tokens per step, exceeding FSDP2's measured peak by 11.2%. It also supports 256K-token sequences and fine-tunes Qwen2.5-72B on four RTX 4090 GPUs. Code open-sourced.
More from Infra
- tandem.cpp v0.3.0: 2-6x faster prefill for models too big for VRAM, Maya DeepSeek-V4.1 support — smflx · 2026-11-16
- Developers poll real costs of running coding agents on remote VMs for work — radim11 · 2026-11-16
- Datadog's new gateway returns 402 when a model's budget runs out—so agents just retry — YvesMulkers · 2026-11-16
- Max-quality local Minimax H3 video on a 4090: community-tested int8 setup — newaccount47 · 2026-11-16
- Local V2V outfit transfer on a 16GB M3 MacBook Air: 15.7 min cut to 2.4 min — might_thinker · 2026-11-16
- FlashStates hits 3.14x vLLM prefill throughput for hybrid-architecture agentic inference — 机器之心 · 2026-11-16