Qwen model 14x slower on 4070 due to layers offloading to RAM
luckokkkk · reddit · 2026-08-24
A user discovered that Qwen3.8-27B ran at a sluggish 5.7 tok/s on an RTX 4070 Super, compared to 81.5 tok/s on an RTX 5090.
Investigation revealed:
- While the model has 66 layers, only 38 ran on the 4070 Super's GPU. The remaining 28 layers offloaded to system RAM due to VRAM constraints, creating a massive bottleneck.
Solution:
- The author released Picchio, an open-source tool that visualizes actual layer placement, GPU activity, and separate prefill/decode speeds in a single run, helping developers debug inference performance issues effectively.
More from Infra
- How Will Model Architecture Evolve When NVL72 Rack Equals Today's Node? — AashaySachdeva · 2026-08-24
- DeepSeek Local Deployment: 4x W7900 vs 2x Blackwell for Agentic Workflows — Retumbo77 · 2026-08-24
- Running Local LLMs on Shared Machines with Zero Data Retention — Developer-Y · 2026-08-24
- 3D DRAM touted as more efficient and higher bandwidth than HBM4 — zephyr_z9 · 2026-08-24
- UK data center water use is a fraction of leak losses — Yamapama · 2026-08-24
- PSA: Commercial GPUs offer marginal speed gains for video generation despite high cost — mwoody450 · 2026-08-24