Running 2.4T Qwen3.8 Model on RTX 5090 + 5060 Ti: 0.8 tok/s Tested
mossy_troll_84 · reddit · 2026-08-14
A developer successfully ran the massive 2.4T parameter Qwen3.8-2.4T-A95B model on consumer hardware, sharing detailed benchmarks and configuration insights.
Hardware & Environment
- GPUs: RTX 5090 (32GB) + RTX 5060 Ti (16GB)
- Other: AMD Ryzen 9 9950X3D, 128GB DDR5 RAM
- Model: Unsloth Q10 GGUF quantization (397 GiB)
Performance
In a controlled 32-token generation test, the setup achieved a generation speed of 0.80 tok/s. Enabling native MTP (speculative decoding) resulted in a 3.64% throughput increase and a 2.74% reduction in total wall time, with an MTP acceptance rate of 90.48%.
Configuration
The author shared optimal llama.cpp parameters, including a tensor split of 4,1 and offloading 91 MOE experts to the CPU. To prevent MTP from running out of VRAM, block 92 expert tensors were forced to remain on the CPU.
Related event: Developers Test Run 2.4T Parameter Qwen Model on Consumer Hardware(3 posts)→
More from Infra
- Musk says he wants 'maybe more than 10GW' of compute; Nvidia now counts SpaceX as a CSP and co-designs a space server — elonmusk · 2026-10-16
- Jensen Huang: Musk's AI buildout moved from K's to M's, now a CSP-tier NVIDIA customer — elonmusk · 2026-10-16
- EpochAI pegs GPT-6 training at up to 1.8e27 FLOP; scaling01 calls param estimates way too high — scaling01 · 2026-10-16
- Meta's Muse Spark Open scores 39 vs 48 for closed Spark 1.3 on AA Intelligence Index — EAccelerate_42 · 2026-10-16
- LLM spots useless 95GB-per-frame memcpy in ffmpeg dblur, cutting runtime from 2452ms to 49ms — IgorCarron · 2026-10-16
- DeepSeek-V4-Flash on 4x Radeon AI PRO R9700: the 'refused' 4-card config hits ~79 tok/s — brainchillzZ · 2026-10-16