Running 2.4T Qwen3.8 Model on RTX 5090 + 5060 Ti: 0.8 tok/s Tested

mossy_troll_84 · reddit · 2026-08-14

A developer successfully ran the massive 2.4T parameter Qwen3.8-2.4T-A95B model on consumer hardware, sharing detailed benchmarks and configuration insights.

Hardware & Environment

Performance

In a controlled 32-token generation test, the setup achieved a generation speed of 0.80 tok/s. Enabling native MTP (speculative decoding) resulted in a 3.64% throughput increase and a 2.74% reduction in total wall time, with an MTP acceptance rate of 90.48%.

Configuration

The author shared optimal llama.cpp parameters, including a tensor split of 4,1 and offloading 91 MOE experts to the CPU. To prevent MTP from running out of VRAM, block 92 expert tensors were forced to remain on the CPU.

Original post →

More from Infra

Infra channel →