Heterogeneous GPU benchmark of Qwen3.8-27B: eGPU layer-split and MTP acceleration analyzed
CoffeeToCode99 · reddit · 2026-08-30
The author benchmarked the Qwen3.8-27B model on a heterogeneous setup featuring an RTX 5070 Ti laptop GPU and an RTX 5060 Ti eGPU.
Setup:
- Used llama.cpp's layer-split mode with a 38/62 VRAM ratio.
- Compared Q4, Q5, and Q6 quantization levels.
Findings:
- Quantization Trade-off: Moving from Q4 to Q6 costs only 18% generation speed for significant quality gains, with diminishing returns kicking in well before Q8.
- MTP Acceleration: Enabling the built-in MTP (multi-token prediction) speculative decoding boosted Q5KXL speed from 19 tok/s to 32-38 tok/s—nearly double—without a separate draft model.
- Recommendation: Q6 offers the best value for accuracy-critical tasks, while Q4 is sufficient for high-throughput, low-stakes generation.
More from Infra
- Magnitude open-source inference engine beats Ollama by over 2x with auto-tuned kernels — nickbaumann_ · 2026-10-15
- Nvidia to announce $28B in AI investments in Europe this week, report says — georgemillo · 2026-10-15
- A $10,811 Cloudflare Bill From an Unused AI-Written Test Project: Refunds Aren't a Safety Net — sujingshen · 2026-10-15
- Retraining-Free Method Turns Transformers Into Elastic Models Adapting to Any Compute Budget — VISLab-Amsterdam · 2026-10-15
- vLLM's MTP3 validation: AIME25 up to 0.9667, prefill kernels up to 84% lower latency — vllm_project · 2026-10-15
- Open-weights s1-llm-auto-router picks the right LLM in 26ms on a CPU — airesearch12 · 2026-10-15