Dual RTX 5090 + 4070 TiS runs 27B FP8 with 258K context at 81 tok/s
Fz1zz · reddit · 2026-09-27
A Reddit user got Qwen3.8-27B block-FP8 (28.75 GiB) running on a 48GB setup combining an RTX 5090 (32GB) and RTX 4070 Ti Super (16GB) using vLLM 0.30.0 pipeline parallelism: the 4070 hosts layers 0-19 plus the vision encoder, while the 5090 handles layers 20-63, lmhead, and MTP.
Benchmarks:
- 1K in / 512 out, c1: 81.3 tok/s (ITL 36.7ms), 100 tok/s on code
- MTP acceptance 87%, full 262,144 context, fp8 KV
- 32K prefill: 3,160 tok/s; 145K fresh prefill: 68.6s; 258K prefill: 169s with 5090 peaking at 31,472 MiB
The 4070 Ti Super had been collecting dust for two months over PCIe x1 bandwidth concerns, but the tests showed the impact was negligible — useful data for anyone running large models on mixed GPUs.
More from Infra
- India's Sarvam launches Token Factory, hosting top open models on local infra — HarveenChadha · 2026-11-16
- Gemini 3.5 Flash now runs on air-gapped Google Distributed Cloud, offline up to 12 months — shashib · 2026-11-16
- Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8 — This-Purchase-3325 · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16