Grandma GPUs reborn: 2x Tesla P40 hits 48 tok/s on Qwen 27B via F16 cache + MTP
Jumpy-Operation-4615 · reddit · 2026-09-06
A self-described non-coder used Codex to tune a $1,100 self-built 2x Tesla P40 cluster, pushing Qwen 27B Q8 inference from 15 tok/s to up to 48 tok/s short-context decode and 440 tok/s prefill.
Key findings:
- F16 KV cache over Q8: F16 gives much better MTP speculative-decoding acceptance — the biggest single win, though it requires retuning other params.
- Runs llama.cpp/p40.cpp with locally built NCCL: tensor split 1:1, Flash Attention on, draft-mtp + ngram-simple speculative decoding with draft max 6.
- Single 220K-token context slot; a 63,900-token prefill measured 258 tok/s, and with a changed suffix the LCP prefix cache hit fkeep=1.000, recomputing only 4 tokens.
- Full production command line included, with sampling and reasoning settings.
Takeaway: old datacenter GPUs have plenty of headroom; the bottleneck is configuration, not hardware.
More from Infra
- Meta researcher: horrifying infra, not talent, is why OSS can't catch closed frontier — A_K_Nain · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23
- LatentLoop: asymmetric compute allocation cuts looped-MoE training cost by 19% — pmttyji · 2026-10-23
- Same AI request emits 8.5x more CO2 in Texas than in Sweden — jlundberg · 2026-10-23
- SK Hynix Pulls Plug on Solidigm US IPO as Chip Stocks Slide, Shares Down ~30% This Quarter — rohanpaul_ai · 2026-10-23
- Designer mocks Nvidia: buy idle GPUs or lose access to new ones—definitely not a bubble — michalmalewicz · 2026-10-23