Self-hosted Qwen 27B on RunPod hits only 17 tok/s generation, making OpenAI API hard to beat on cost per job
yeah280 · reddit · 2026-09-11
Testing Qwen3.8-27B GGUF Q80 on RunPod Serverless ($1.22/hr, 48GB VRAM) against OpenAI for 47k-token transcript analysis: prompt processing is fast (1073 tok/s) but generation crawls at 17.2 tok/s, meaning 8.7 minutes for 9k output tokens and $0.20-0.30 per video. OpenAI cost $5 per 1.6M tokens. llama.cpp ignores NextN/MTP speculative-decoding tensors. Poster weighs quantization, vLLM, better GPUs, and argues cost-per-completed-job is the right metric.
More from Infra
- SemiAnalysis: OpenAI reportedly negotiating ~1GW TPU commitment while seeking sites — dylan522p · 2026-10-28
- I tested 11 video upscaling methods for AI video: local, Topaz and cloud compared — lopas99 · 2026-10-28
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- SK Hynix says memory demand will outpace supply through 2030 — Chemical_Hawk_6307 · 2026-10-28