Running Qwen 3.8 27B locally for 8+ hours: 131M tokens, ~$650 saved vs. Opus API
illgettheownerforyou · reddit · 2026-08-18
A developer ran DeepSeek Harness on a Windows client, with inference over LAN on a separate RTX Pro 6000 box via NInfer, using Qwen3.8-27B (NInfer groupwise-int mixed Q4/Q5/Q6 quant, 262K context). All shell commands and file operations stayed on the client PC.
Hard numbers from the 8+ hour run:
- 966 model calls, 130.2M task input tokens and 812.5K output tokens (131.2M/853.3K including compaction)
- 972 model-facing tool calls and 1,421 local tool operations, only 30 failures (2.11% error rate)
- 31 automatic compaction attempts; median root request 136.6K tokens, p95 205.9K, max 231.2K
- 104.83 output tok/s weighted decode, zero model-generation failures
The bottleneck wasn't generation but prefill and queueing: median root TTFT was 0.8s but p95 hit 136s, with two agents contending for one endpoint; client-side prompt throughput was roughly 12.4K tok/s for root requests.
Priced at current API rates (no cache discounts), the workload would cost $18.61 on DeepSeek V4 Flash, $27.26 on GPT-5.6 Luna, $270.93 on Claude Sonnet 5, and $677.32 on Claude Opus 4.6. The takeaway: long-horizon agent workloads are dominated by repeated six-figure prompts and prefill, not decode. Next up: trying NInfer's NVFP4 profile.
More from coding & agent
- Native vLLM on Windows 11 for AMD RX 6000: One-Click Installer — Dizzy_Counter2481 · 2026-08-18
- Healthcare Agent Case Study: Map Data Before Touching Models — alex_verem · 2026-08-18
- Stop Writing CLAUDE.md Like a Wiki: Config Guide — socialwithaayan · 2026-08-18
- Best platform for training LoRA on low VRAM GPUs — StoicSage09 · 2026-08-18
- User asks about the meaning of subagent icons in Codex — lucasmeijer · 2026-08-18
- Using Obsidian as a Personal Agent Knowledge Base — hugobowne · 2026-08-18