Running a 27B Qwen model on RTX 3060: full llama.cpp config hits 10-20 tok/s

SummarizedAnu · reddit · 2026-09-10

A Reddit user shares a complete, working llama.cpp setup for running a Qwen3.8-27B IQ3XXS quant (GSQ-RCO-MTP GGUF) on a 12GB RTX 3060:

• Performance: 10-20 tok/s generation, 334 tok/s prompt processing, with reasoning off and MTP speculative decoding (draft acceptance 0.41); the full model plus 64K KV cache still leaves 1GB VRAM free.
• Key flags: q40 KV cache quantization, --override-tensor pinning layers 10-16 to GPU, --spec-type draft-mtp, flash attention, and more — the full command line is posted.
• Links to a community guide for RTX 3060 deployment and offers troubleshooting help in the thread.

A concrete reference for anyone running 27B-class models locally on consumer GPUs.

Original post →

More from Infra

• Day 249 of GPU Programming: Tracking Cerebras From CS-1 to WSE-3 Turbo-Powered CS-4 — blaizedsouza · 2026-09-10
• GPT-6 Astra pre-training cost estimated at $432M, full model $1-2B — scaling01 · 2026-09-10
• Gensyn builds IR3DE-AXL, a decentralized collective inference network with no central gateway — benfielding · 2026-09-10
• Sam Altman: average person may burn 500B tokens a month within six years — rohanpaul_ai · 2026-09-10
• AI server demand is splitting into three ownership-based markets; ~1-1.5M on-prem servers need refresh — BenBajarin · 2026-09-10
• Marvell Ramps Supply Chain for AI Scale, Analyst Flags Substrates as the Bottleneck Few Can Master — BenBajarin · 2026-09-10

Infra channel →