Full llama-server config for local Qwen3.6-35B-A3B with MTP speculative decoding
haydendevs · x · 2026-09-22
The author shares a complete llama-server launch command for running Qwen3.6-35B-A3B-MTP-UD-Q4KXL locally:
- 131072 context, 32768 prediction tokens, -fitt 1536
- Flash Attention on, K/V and draft caches quantized to q80 to save VRAM
- --spec-type draft-mtp enables MTP speculative decoding with --spec-draft-n-max 2
- --ctxcp 64, --no-mmap, --mlock, --no-warmup for memory behavior tuning
- Chat template kwargs preservethinking:true to keep reasoning traces
- Sampling: temp 0.6, top-p 0.95, top-k 20, no repeat penalty
Serving on 0.0.0.0:8080 as an OpenAI-compatible API — a ready-to-copy recipe for local MoE deployment.
Related event: Qwen3.6-35B Runs Smoothly on a Single RTX 4070(3 posts)→
More from Infra
- Cloudflare Talk: Swapping Some Agent LLM Calls for Fast Models Like Jev Gives Big Speedups — michellechen · 2026-11-12
- Cloudflare Trains Its Own Clef and Clairic Models to Complete Its Agent Stack — michellechen · 2026-11-12
- AWS open-sources 23 production-ready GenAI assets, cutting build time from months to weeks — AWS ML Blog · 2026-11-12
- Sixth Street CEO: ~$200B of Chips Sitting in Warehouses as Data Centers Lag Behind — SumitGup · 2026-11-12
- AWS CEO Matt Garman: Space Data Centers Are 10-20 Years Away, Likely Not Economically Feasible — rohanpaul_ai · 2026-11-12
- CoreWeave Database Goes GA: Managed PostgreSQL Running Inside Your CKS Cluster Next to GPUs — _ScottCondron · 2026-11-12