Running Qwen at 1Q quant: dev shares DeepSeek Harness fix plus Nemotron 3.5 tips
gnukeith · x · 2026-11-05
Dev gnukeith shares tactics for running open models on modest hardware: running Qwen 3.8 Next-Flash at ultra-low (1Q) quant degrades quality badly, but you can offload n-gram lookups to SSD, run MTP speculative decoding on GPU, then recover quality using DeepSeek Harness to find issues and fix them via plugins/steering. He also links several small models that fit fully in GPU VRAM at higher quants. The post cites NVIDIA's Nemotron 3.5 report: Super is a 120B-total/12B-active multimodal MoE Hybrid Mamba-Transformer supporting 1M-token context, while Lightning is a 30B-total/3B-active text model; both ship BF16/NVFP4 checkpoints with native multi-token prediction for fast inference.
Related event: Dev shares low-end tricks for running Qwen3.8-Flash-Next and RLM concept(3 posts)→
More from coding & agent
- Conductor, the AI coding agent startup, is joining Meta's MSL — morgymcg · 2026-11-05
- AI Pays a Tax on Bad Code: Generating Tests for 25K Files Costs 35-45% More Tokens — blaizedsouza · 2026-11-05
- Atomic Chat launches open-source Local AI Server tying models and agents together — SucceededMind · 2026-11-05
- OpenAI Promotes GPT-Live Outbound Telephony API to GA With Any SIP Trunk Support — juberti · 2026-11-05
- Fizgig 9.0.4: train Fibo 2.0 on a 12GB card, SAM 3.1 text-to-mask — shootthesound · 2026-11-05
- Single source of state could help coding agents — Redux comeback? — cnakazawa · 2026-11-05