Unsloth quantizes Inkling 2 Small: 594B MoE runs locally at 171.7GB in 1-bit
gnukeith · x · 2026-11-13
Unsloth published a guide to run Thinking Machines Lab's Inkling 2 Small locally — a 594B-parameter MoE with 17B active params, text+image input, 1M-token context, scoring 72.1% on DeepSWE v1.1 and 86.3% on SWEBench Pro v2 (Hard).
Key quantization data: the UD-IQ1S dynamic 1-bit GGUF needs only 171.7GB vs 394GB (NVFP4), retaining 89.7% of top-1% accuracy at 57% smaller. The guide recommends temperature 1.0, top-p 1.0, min-p 0.0 with xhigh reasoning effort, and explains switching reasoning effort (none/minimal/low/high/xhigh) via the GGUF template (--jinja in llama.cpp). Author gnukeith adds that Unsloth remains hugely underrated.
More from Infra
- InferenceForge claims 43% higher LLM inference throughput than best AgentX baseline — kaiwei_chang · 2026-11-14
- Redditor benchmarks a DGX Station: GB300 throughput numbers revealed — CookEasy · 2026-11-14
- Hugging Face launches llama.cpp v1 engine to run AI models locally everywhere — LinkedInNews · 2026-11-14
- China Took 59% of 600,000+ Industrial Robots Installed in 2025; Humanoid Shipments Still Tiny — ingliguori · 2026-11-14
- SGLang joins NVIDIA AI Enterprise with NIM-certified inference microservices — ying11231 · 2026-11-14
- LMSYS and NVIDIA: Vera CPUs cut agentic RL episode time by 11% in Miles tests — ying11231 · 2026-11-14