594B model now runs on two DGX Sparks at 37.9 tok/s; drafter boosts 27B to 78 tok/s
AIFlow_ML · x · 2026-11-18
Local AI Daily (18 Nov 2026) rounds up notable local-inference results:
- Inkling 2 Small (Thinking Machines, 594B total/17B active, Apache-2.0) now runs on just two DGX Sparks (down from four) via TensorFold plus a drafter and a 2.2-bit 170GB pack — 37.9 tok/s at 128K context.
- A drafter setup took a 27B model from 11.9 to 78.2 tok/s on a single Spark.
- One builder returned his 512GB Mac Studio: for DeepSeek V4.1 Flash at similar cost, 4× GX10 delivers 330-847 total tok/s across 16 sessions vs 137-148 on the Mac — though the Mac wins on idle power (5W vs 215W) and cold start (36s vs 150s).
- Also noted: llama.cpp PR #32407 rotates K/V read order for better L2 cache hit rates.
Every number is labelled as measured-by-source or vendor-claim.
More from Infra
- Why Cheaper Tokens Can Still Mean a Bigger AI Bill: The Agent Call-Stacking Problem — samcharrington · 2026-11-18
- New Blog Post Explains TorchTPUs and How They Differ From Prior TPU Approaches — algo_diver · 2026-11-18
- Nvidia reportedly delayed its 10-Q filing by weeks, an extraordinary rarity — GaryMarcus · 2026-11-18
- TSMC starts checking customer balance sheets, demands cash to lock in capacity — SumitGup · 2026-11-18
- Cloudflare Workers static assets are free: 25MB per file, 20k files per version — ezshine · 2026-11-18
- Inference for fine-tuned models is underexplored: LoRA vs multi-tenant KV debate — JoshPurtell · 2026-11-18