Local NCM benchmarks: 4x faster for one request, 4-8x slower at 64 concurrent
Prompt Engineering · youtube · 2026-10-28
The Prompt Engineering YouTube channel independently tested Stanford Hazy Research's Next-Concept Modeling (NCM), which drafts an answer as a rough sketch and writes exact words in a few passes instead of one pass per token. The paper claims 31x speedup and 86% on hard Sudoku.
Local results on a DGX Spark were more sobering:
- Up to 4x faster than a same-size token-by-token LLM for a single request.
- At 64 concurrent requests, NCM was 4-8x slower — the advantage flips under throughput.
- The author trained his own diffusion Sudoku model; adding most-confident-first decoding (Kim et al. 2025) tied NCM's 86%, suggesting that strategy, not the architecture, may drive the benchmark gains.
Full chapter breakdown, plus links to the paper (arXiv 2610.14162), code, checkpoints and the Hazy Research blog.
More from Infra
- Row-Bot: a local 27B model runs a shop's morning ops entirely on one PC via MCP — Acceptable-Object390 · 2026-10-28
- Streaming inference stack runs full-precision DeepSeek V4.1 on one 5090 at 1770 tok/s prefill — Front_Eagle739 · 2026-10-28
- Richard Socher's Recursive open-sources Flint: 40B MoE, 2.5B active, Apache 2, runs coding locally on Mac — altryne · 2026-10-28
- Cantor: SK hynix trades at just 3x CY27 EPS as AI cycle proves durable — firstadopter · 2026-10-28
- vLLM Semantic Router and AMD release Vela 3.0: 380M model beats GPT-5-mini at zero-shot — vllm_project · 2026-10-28
- Wall Street gripped by new anxiety as it bankrolls AI's vast data center buildout — shiringhaffary · 2026-10-28