TensorFold 1.1.0 hits 3,962 tok/s prefill at 205k ctx on M5 Ultra
GabGarrett · x · 2026-10-19
TensorFold 1.1.0 is out with a wave of fixes: Nemotron loading on M1-M4, Super VL tool calls, GLM long-chat caching, streaming delays, external-drive model scans, plus new native CUDA models.
A user benchmarked Qwen3.8-Flash-Next-MLX-6bit-MTP on an M5 Ultra (80-core, 256GB): prefill reaches 3,962 tok/s at 205k context (TTFT 52s) and 4,018 tok/s at 128k, rising to 4,700-4,800 tok/s at short contexts. Decode runs at 170 tok/s for prose and 209 tok/s for structured output, with TTFT of just 62-63ms.
More from Infra
- KV Cache Compression Makes 1M Context Manageable at Under 100GB Per User — khademinori · 2026-10-19
- Elon Musk mocks what it's like trying to rent AI compute today — elonmusk · 2026-10-19
- New blog series part 1: a visual introduction to tensor core arithmetic — akbirthko · 2026-10-19
- Weekly recap: five vendors ship agent controls at five layers, desk-side AI draws most readers — shashib · 2026-10-19
- Hugging Face kicks off Open Source AI Week with ExecuTorch edge-AI hackathon — Thom_Wolf · 2026-10-19
- Local models are now good enough for first-round code review, dev finds while paying frontier API prices — draginol · 2026-10-19