TensorFold 1.0: open-source local inference hits 874 tok/s on RTX PRO 6000
Jasonio · x · 2026-11-03
TensorFold for Mac 1.0.0 marks the first stable release of this open-source local inference project, with Linux and Windows builds coming soon. It runs open models fast on GPUs you own via native Metal/CUDA, supporting Qwen, Nemotron, GLM, and LFM, with an OpenAI-compatible API.
Published measurements: 874 tok/s decode with Nemotron 3.5 Lightning on an RTX PRO 6000 Blackwell (speculative decoding + MLX 4-bit); 90–110ms prompt reuse on NVIDIA GB10; 199 tok/s aggregate across eight concurrent streams for GLM-5.3-Flash on two M5 Ultras. Features include parallel decode, prompt reuse, and models spanning connected Macs, plus StarLight for enterprise fleet management.
Version 1.0 adds a compact Ask bar with window selection and local text extraction, background menu-bar mode, connection health/inbox improvements, and engine updates without replacing the app.
More from Infra
- rauchg: stateless sandboxes are the wrong benchmark as Vercel Sandbox gets 2.4x faster — cramforce · 2026-11-03
- Microsoft Cloud hits $63B quarter at 30% growth as AI buildout keeps scaling — thedealdirector · 2026-11-03
- Open-source sparse attention: Qwen 27B with no accuracy loss, 10M-token context — Immediate_Sky_6566 · 2026-11-03
- A year of renting an H100 costs $37.6K as warnings grow about tightening compute supply — teortaxesTex · 2026-11-03
- Mistral Large 4 (1T params) runs locally on two M3 Ultras at 20 tok/s via MLX-VLM — andrejusb · 2026-11-03
- AEHR jumps 16% and LSCC 9% after hours as both chip stocks report earnings — BenBajarin · 2026-11-03