TensorFold Runs Local LLMs at 874 tok/s With Native Metal/CUDA Inference
AIFlow_ML · x · 2026-11-17
TensorFold is a local AI inference product (Apache-2.0, OpenAI/Anthropic API compatible) that runs open models like Qwen, Nemotron, GLM and LFM on your own GPUs via native Metal, CUDA, HIP and Vulkan. Named-hardware measurements include 874 tok/s decode on Nemotron 3.5 Lightning with an RTX PRO 6000 Blackwell (drafting + 4-bit MLX checkpoint), 90–110ms prompt reuse on NVIDIA GB10, and 199 tok/s aggregate across 8 concurrent streams for GLM-5.3-Flash on two M5 Ultras. Key techniques: speculative decode (propose-verify-commit), prompt reuse, and models spanning multiple connected machines. It also powers Omarchy's NVIDIA model recipes (DGX Spark, N1x) with a ComfyUI integration in progress.
More from Infra
- SpaceX uniquely owns the ride to orbit, the global network, and the AI computers at once — ccerrato147 · 2026-11-17
- Nvidia earnings could be good for NVDA but bad for the rest of the AI trade — pdamodaran · 2026-11-17
- AMD's Dan McNamara: billions of AI agents will reshape CPU demand, Venice chips now shipping — firstadopter · 2026-11-17
- User Slams Nvidia for Capping VRAM at 128GB While Claiming to Push Local AI — AIFlow_ML · 2026-11-17
- Commerce Dept and DOJ issue joint statement on Megaspeed investigation — ShakeelHashim · 2026-11-17
- Cloudflare open-sources Kitesurf, a Chromium-free browser engine for AI agents — shoumikgoswami · 2026-11-17