TensorFold Runs Local LLMs at 874 tok/s With Native Metal/CUDA Inference

AIFlow_ML · x · 2026-11-17

TensorFold is a local AI inference product (Apache-2.0, OpenAI/Anthropic API compatible) that runs open models like Qwen, Nemotron, GLM and LFM on your own GPUs via native Metal, CUDA, HIP and Vulkan. Named-hardware measurements include 874 tok/s decode on Nemotron 3.5 Lightning with an RTX PRO 6000 Blackwell (drafting + 4-bit MLX checkpoint), 90–110ms prompt reuse on NVIDIA GB10, and 199 tok/s aggregate across 8 concurrent streams for GLM-5.3-Flash on two M5 Ultras. Key techniques: speculative decode (propose-verify-commit), prompt reuse, and models spanning multiple connected machines. It also powers Omarchy's NVIDIA model recipes (DGX Spark, N1x) with a ComfyUI integration in progress.

Original post →

More from Infra

Infra channel →