TensorFold 0.6.1: 36% faster 27B inference, first token drops to 0.1s on Blackwell
EAccelerate_42 · x · 2026-10-02
TensorFold 0.6.1 ships major inference optimizations: 27B models run directly on NVIDIA RTX PRO 6000 Blackwell with NVFP4 4-bit checkpoints; Flash Next on CUDA enables one-pass prompt filling, up to 36% faster throughput with 8 streams, and slowest first-token latency cut from 0.8s to 0.1s. A shared 40,900-token system prompt now answers in under a second instead of 16-56s; forks resume from split points, short requests no longer queue, and images work alongside text. DGX Spark prompts fill 15% faster; Macs get up to 10%. The author notes speculative decoding and TensorFold dramatically boost DGX Spark's value.
More from Infra
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19
- Leak Hints at Classified 1M-GPU Training Cluster in China That Skips HBM — teortaxesTex · 2026-11-19
- Tencent leaned on Japanese neocloud for AI compute; DeepSeek, Zhipu best positioned — teortaxesTex · 2026-11-19
- Open-source prefill engine for GLM-5.3-Flash beats vLLM by 3.2x on 4x B200 — CShorten30 · 2026-11-19