FreeToken fork adds DeepSeek-V4.1, vision and speculative decoding, with 2x3090 benchmarks
ApeGrower · reddit · 2026-09-22
The author shares an Apache-2.0 fork of FreeToken, an edge-native MoE serving engine that offloads experts to host RAM/NVMe and co-executes on CPU+GPU to run big MoE models on consumer hardware.
Additions over upstream:
- DeepSeek-V4.1-Flash support (mHC, CSA2 sparse attention, lightning indexer, Engram n-gram memory, DSpark draft)
- Vision for Qwen3.8-Flash-Next with end-to-end OpenAI imageurl input
- Speculative decoding: MTP draft head, rejection sampling, commit-the-accepted-prefix, optional round chaining
- FTW format now carries side tables (PLE/Engram) and the MTP head, plus TP slicing; tensor-parallel loading and optional fp8 all-reduce
On 2x RTX 3090 (TP=2) + EPYC 7203P, 256-token benchmarks show greedy without MTP at 48.0 tok/s; with MTP it drops to 21.5 tok/s (56% accept rate) under CUDA graphs — a net loss on this config. But under matched eager conditions MTP was 1.5x plain (17.2 vs 11.4); the author is optimizing the per-batch verify graph and isn't claiming a win yet. Also note DeepSeek-V4.1's Engram table is stored on disk by default.
More from Infra
- Rat-brain AI models land on AWS as biological computing goes mainstream — nordicinst · 2026-09-22
- Cerebras strategy chief: safety testing is becoming a growth driver for compute providers — RihardJarc · 2026-09-22
- Un-fusing a realtime voice stack (STT → LLM → TTS) cut costs 14x — and the real win was text-level guardrails — Cloudsurfer_90 · 2026-09-22
- Jensen Huang: A Gigawatt-Scale AI Factory Costs $50-60B, Architecture Must Stay Fungible — nvidia · 2026-09-22
- Bloomberg calls Nvidia's low P/E a warning sign; investor argues it makes the stock more attractive — firstadopter · 2026-09-22
- CNCC Forum on On-Device LLMs: Agents, Real-Time Multimodal and AI-Forged Chip Stacks — 面壁智能 · 2026-09-22