FreeToken fork adds DeepSeek-V4.1, vision and speculative decoding, with 2x3090 benchmarks
ApeGrower · reddit · 2026-09-22
The author shares an Apache-2.0 fork of FreeToken, an edge-native MoE serving engine that offloads experts to host RAM/NVMe and co-executes on CPU+GPU to run big MoE models on consumer hardware.
Additions over upstream:
- DeepSeek-V4.1-Flash support (mHC, CSA2 sparse attention, lightning indexer, Engram n-gram memory, DSpark draft)
- Vision for Qwen3.8-Flash-Next with end-to-end OpenAI imageurl input
- Speculative decoding: MTP draft head, rejection sampling, commit-the-accepted-prefix, optional round chaining
- FTW format now carries side tables (PLE/Engram) and the MTP head, plus TP slicing; tensor-parallel loading and optional fp8 all-reduce
On 2x RTX 3090 (TP=2) + EPYC 7203P, 256-token benchmarks show greedy without MTP at 48.0 tok/s; with MTP it drops to 21.5 tok/s (56% accept rate) under CUDA graphs — a net loss on this config. But under matched eager conditions MTP was 1.5x plain (17.2 vs 11.4); the author is optimizing the per-batch verify graph and isn't claiming a win yet. Also note DeepSeek-V4.1's Engram table is stored on disk by default.
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12