FreeToken open-source engine runs 284B DeepSeek locally at 25 tok/s on RTX 5090
aigclink · x · 2026-11-15
FreeToken, an open-source inference engine from Berkeley/MIT researchers, lets consumer PCs run 280B+ MoE models: DeepSeek 284B hits 22–25 tok/s on an RTX 5090, with decode throughput 1.3–2.1x faster than existing local engines.
- Problem: a 290B model needs 145GB even at 4-bit; system-RAM offloading hits memory bandwidth and PCIe limits
- Approach: treat GPU + CPU + RAM + PCIe as one system, exploiting MoE sparse activation to optimize compute placement, transfers, and caching
- Four key techniques:
- Bandwidth-adaptive CPU-GPU co-inference with full-layer double-buffered prefetch during prefill
- Agent-oriented semantic-aware caching (reuse states at semantic boundaries, recompute only affected suffix) plus global LRU expert cache
- Elastic VRAM management between expert cache and KV cache, no reload needed
- OpenAI/Anthropic-compatible API to plug into Claude Code, Codex, OpenCode as a local backend
- Benchmarks: RTX 4060 Laptop: Qwen 35B @ 39.3 tok/s; RTX 5090: DeepSeek 284B @ 22–25 tok/s; RTX PRO 6000: GLM 753B @ 14.9 tok/s; worst TTFT under agent workloads <44s vs 150s+ for baseline
A notable edge-local route beyond quantization/offloading: coordinated use of all hardware for local agent infrastructure.
Related event: FreeToken Open-Source Engine Runs 284B Models on Consumer GPUs(2 posts)→
More from coding & agent
- Design engineering tip: show the first frame before motion content is ready — floguo · 2026-11-15
- Solo lawyer runs an AI-native law firm on a Claude agent team organized like company departments — prasanna_says · 2026-11-15
- Claude builds a colorful 3D city in three.js with zero assets, all code — Eric520CC · 2026-11-15
- fate 2.2.1 ships massive perf gains: up to 343x faster record deletion — cnakazawa · 2026-11-15
- Survey maps how LLM agents recursively self-improve via prompts, memory, skills, harness — 机器之心 · 2026-11-15
- METR open-sources its AI agent evaluation platform for safely testing frontier models — BethMayBarnes · 2026-11-15