Zero-Cost 25 tok/s 7B MoE Inference on CPU

Annual_Manner_5901 · reddit · 2026-07-18

The author showcased Reame, a CPU-first inference server built on llama.cpp, and provided a live demo accessible directly via a web browser.

Core Performance

Running OLMoE 7B (1B active MoE, Q4) on an Oracle Always-Free ARM 4 vCPU machine achieves speeds around 25 tok/s.
On the same machine, a dense 9B model only manages about 4.6 tok/s.
The author notes that MoE's advantage lies in delivering 7B-class output while only activating 1B parameters per token, making it highly cost-effective on CPUs.

Zero-Cost Deployment Stack

System Design

The author emphasizes a core principle: never compute the same thing twice on a CPU.

Prompt prefixes and historical generation results are snapshotted to disk and reused across requests and restarts.
A Redis-compatible daemon (ARCA) was built to share exact response caches and generation corpora across multiple machines.
The target use case isn't as a ChatGPT replacement, but rather for narrow tasks like extraction, classification, and batch processing.

The post concludes by noting that this is a free, single-machine, single-request demo, and OLMoE is currently optimized primarily for English.

More from Venture