Zero-Cost 25 tok/s 7B MoE Inference on CPU
Annual_Manner_5901 · reddit · 2026-07-18
The author showcased Reame, a CPU-first inference server built on llama.cpp, and provided a live demo accessible directly via a web browser.
Core Performance
- Running OLMoE 7B (1B active MoE, Q4) on an Oracle Always-Free ARM 4 vCPU machine achieves speeds around 25 tok/s.
- On the same machine, a dense 9B model only manages about 4.6 tok/s.
- The author notes that MoE's advantage lies in delivering 7B-class output while only activating 1B parameters per token, making it highly cost-effective on CPUs.
Zero-Cost Deployment Stack
- Compute: Oracle Always-Free ARM box
- Public HTTPS: Free Tailscale Funnel
- Frontend: GitHub Pages static site
- Total hosting cost: €0
System Design
The author emphasizes a core principle: never compute the same thing twice on a CPU.
- Prompt prefixes and historical generation results are snapshotted to disk and reused across requests and restarts.
- A Redis-compatible daemon (ARCA) was built to share exact response caches and generation corpora across multiple machines.
- The target use case isn't as a ChatGPT replacement, but rather for narrow tasks like extraction, classification, and batch processing.
The post concludes by noting that this is a free, single-machine, single-request demo, and OLMoE is currently optimized primarily for English.
More from Venture
- Beth Kindig says her Q3 AI stocks report flags 5 winners up more than 100% YTD — Beth_Kindig · 2026-07-22
- AI’s economic forecasts are split by nearly a quadrillion dollars by 2035 — bittingthembits · 2026-07-22
- LLM Agent Trader pairs backtesting with model-generated trading signals — tom_doerr · 2026-07-22
- OpenRouter spend data suggests frontier models still capture most paying usage — thedealdirector · 2026-07-21
- AI is collapsing startup and operating costs as U.S. business formation hits a record — sanjaykalra · 2026-07-21
- Soxton lands on investors’ 2026 legal-startup watchlist with $2.5M raised — ashleymayer · 2026-07-21