Running 276B MoE Models on <10GB RAM: Mference Hits ~3 tok/s
Blahblahblakha · reddit · 2026-08-06
A developer open-sourced Mference (built on Swift + Metal), successfully running the 276B parameter MoE model Inkling-Small (12B active) on consumer hardware with less than 10GB of memory.
- Performance: On a 24GB M5 chip, the resident set was only 3.4GB with decode speeds of 2.56-2.93 tok/s. On a 256GB M3 Ultra, it hit 5.31-6.92 tok/s.
- Limitations: Long prompt prefill is extremely slow (almost 9 minutes to first token for 2,800 tokens), and it's text-only for now.
- Features: Not a wrapper around MLX or llama.cpp. Provides a Mac app, CLI, and an OpenAI-compatible server. Currently supports low-resource deployment for four major model families.
More from Infra
- Google Cloud's Filestore Migrates to Colossus, Decoupling Capacity from IOPS — rseroter · 2026-08-06
- Testing 8x DGX Spark Nodes in Open World Multi-Agent Setup — NVIDIAAI · 2026-08-06
- Open-Source Benchmarks: RTX 5090 LLM Quants and 8GB VRAM Agentic Scores — max_paperclips · 2026-08-06
- NVIDIA Discusses Building Secure Enterprise AI with Proprietary Data — nvidia · 2026-08-06
- Chorus: Open-Source Pre-trained Model Library Enables Fast CPU Inference Without GPUs — jmschreiber91 · 2026-08-06
- Running DeepSeek V4 Locally on Spark Hardware Hits ~95 tok/s — Rasmic · 2026-08-06