BigMoeOnEdge Runs MoE Models Bigger Than Your RAM, Auto-Placing Weights Across VRAM/RAM/Flash
dai_app · reddit · 2026-11-08
The author open-sourced BigMoeOnEdge, built on llama.cpp, which runs MoE models larger than a device's memory by automatically placing each weight on VRAM, RAM, or flash to maximize prefill and decode speed.
- Idea: MoE activates few experts per token, so not all weights need to be resident—hot weights go to the fastest memory, rare experts stream from flash
- Auto planning (--auto): measures free RAM/VRAM, disk speed, and per-device timing on a real layer, plus model size and routing, then decides weight placement, which device handles prefill vs decode, cache sizes, and threads
- Compatibility: any MoE GGUF llama.cpp loads, any quantization, no per-model code; Android/iOS/macOS/Windows/Linux with CPU/CUDA/ROCm/Vulkan/Metal/OpenCL/Hexagon NPU backends
- Benchmarks: 12GB Android phone runs 284B DeepSeek V4 Flash at 1 tok/s; 16GB Apple Silicon laptop runs Qwen3.6-35B-A3B at 18-24 tok/s; 24GB+96GB desktop hits 70 tok/s on a 125B model
- vs vanilla llama.cpp: DeepSeek V4.1 Flash on RTX PRO 6000 96GB improves from 2.8 to 6.2-6.6 tok/s
A meaningful advance for on-device LLM inference, from phones to datacenter GPUs.
More from Infra
- Musk mocks $150k Mac Studio local LLM setup that's worse than a $200/mo subscription — elonmusk · 2026-11-08
- Bank dev tests 1.7B local model offline: all traffic stayed on localhost, emails in 2.5s — Ok_Resolve4741 · 2026-11-08
- Musk endorses orbital AI compute pitch: 100 GW/year could add $1T annual revenue to SpaceX — elonmusk · 2026-11-08
- BF16 GPU TFLOPs comparison table proves its value over time, says HF dev Zach Mueller — TheZachMueller · 2026-11-08
- Starlink adds 1.6M consumer subscribers in a quarter, hits 13.6M worldwide — XFreeze · 2026-11-08
- Leeds AI datacentre row: water use equals just 5 car washes, 1,000+ jobs at stake — danbri · 2026-11-08