BigMoeOnEdge runs 35B MoE at 35 tok/s on 8GB cards, beating Strata and colibri
dai_app · reddit · 2026-11-16
BigMoeOnEdge (863 stars, open source) runs any MoE GGUF llama.cpp can load on low-end hardware—NVIDIA/AMD/Intel/Mac/phone/CPU, all major OSes—lossless and with --auto setup. Benchmarks: Qwen3.6-35B hits 35 tok/s on an 8GB card (colibri 23, Strata fails at 16k context); 5s first token on long prompts vs 14s/2min; 20 tok/s on a 16GB MacBook; 10 tok/s on a 125B model. Strata still wins 1.3x only on the two cards it's kernel-tuned for. Repo and logs public.
More from Infra
- Agent-run PageRank analysis of ~2,000 stories traces how NYT narrative wiped $1T off Nvidia — dotey · 2026-11-16
- Go.AI reportedly paid $5M for its domain; on-prem AI claims 99.5% accuracy, 900K users — IgorGabrielan · 2026-11-16
- Commerce Secretary Lutnick Backs Narrower AI Chip Controls Amid Conflict-of-Interest Scrutiny — nordicinst · 2026-11-16
- Per-user agent cloud VMs: how do you handle the wake-up wait vs cost tradeoff? — SnooPuppers9163 · 2026-11-16
- OpenAI made ~$13B last year but committed ~$1.15T to compute — Qweskj · 2026-11-16
- Pirateface launches P2P network for sharing open models without centralized hosting — alohahaho · 2026-11-16