Janas streams MoE experts from SSD: Qwen3-Next-80B at 37 tok/s on a 32GB laptop vs 2.8 for llama.cpp
petrusenko_max · x · 2026-10-14
Janas, a pure C inference engine, streams MoE experts from SSD to run models larger than RAM.
- Benchmark: Qwen3-Next-80B (48 GB of weights) runs at 37 tok/s on a 32 GB laptop — 24 tok/s without speculative drafts — versus 2.8 tok/s for llama.cpp.
- It outperforms llama.cpp on all 13 Q4KM models compared.
Open-sourced at prabanta-dev/janas (294 stars), it's an aggressive I/O-optimization play for local inference.
More from Infra
- Fluidstack opens 200+ roles as it builds toward tens of gigawatts for leading AI labs — MxMnr · 2026-10-14
- Same model, 9x cost gap: Artificial Analysis publishes cache hit rates per provider — ArtificialAnlys · 2026-10-14
- Apple's Minions turns idle Mac GPUs into a pooled cloud for ML inference — MikePFrank · 2026-10-14
- AnythingLLM ditches Ollama for llama.cpp, with up to 2x faster inference — solyarisoftware · 2026-10-14
- LlamaForge: a browser panel for upstream llama.cpp with local agents that must quote their sources — Sleepybear2611 · 2026-10-14
- CoreWeave's GPU fleet is mostly older boxes, raising questions about depreciation math — ivan_bezdomny · 2026-10-14