Hands-on: Magnitude claims 2x over llama.cpp, but disappoints on mid/low-end hardware
OsmanthusBloom · reddit · 2026-10-30
An r/LocalLLaMA user tested Magnitude, an open-source local inference app claiming 2x speed over Ollama/llama.cpp by auto-tuning kernels per hardware, on an RTX 3060 Mobile laptop and a Raspberry Pi 5.
Key findings:
- Slow startup and heavy CPU: a calibration phase runs on first launch, with little feedback on progress.
- Limited model support: only a few dozen models (Qwen, Gemma, LFM, MiniCPM families); it reads GGUF caches from Ollama/llama.cpp but rejects many of them.
- Storage quirk: models go to /.magnitude/models/; the author symlinked to a faster data partition since /home was encrypted (laptop) or on SD card (RPi).
- No partial CPU offload: larger MoEs like Qwen3.6-35B-A3B are marked "doesn't fit" on the 6GB-VRAM laptop, unlike llama.cpp.
- VRAM overestimated: it reported 5.3GB for LFM3-1.7B Q8 while 4.2GB was actually reserved, meaning some "too large" models would fit.
- Architecture resembles Ollama: background server, GUI/CLI, chat UI, and an OpenAI-style API at localhost:10100/inference/v1.
- Benchmarks vs llama.cpp used llama-benchy (PP 2048 / TG 512) on the same LFM3-1.7B Q8 GGUF.
More from Infra
- Altman once said open-sourcing was to offload OpenAI's servers—now he calls local inference wasteful — TheZachMueller · 2026-10-30
- Altman says local AI wastes scarce memory; critics say he misses the point — kimmonismus · 2026-10-30
- AI workloads may accelerate the shift from HDDs to all-flash data centers — IBDinvestors · 2026-10-30
- Intel data center revenue jumps 78% to $7.3bn, strongest growth in 15 years — BenBajarin · 2026-10-30
- Backprop-free pretraining hits 4,000x speedup in 10 days, now 1.6 seconds per run — SonglinYang4 · 2026-10-30
- OpenRouter's new router: ~20% faster TTFT, ~60% fewer rate-limit errors for open-weight models — AccBalanced · 2026-10-30