AnythingLLM ditches Ollama for llama.cpp, with up to 2x faster inference
solyarisoftware · x · 2026-10-14
AnythingLLM Desktop v1.18.0 replaces its bundled Ollama engine with llama.cpp, managed by AMD's open-source Lemonade server.
- Performance: up to 2x faster inference depending on hardware
- Size: runtime install is 13x smaller
- New models: direct llama.cpp integration means Day 0 support as new architectures, quantization formats and GPU kernels land
The team cites two reasons: any layer between the app and llama.cpp lags behind upstream optimization, and Ollama upgrades kept breaking things over the past year. As one of the largest installed bases of Ollama, the switch is a notable shift in the local inference stack landscape.
More from Infra
- Ayar Labs, Wiwynn and GUC demo integrated optical compute system with live telemetry at OCP — bookwormengr · 2026-10-14
- Gavin Baker concedes some AI deals were fake, cites $1.4T commitments talking point — GavinSBaker · 2026-10-14
- Argon pushes Gemma past 120k tokens/sec on a single A100 with vLLM, a 2.9x speedup — osanseviero · 2026-10-14
- Espresso now auto-optimizes Databricks Spark jobs and configs — mattturck · 2026-10-14
- Intel Diamond Rapids motherboard shown off: 16 DDR5 slots, MRDIMM at 12800MT/s — firstadopter · 2026-10-14
- SayGM claims 70% cheaper LLM gateway with $5M ARR and crypto referral rewards — markjeffrey · 2026-10-14