Local LLM inference is hopelessly fragmented: one user now juggles 5 engines per model
Fancy-Snow7 · reddit · 2026-11-23
A local-LLM user laments inference engine fragmentation: llama.cpp alone now leaves huge performance on the table. He runs 5 engines — Strata (180 t/s on Qwen FN with 16GB VRAM vs 30 t/s on llama.cpp), Basalt (60 t/s on Inkling-2-Mini, 2x+ a fork), an Adaptive KV Streaming fork (collapses from 70 to 20 t/s at long context), beelama, and upstream llama.cpp. Dozens of Strata forks exist; some boost speed 5 t/s while silently worsening KLD by 50%. Root cause: llama.cpp is slow to support new architectures and rejects useful PRs, forcing a new engine install with every model release.
More from Infra
- Apple cuts per-dev daily token budget from $300 to $100; cheaper-model strategies follow — Zachly · 2026-11-23
- Custom sm_60 kernels push Tesla P100 decode from 11.6 to ~52 tok/s, repo open-sourced — Relative-Ingenuity15 · 2026-11-23
- Open-Source Maestro Video Generator Lures Users Away From $40/Month AI Subscriptions — cocktailpeanut · 2026-11-23
- A 9-machine local AI homelab: 5 agents on one dual-3090 llama-server — StandardLovers · 2026-11-23
- Prune, train, upcycle: one model that scales with your RAM — minhnguyen · 2026-11-23
- Koboldcpp ships v1.125.1 release — Fcking_Chuck · 2026-11-23