Local LLM inference is hopelessly fragmented: one user now juggles 5 engines per model

Fancy-Snow7 · reddit · 2026-11-23

A local-LLM user laments inference engine fragmentation: llama.cpp alone now leaves huge performance on the table. He runs 5 engines — Strata (180 t/s on Qwen FN with 16GB VRAM vs 30 t/s on llama.cpp), Basalt (60 t/s on Inkling-2-Mini, 2x+ a fork), an Adaptive KV Streaming fork (collapses from 70 to 20 t/s at long context), beelama, and upstream llama.cpp. Dozens of Strata forks exist; some boost speed 5 t/s while silently worsening KLD by 50%. Root cause: llama.cpp is slow to support new architectures and rejects useful PRs, forcing a new engine install with every model release.

Original post →

More from Infra

Infra channel →