5 Inference Engines Powering Modern AI: vLLM, SGLang, TensorRT-LLM, llama.cpp, MLX
goyalshaliniuk · x · 2026-11-02
A thread breaks down 5 LLM inference engines and when to use each: vLLM (high-throughput serving with PagedAttention and continuous batching), SGLang (RadixAttention and prefix caching, ideal for agentic workflows and long prompts), NVIDIA TensorRT-LLM (optimized kernels and quantization for max GPU performance), llama.cpp (GGUF quantized models on CPU/GPU/Apple Silicon for local and edge use), and Apple's MLX/mlx-lm (unified-memory optimization for Mac). Choosing the right engine meaningfully affects speed, cost, and scalability.
Related event: A Look at 5 LLM Inference Engines: vLLM and SGLang Lead the Pack(3 posts)→
More from Infra
- Blackwell servers now fetch $2.4M each in China, prices tripled in four months — teortaxesTex · 2026-11-02
- Apple's MLX and mlx-lm: Local LLM Inference on Apple Silicon — goyalshaliniuk · 2026-11-02
- llama.cpp: Running GGUF-Quantized LLMs on CPUs, GPUs and Apple Silicon — goyalshaliniuk · 2026-11-02
- Particle-1 trails Opus 5.5 by 3.4 points on memory eval but costs 1/23 per correct answer — garrytan · 2026-11-02
- Nvidia's 27-year market cap history: from $800M to $5T, with 70% crashes along the way — mitchdeg · 2026-11-02
- 38 hours on one RTX 3060: SLQ quantization edges out AtomicChat GGUF at same size — maximelabonne · 2026-11-02