llama.cpp: Running GGUF-Quantized LLMs on CPUs, GPUs and Apple Silicon
goyalshaliniuk · x · 2026-11-02
Part 4 of an inference-engine series covers llama.cpp, which runs LLMs across CPUs, GPUs, and Apple Silicon, supports GGUF quantized models for memory-efficient inference, and targets local AI, edge devices, and consumer hardware. The MIT-licensed project has 133k stars on GitHub.
Related event: llama.cpp for Edge Devices and TensorRT-LLM for Peak NVIDIA GPU Performance(2 posts)→
More from Infra
- Blackwell servers now fetch $2.4M each in China, prices tripled in four months — teortaxesTex · 2026-11-02
- Apple's MLX and mlx-lm: Local LLM Inference on Apple Silicon — goyalshaliniuk · 2026-11-02
- 5 Inference Engines Powering Modern AI: vLLM, SGLang, TensorRT-LLM, llama.cpp, MLX — goyalshaliniuk · 2026-11-02
- Particle-1 trails Opus 5.5 by 3.4 points on memory eval but costs 1/23 per correct answer — garrytan · 2026-11-02
- Nvidia's 27-year market cap history: from $800M to $5T, with 70% crashes along the way — mitchdeg · 2026-11-02
- 38 hours on one RTX 3060: SLQ quantization edges out AtomicChat GGUF at same size — maximelabonne · 2026-11-02