Transformers now runs llama.cpp GGUF quants via ggml kernels, faster local inference on Mac
LysandreJik · x · 2026-09-22
Hugging Face announced that transformers now supports running GGUF models efficiently, letting users load the millions-downloaded llama.cpp quantized checkpoints through the familiar frompretrained API.
- Compatibility: Pick a GGUF sized for your laptop's memory from the Hub, with ready-made checkpoints from ggml-org, Unsloth, LM Studio Community, and bartowski in various quantizations.
- Performance: To get close to llama.cpp speed, the team reuses its underlying ggml kernels via the kernels library and cuts overhead in generate; the initial focus is local inference on Apple Silicon.
- Context: GGUF powers local tools like Ollama, LM Studio, and Jan. The post cites Julien Chaumond's demo running Qwen3.6 27B via llama.cpp inside the Pi coding agent on a MacBook Pro, described as close to latest Opus.
More from Infra
- Cerebras strategy chief: safety testing is becoming a growth driver for compute providers — RihardJarc · 2026-09-22
- Un-fusing a realtime voice stack (STT → LLM → TTS) cut costs 14x — and the real win was text-level guardrails — Cloudsurfer_90 · 2026-09-22
- Jensen Huang: A Gigawatt-Scale AI Factory Costs $50-60B, Architecture Must Stay Fungible — nvidia · 2026-09-22
- FreeToken fork adds DeepSeek-V4.1, vision and speculative decoding, with 2x3090 benchmarks — ApeGrower · 2026-09-22
- Bloomberg calls Nvidia's low P/E a warning sign; investor argues it makes the stock more attractive — firstadopter · 2026-09-22
- Eugene Plexus open-source alpha: a browser control plane for multi-machine local inference — ScreamingAmish · 2026-09-22