Transformers now natively supports GGUF, beating llama.cpp on one M2 Max benchmark
Disastrous-Work-1632 · reddit · 2026-09-23
Aritra from Hugging Face announced that transformers now natively supports GGUF (llama.cpp-style quants). After loading, you get the normal Transformers APIs, making debugging, evaluation and custom generation much easier, and quantized models fit on laptops.
Usage:
python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.frompretrained("unsloth/Qwen3.5-4B-GGUF", gguffile="Qwen3.5-4B-Q4KM.gguf")
On supported Apple Silicon setups the library reuses ggml kernels and runs directly from packed quantized weights. Benchmarks on an M2 Max with Qwen checkpoints:
- Qwen3.5-4B Q4KM: 70.4 tok/s vs 71.8 with llama.cpp
- Qwen3.8-27B UD-Q4KM: 15.9 tok/s, beating llama.cpp's 13.4
- Qwen3.5-35B-A3B UD-IQ4XS: 60.2 vs 61.3 tok/s
The team stresses this isn't meant to replace llama.cpp — for maximum local inference speed llama.cpp is still the better choice. The point is flexibility: use the same GGUF models inside the PyTorch ecosystem.
Related event: Transformers Natively Runs GGUF, Matching llama.cpp Performance(8 posts)→
More from Infra
- French netizen proposes datacenters pay local residents compensation — clemnt · 2026-11-11
- GPU Rental Prices Hit Record Highs: B200 at $6.97/hr, H100 at $3.06 — inductionheads · 2026-11-11
- A Datacenter in Every Office: Building Work-Safe Local AI Racks — dee_hw · 2026-11-11
- "Compute is about power": the AI race's real bottleneck is energy, not chips — _sholtodouglas · 2026-11-11
- CoreWeave CEO says he canceled his 10b5-1 plan, halting insider selling — firstadopter · 2026-11-11
- UK chip start-up Fractile hits $7bn valuation, per FT — evisoft · 2026-11-11