Transformers now natively supports GGUF, beating llama.cpp on one M2 Max benchmark
Disastrous-Work-1632 · reddit · 2026-09-23
Aritra from Hugging Face announced that transformers now natively supports GGUF (llama.cpp-style quants). After loading, you get the normal Transformers APIs, making debugging, evaluation and custom generation much easier, and quantized models fit on laptops.
Usage:
python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.frompretrained("unsloth/Qwen3.5-4B-GGUF", gguffile="Qwen3.5-4B-Q4KM.gguf")
On supported Apple Silicon setups the library reuses ggml kernels and runs directly from packed quantized weights. Benchmarks on an M2 Max with Qwen checkpoints:
- Qwen3.5-4B Q4KM: 70.4 tok/s vs 71.8 with llama.cpp
- Qwen3.8-27B UD-Q4KM: 15.9 tok/s, beating llama.cpp's 13.4
- Qwen3.5-35B-A3B UD-IQ4XS: 60.2 vs 61.3 tok/s
The team stresses this isn't meant to replace llama.cpp — for maximum local inference speed llama.cpp is still the better choice. The point is flexibility: use the same GGUF models inside the PyTorch ecosystem.
Related event: Hugging Face Transformers Now Runs GGUF at llama.cpp Speed via GGML Kernels(8 posts)→
More from Infra
- Programmable Si photonic circuit hits 29 fW static power per pi phase shift — jwt0625 · 2026-09-23
- Clean pre-dicing photonic wafer shows low-power InGaAsP-on-silicon phase modulators — jwt0625 · 2026-09-23
- Gas turbine orders booked to 2030, prices up 195% as AI power gap widens — FinanceYF5 · 2026-09-23
- US datacenters need 18GW in 2026 but face ~5GW shortfall after fixes — FinanceYF5 · 2026-09-23
- Morgan Stanley: US datacenter power gap equals six New York Cities — FinanceYF5 · 2026-09-23
- Sail Research doubles Gemma 4 31B prefill MFU to 63% on TPU v6e — xennygrimmato_ · 2026-09-23