Transformers Now Calls GGML Kernels Directly, Matching llama.cpp Performance
LysandreJik · x · 2026-09-22
Hugging Face's Lysandre announced Transformers can now use GGML kernels via the kernels library at the same performance as llama.cpp — previously it only loaded GGUF files by unquantizing them. As proof of concept, only Qwen 3.5 and its MoE counterpart are supported; users can request models via GitHub issues.
More exciting: models outside llama.cpp — even without GGUF files — can leverage GGML kernels. LLM coverage in llama.cpp is already huge, but modalities with heavy pre/post-processing (raw CV, diffusion, TTS) could benefit most. Still PoC, but promising.
More from Infra
- Local H3 video gen: 15s clip in ~5 min at zero cost vs ~$0.60 per generation on platforms — AIandDesign · 2026-09-22
- MotherDuck makes text classification 50x faster at ~1% of LLM cost — josh_wills · 2026-09-22
- Google serves 3.2 quadrillion AI tokens a month — ~12GW, and energy could run out in 3 years — victor_explore · 2026-09-22
- M5 Ultra 256GB early test: Mimo2.6-Flash hits 49 tok/s at 32K context — bakawolf123 · 2026-09-22
- TRL async GRPO adds LoRA sync via storage bucket and proxy, cutting 500-step training to 53 min — SergioPaniego · 2026-09-22
- GGUF models can now run directly in Hugging Face transformers with ggml Metal kernels — pcuenq · 2026-09-22