Transformers gains GGUF loading and borrows llama.cpp kernels for it
Serguey Asael Shinder2 min readHugging Face has added support for loading GGUF checkpoints directly in transformers. A quantised file is passed to from_pretrained as gguf_file alongside the Hub model id, and generation runs from there. The announcement is dated 22 September 2026.
The interesting part is not the loader but what sits underneath it. Rather than writing new quantised kernels, the team reuses llama.cpp's own ggml kernels through the kernels library, and uses ggml-org/ggml-attn as the attention implementation when the weights stay packed on Metal. The stated aim is to bring performance close to llama.cpp rather than merely to accept the file format.
The first target is narrow and named: local inference on Apple Silicon, beginning with the Qwen3.5 architecture. It needs transformers from main until the next release, a compatible kernels version, and a PyTorch release that the published ggml-quantization kernel builds support — in practice one of the two latest.
The quantisation arithmetic is published for Unsloth's Qwen3.5-4B: 8.42 GB unquantised at BF16, 3.53 GB at Q6_K, 3.14 GB at Q5_K_M, 2.74 GB at Q4_K_M. Hugging Face suggests starting at Q4_K_M and moving up if memory allows.
What it means
Format compatibility is cheap to announce and expensive to make usable, and the team says so in as many words: compatibility is only useful if the model is pleasant to run. Borrowing the kernels rather than reimplementing them is an admission that the performance work already exists somewhere else and is not worth duplicating.
For anyone maintaining a local inference stack, the practical question is no longer which format to standardise on but which runtime to route through. The same checkpoint now has two plausible consumers on the same machine, and the choice moves to tooling and API familiarity rather than to file layout. Read the quantisation table as guidance and not as a result: the quality tradeoff depends on the model and the task, which is the one thing a size column cannot tell you.