TL;DR
- Hugging Face Transformers now loads GGUF quantized checkpoints directly through
from_pretrained, no separate llama.cpp build required. - The integration reuses llama.cpp’s own ggml kernels via the
kernelslibrary, aiming to close the speed gap instead of settling for a slow compatibility layer. - Initial optimization targets Apple Silicon and Qwen3.5 architecture models, benchmarked on a MacBook Pro M2 Max with 32 GB unified memory.
- GGUF models can also be served through
transformers servewith an OpenAI-compatible API, so desktop apps like Jan or Pi can talk to them.
What Hugging Face Just Shipped
Transformers can now open GGUF files the same way it opens any other checkpoint. You point from_pretrained at a quantized model, and it loads, no llama.cpp binary sitting off to the side doing the actual work in secret.
The team put it plainly in their announcement: “We’re adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop’s memory through the familiar transformers APIs.” That’s the whole pitch, really. Small files, big library, one workflow.
Under the hood, this isn’t a reimplementation. Hugging Face says the plan is to reuse llama.cpp’s own kernels rather than write a slower Python-native imitation: “To bring performance close to llama.cpp, we’re reusing its underlying ggml kernels through the kernels library, and reducing overhead in generate.” That distinction matters more than it sounds. A compatibility layer that loads GGUF but runs at a crawl would be a novelty. Borrowing the actual kernels is an attempt at something usable.
The numbers give a sense of scale. A Qwen3.5-4B model ships as an 8.42 GB file at BF16 precision. Quantized down, it shrinks to 2.74 GB at Q4_K_M, with intermediate quantization levels landing at 3.53 GB and 3.14 GB along the way. That’s the difference between a model that fits on a laptop and one that doesn’t, and it’s the entire reason GGUF exists in the first place.
On the serving side, transformers serve now exposes GGUF models through an OpenAI-compatible API, which means desktop inference clients like Jan or Pi can point at a Transformers-served model without any translation layer. Initial optimization work has focused on Apple Silicon, using ggml’s Metal kernels, with throughput testing done on a MacBook Pro M2 Max carrying 32 GB of unified memory.
The Bridge Between Two Codebases
For years, GGUF and Transformers lived in separate houses. llama.cpp people quantized, tuned, and shipped in C++. Transformers people trained, fine-tuned, and evaluated in Python. If you wanted to do both to the same model, you converted formats, lost fidelity somewhere in the process, or just kept two entirely different toolchains open and cursed quietly.
I’ve spent enough time babysitting llama.cpp builds on random Macs just to sanity-check a quantized checkpoint to know exactly how much friction that created. This removes a step that never should have taken as long as it did to remove.
Think of it like finally building a universal power adapter for a country that’s used a dozen incompatible plug shapes for a decade. The appliance itself doesn’t change. You just stop carrying a suitcase full of adapters everywhere you go. That’s what this integration does for GGUF files inside the Transformers world: the model doesn’t get better, the friction of using it just disappears.
Hugging Face is careful not to oversell this as a replacement. Its own competitive framing says llama.cpp remains the primary recommended standalone engine when raw local inference performance is the only goal. Fair enough, that engine has years of Metal and CUDA tuning behind it specifically for that job. What Transformers adds instead is everything llama.cpp was never built to do: inspecting activations layer by layer, fine-tuning a quantized checkpoint, prototyping a custom layer on top of it, or running an eval suite, all without leaving the Python API most ML engineers already live in.
Is that a bigger deal than it sounds? I’d argue yes. Quantized models have mostly been treated as finished products, things you download and run, not things you poke at. Making them inspectable and fine-tunable inside a standard workflow changes what people do with the smallest, cheapest versions of a model, not just how fast those versions run.
The narrow initial scope is honest, too. Apple Silicon first, Qwen3.5 architecture first, benchmarked on one specific machine. That’s not a company claiming universal support on day one. It’s a company shipping the case they’ve actually tested and leaving the rest for later.
Why GGUF Became the Format Everyone Quantizes To
GGUF exists because most people running AI models locally aren’t running them on a server farm. They’re running them on a MacBook, a gaming PC, or whatever machine happens to be on the desk. Quantization shrinks a model’s weights down from full precision to smaller representations, trading a bit of accuracy for a lot less memory and disk space.
llama.cpp popularized GGUF as the container format for exactly that trade-off. It became the default way to package a quantized model so any compatible engine could load it without needing to know how the original model was trained or in what framework. That’s the whole appeal: one file format, many quantization levels, portable across whatever hardware you’ve got.
But GGUF’s popularity created a split ecosystem. The people building and quantizing models often work in C++ tooling. The people training, evaluating, and fine-tuning models mostly work in Python, inside libraries like Transformers. Bridging that split without sacrificing speed has been the hard part, and it’s the part this update is actually trying to solve, not just paper over.
What to Track From Here
Watch whether this optimization work spreads beyond Apple Silicon. Metal kernels on a MacBook Pro are one environment; CUDA on a desktop GPU or a cheap ARM board are very different tests, and Hugging Face hasn’t published numbers there yet.
Watch which model architectures get support after Qwen3.5. If this stays a one-family showcase, it’s a demo. If Llama, Mistral, and Gemma variants show up in GGUF form inside Transformers within a few months, it’s a real platform shift.
And watch the actual throughput comparisons against raw llama.cpp once independent benchmarks show up. “Close to llama.cpp” is the claim right now. Whether that holds under real workloads, with real context lengths and real batch sizes, is the number that will decide if developers switch or just keep both tools installed out of habit.
Editor's Note
What gets me about this one isn't the feature, it's the honesty of the scope. Apple Silicon first, one model family first, benchmarked on one specific laptop. I'd rather see that than a company claiming universal support it hasn't tested. My real question is whether Transformers actually holds throughput parity once someone runs this on a GPU box with a long context window, because that's the test that decides if anyone drops llama.cpp entirely instead of just keeping both installed.
– Sanket Chaukiyal, founder, SmartChunks
FAQ
What exactly did Hugging Face add support for?
Transformers can now load GGUF quantized model checkpoints directly through from_pretrained, and serve them through transformers serve with an OpenAI-compatible API, without requiring a separate llama.cpp installation.
Do you still need llama.cpp installed to use this?
No. The point of the integration is that Transformers reuses llama.cpp's underlying ggml kernels through the kernels library internally, so the GGUF file loads and runs inside the standard Transformers workflow on its own.
Which models and hardware work best right now?
Initial optimization is focused on Apple Silicon using ggml Metal kernels, with the Qwen3.5 architecture as the tested case. Benchmarks were run on a MacBook Pro M2 Max with 32 GB of unified memory, and a Qwen3.5-4B checkpoint ranges from 8.42 GB at BF16 down to 2.74 GB at Q4_K_M.
Can you fine-tune or evaluate a GGUF model through Transformers now?
Yes, that's the core advantage over using llama.cpp alone. Bringing GGUF into Transformers lets developers inspect activations, fine-tune, prototype custom layers, and run evals in PyTorch without switching out of the standard API.
Source: Hugging Face
