IFM’s K2-Horizon-MoVA-36B-A4B Goes Open: 36B Parameters, Only 4B Active

TL;DR

• IFM released GGUF checkpoints for K2-Horizon-MoVA-36B-A4B, a Mixture-of-Experts model that stores 36 billion parameters but only activates 4 billion per token.
• The model ships with a native 512K token context window, precisely 524,288 tokens, built into the architecture rather than bolted on later.
• IFM says the final checkpoint is out now, with intermediate checkpoints, training data, and training code still to come.
• On agentic and reasoning benchmarks, the 4B active model reportedly beats open dense models around 30B parameters and MoE models up to 15 times its size.

The Release, In Plain Terms

The Institute of Foundation Models quietly dropped GGUF files for K2-Horizon-MoVA-36B-A4B on Hugging Face this week, and the naming does a lot of the explaining for you. Thirty-six billion parameters sit inside the model. Only four billion of them switch on for any given token.

That gap is the point. IFM built the model as a Mixture-of-Experts system paired with what the company calls Mixture-of-Values attention, or MoVA. Per the repository: “K2-Horizon-MoVA-36B-A4B is the sparse member of the K2-Horizon family: a Mixture-of-Experts model with Mixture-of-Values attention (MoVA) that stores 36B parameters and runs 4B per token.”

The model also carries a native 512K context window, exactly 524,288 tokens, wired in from the start rather than added through extension tricks after the fact. IFM says the final checkpoint is live now. Intermediate checkpoints, the raw training data, and the training code itself are still coming, according to the repository page. That last part matters more than it sounds. Most open weight drops give you a finished model and nothing else. A promise of intermediate checkpoints means researchers could eventually watch the model’s capabilities form step by step, rather than just poking at the finished product from the outside.

You can read the repository directly on Hugging Face.

The Sparsity Bet

Here’s what actually surprised me about this release. It’s not the 512K context window, plenty of labs claim big windows now. It’s IFM putting a number on how much dead weight a 36B model can carry and still win.

According to the company’s own benchmark claims, K2-Horizon-MoVA-36B-A4B beats open dense models around 30B parameters on agentic and reasoning tasks, and it beats MoE models up to 15 times its size, while staying competitive with closed frontier systems. Fifteen times. If that number survives independent testing, it’s a genuinely uncomfortable data point for anyone still selling parameter count as a proxy for capability.

Think of the architecture less like a single big engine and more like a freight train with thirty-six cars coupled together, only four of which are actually loaded with cargo at any given moment, and yet the train outpaces rivals hauling far heavier loads down the same track. That’s MoVA and the MoE routing doing their job: most of the model sits idle per token, and the four billion parameters that do fire seem to be doing the work of something much larger.

Does raw parameter count still deserve the attention it gets, or has activation efficiency quietly become the metric that matters more? I’ve sat through a dozen “open” model launches this year that promised training data and code, then let that promise rot on a changelog nobody revisited. If IFM actually ships the intermediate checkpoints, this stops being just another leaderboard entry and turns into one of the more useful research-grade releases of the year, the kind that lets people study how a model’s reasoning ability actually forms during training instead of guessing from the outside.

That is a big if. But it’s the right kind of ambition for an open weights project to have.

What MoVA and MoE Actually Buy You

K2-Horizon is IFM’s broader family of open weight models, built around the idea that sparse activation can substitute for sheer size. Mixture-of-Experts architectures aren’t new, plenty of labs route tokens through subsets of a larger network to save on compute. What’s less common is pairing that routing with Mixture-of-Values attention, a mechanism IFM designed specifically for this model family, which changes how the attention layer weighs and combines information rather than just deciding which expert block handles a token.

The GGUF format matters here too. It’s the file format the llama.cpp community uses to run models locally on consumer hardware, on laptops, on a single GPU, sometimes on nothing more exotic than a decent CPU. Releasing GGUF checkpoints alongside the full weights means people without a server rack can actually run a 36B parameter model with a 512K context window on their own machines, provided they’ve got the RAM to hold that context.

Open weight releases have gotten more common, but weights-only drops are still the norm. A commitment to release training data and training code, not just the finished model, is the rarer move, and it’s the one that actually lets outside researchers check whether the benchmark claims hold up under conditions IFM didn’t hand pick.

The Checkpoints Will Tell the Real Story

The real test isn’t this release, it’s the next one. Watch for whether IFM actually publishes those intermediate checkpoints and training data on the timeline implied by the repository, since plenty of “open” projects have quietly dropped that commitment once the initial buzz faded.

Watch too for independent verification of the 15x claim. Benchmark numbers from a company’s own repository page are a starting point, not a conclusion, and the open source community tends to move fast once it gets its hands on a model that claims to punch above its active parameter weight.

And keep an eye on adoption among llama.cpp users specifically. A 512K context window sounds great on paper, but running it locally means enough RAM to hold that context in memory, and how quickly builders find workable configurations for consumer hardware will say a lot about whether this model becomes something people actually use day to day, rather than just something people admire from a leaderboard screenshot.