TL;DR
- FlashInfer v0.7.4 adds DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE for Nvidia Rubin (SM107).
- A new batch-invariant mode for GEMMs and quantizers on Blackwell and Rubin returns the same bits for a row at any batch size.
- The release also adds SDSA and MiniMax-M3 indexers, Gemma 4 decoding from an NVFP4 KV cache, and a fused FP8 GEMM with all-reduce on Blackwell.
- The release notes don’t say what batch-invariant mode costs in throughput, and the few speedups they cite come without baselines.
FlashInfer v0.7.4 ships Rubin kernels and bit-exact GEMMs
The v0.7.4 release landed on 11 November 2026. Its own summary is blunt: it “brings DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE to Rubin, and adds a batch-invariant mode for GEMMs and quantizers on Blackwell and Rubin that returns the same bits for a row at any batch size.”
The Rubin work targets SM107. There’s a DeepSeek-V4 FP8 decode route on that architecture using a backend named ‘cake’, plus MXFP4 W4A4 support for MegaMoE. Batch-invariant GEMMs are supported on B200, B300 and Rubin.
Blackwell gets its own list. The release adds an SDSA indexer on Blackwell, a MiniMax-M3 indexer that also covers Rubin, Gemma 4 decoding from an NVFP4 KV cache, and a fused FP8 GEMM with all-reduce. It’s a dense release. Most of it is hardware-specific plumbing.
Why identical bits matter more than another speed claim
Batch-invariant execution is the part I’d circle. The point is simple: a row gets the same bits whether it travels alone or inside a big batch. The stated payoff is bit-for-bit parity between inference engines and RL training pipelines.
Think of a kitchen scale that reads a different weight for your flour depending on how many other bags are on the counter. You’d never bake with it. Inference stacks have often behaved a bit like that, and most teams just live with it. Why? Because chasing exact reproducibility usually means giving something up.
And that’s the open question here. The release notes don’t say what the batch-invariant mode costs in throughput, or whether it’s meant to run all the time or only when you’re debugging or training. I’d bet on the second. But that’s a guess, not something the release states.
Still, I’d rather see a boring reproducibility feature than another unverified speedup headline. If you’ve ever tried to work out whether a model’s output drifted because of the model or because of the batch, you know why. Exact matches shrink the list of suspects.
The Rubin items read differently. They’re early kernel support for next-generation hardware, which prepares serving code for architectures that teams will want to run. Does early support guarantee good performance when the chips arrive? No. It just means the code paths exist.
Where FlashInfer sits in the serving stack
FlashInfer is a GPU kernel library built for LLM serving and inference. It supplies optimized attention, GEMM and MoE operations across multiple GPU architectures. It’s a layer under the engines rather than an engine itself.
That makes serving engines its customers more than its rivals. The release notes even describe a change made so vLLM can keep Mamba state trackers inside its own per-slot state page. They make no head-to-head comparisons with other kernel libraries, and I won’t invent any. What the notes do show is the shape of the race: support for new model families and new silicon, shipped as kernels.
The model coverage in this release is telling. DeepSeek-V4, MiniMax-M3 and Gemma 4 all show up by name. Kernel libraries live or die on how fast they cover whatever models people actually deploy.
The release notes cite a few relative speedups for individual kernels, but without baselines or hardware details, and nothing in them measures the batch-invariant mode itself. That’s thin ground for performance claims, so treat this as a capability announcement. It’s not a benchmark result.
Three things to check before you upgrade
First, test the batch-invariant mode against your own outputs. Run the same rows at different batch sizes and compare bits. The release says they should match on B200, B300 and Rubin, so verify it on your workload.
Second, watch for measurements of its cost. Independent throughput numbers for the batch-invariant GEMMs would tell you whether this is an always-on setting or a special mode. Third, keep an eye on the SM107 paths once Rubin hardware is actually in people’s hands, because kernel support on paper and kernel behavior in production are different things.
Editor's Note
I keep coming back to the batch-invariant mode, because it's the least glamorous item in the release and probably the most useful. Anyone who has chased a ghost difference between two serving setups knows the pain. What I want next is a cost number. If bit-exact output carries a steep throughput penalty, it becomes a debugging switch. If it's cheap, it could become a default, and that would surprise me.
– Sanket Chaukiyal, founder, SmartChunks
FAQ
What is new in FlashInfer v0.7.4?
It adds DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE on Nvidia Rubin (SM107), and a batch-invariant mode for GEMMs and quantizers on Blackwell and Rubin. It also adds SDSA and MiniMax-M3 indexers, Gemma 4 decoding from an NVFP4 KV cache, and a fused FP8 GEMM with all-reduce on Blackwell.
What does batch-invariant mean here?
A row returns the same bits at any batch size. The stated benefit is bit-exact reproducible inference output, and bit-for-bit parity between inference engines and RL training pipelines.
Which GPUs support the batch-invariant GEMMs?
The release lists B200, B300 and Rubin. The mode covers GEMMs and quantizers on Blackwell and Rubin.
Does the release include performance benchmarks?
Not in any detail. A few changelog entries cite relative speedups for individual kernels, but without baselines or hardware details, and nothing measures what batch-invariant mode costs in throughput.
