Lossless.shn5 Shrinks BF16 Models 1.5x to Fit a 61GB MoE on a 16GB GPU

TL;DR

• A Hugging Face Forums update describes.shn5, a lossless format that makes BF16 model files about 1.5x smaller.
• The Leviathan engine runs Qwen3-30B-A3B, a 61 GB MoE model, on a 16 GB RTX 4080, using 6.9 to 10.3 GiB of VRAM.
• Decode is reported 38-45% faster (up to 16.2 tok/s), and prompt processing about 19x faster (up to 683 tok/s on a 700-token prompt).
• All figures are self-reported by the author. Vulkan and.shn5 support were announced for release, CUDA first.

A 61 GB model on a card with 16 GB

The claim is blunt. A forum post on Hugging Face, published on 14 November 2026, says the Leviathan engine now runs Qwen3-30B-A3B, a 61 GB Mixture-of-Experts model, on a 16 GB RTX 4080. VRAM use lands between 6.9 GiB and 10.3 GiB, depending on how many expert slots you give it.

The enabling piece is.shn5, a lossless format for BF16 weights. Files come out roughly 1.5x smaller, and nothing is thrown away to get there. That’s the whole point of the word lossless.

The speed numbers are the loud part. Decode is 38-45% faster, reaching up to 16.2 tok/s. Prompt processing is about 19x faster, hitting up to 683 tok/s on a 700-token prompt. The author’s own wording: “Decode is 38-45 % faster, and the prompt pass is about 19x faster. It is also exact now: in full fp32, it matches a float64 reference of the model as closely as our slow exact path does.”

According to the post, the Vulkan build and.shn5 support were being released under Leviathan the same day, with CUDA first.

The PCIe fix is the real story

Running a big MoE locally usually means parking expert layers in system memory and shuttling them to the GPU over PCIe. That link is the bottleneck. Everything else is a rounding error next to it.

So the gains here come from attacking the traffic, not from a faster multiplier. The post credits several changes: experts execute as grouped matrix multiplications, the whole prompt crosses PCIe in a single pass, copy and compute overlap, a faster expert decoder uses on-chip lookup tables, and a new attention kernel handles the rest. Think of moving house. The old method carried boxes one at a time through a revolving door. The new one backs a truck up to the front step and loads it once.

Does that matter for the rivalry with llama.cpp? Quite a bit. The post benchmarks against llama.cpp CUDA with --cpu-moe. On the same 700-token prompt, llama.cpp managed 116-138 tok/s, or 247-365 tok/s under its fastest settings. Leviathan reported 683. That’s nearly double the best llama.cpp figure quoted, and several times the default one.

Decode tells a gentler story. llama.cpp ran at 9.5-10.4 tok/s against Leviathan’s 16.2. Better, clearly, but not a 19x blowout. At 2,048 tokens the prompt gap narrows too: 1,234 tok/s for Leviathan versus 801 for llama.cpp.

I’ve read a lot of forum posts promising miracles on modest hardware. This one at least puts the competitor’s numbers, including the flattering-to-the-rival ones, right next to its own. That’s rarer than it should be.

And the lossless angle matters. Quantization is how most people squeeze big models onto small cards, and it costs precision. A route that stays interactive without that trade would make local inference cheaper in a way that doesn’t ask you to accept a slightly dumber model. If it holds up, of course.

Why MoE models suit this trick

A Mixture-of-Experts model only wakes a few experts per token, but all of them still have to live somewhere. That’s why a 61 GB model is a headache on a 16 GB card. The weights don’t fit, so something has to move.

Offloading experts between CPU and GPU is the standard workaround, and it’s the one that hits the PCIe wall. Smaller files help twice here: less to store, and less to push across the bus. The post’s VRAM range of 6.9 to 10.3 GiB shows the engine trading expert slots against memory. More slots, more VRAM, fewer trips.

The exactness claim deserves a second look. The author says that in full fp32 the engine matches a float64 reference as closely as the slow exact path does. That’s a statement about numerical fidelity, not about quality on any benchmark, and it’s the author’s own test. Still, it’s the right thing to measure for a format that calls itself lossless.

Context matters on scope, too. Everything reported is on one GPU, an RTX 4080, with one model, Qwen3-30B-A3B. Nothing in the source says how it behaves on other cards or other MoE architectures.

Three things to check when the release lands

First, independent reproduction. The numbers come from the author’s forum update, and the material supplied cites no outside reruns. When the CUDA build ships, watch for people running the same model on their own 16 GB cards and comparing against llama.cpp at matched settings. The gap between 247-365 tok/s and 683 tok/s will get tested fast.

Second, the Vulkan build. The post says full Vulkan support is coming under Leviathan, CUDA first. If it works, that widens the audience beyond one vendor’s cards. Whether it keeps the same speeds is an open question.

Third, which models get.shn5 files. A format is only useful if the weights you want exist in it, or convert easily. Check how conversion works, how long it takes, and whether other MoE models see similar shrinkage and speed.