Researchers Use Fisher Information Geodesics to Outperform Standard Neural Network Pruning

TL;DR

• A new arXiv paper swaps parameter-magnitude pruning for geodesic distance measured through the Fisher information metric, calculating how far a model actually moves through probability space when a given parameter hits zero.
• The method was tested on fully-connected networks and vision transformers across MNIST and CIFAR-10, using 5 random seeds and pruning ratios sweeping the full 0% to 100% range.
• It beat both magnitude pruning and local Fisher information pruning on accuracy and Matthews correlation coefficient in every architecture and dataset combination tested.
• The paper (arXiv:2609.16129) was submitted September 14, 2026. It doesn’t include a code release or address computational cost at larger model scales.

Measuring What a Cut Actually Costs

Pruning is how you shrink a neural network without gutting what it can do. Cut enough dead weight and you get a smaller, faster model that still performs close to the original. The problem has always been deciding what counts as dead weight.

Most pruning tools use a shortcut: if a parameter’s value is small, snip it. Cheap to compute. Also, according to this new paper, wrong more often than the field has wanted to admit. Local Fisher information pruning tried to improve on that by accounting for curvature near the current parameters, but it still only looks at the immediate neighborhood, not the full path the model takes when a weight disappears.

The paper, titled “Optimal Pruning for Neural Architectures using Fisher Information Distances” (arXiv:2609.16129, submitted September 14, 2026), takes a different route entirely. It calculates the geodesic distance, through the Fisher information metric, between the unpruned model and the exact hypersurface where a chosen parameter equals zero. The authors put it plainly: “This distance determines the true change in the model, and its performance, under pruning.”

They tested the idea on fully-connected networks and vision transformers, running both MNIST and CIFAR-10 through five random seeds while sweeping pruning ratios from 0% up to a full 100%. Across every single architecture and dataset combination, the geometric method beat magnitude pruning and local Fisher information pruning on both accuracy and the Matthews correlation coefficient, a metric that’s particularly good at exposing how badly a model handles imbalanced classes. In the authors’ own words: “It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient.”

What This Means

Why has magnitude pruning survived this long when everyone doing the work knows it’s a heuristic and not a principle? Mostly because it’s fast and it’s good enough, and good enough has a way of calcifying into standard practice. This paper is an argument that good enough has been leaving real performance on the table.

Here’s an analogy that actually fits. Picture demolishing a building by first looking at which support beams are the thickest and assuming those are the ones holding everything up. That’s magnitude pruning. Local Fisher information pruning is a bit smarter: it checks how much each beam is currently flexing under load. But this new method runs an actual structural simulation, calculating how far the whole building would shift if you pulled each beam out entirely, and removes the ones that move the structure least, even the thick-looking ones. Size and local flex both turn out to be poor proxies for what really holds the thing up.

I’ve sat through enough compression talks at ML conferences to be numb to claims of beating every baseline, because usually that phrase is hiding a compute cost nobody wants to mention on stage. What makes this one worth a second look is the sweep: five seeds, two architectures, two datasets, the full 0 to 100% range, and a clean win in every single cell of that grid. That’s not a cherry-picked result. That’s a pattern.

The competitive framing here is straightforward. Magnitude pruning and local Fisher information pruning are the incumbents, the default tools most compression pipelines reach for because they’re cheap and well understood. This paper doesn’t claim those methods are broken. It claims they’re systematically leaving performance on the table because neither one accounts for the full geometric path a model travels when you zero out a parameter. If that holds up under scrutiny, both incumbents have a real problem: a more principled competitor that wins on accuracy and on Matthews correlation coefficient, every time, isn’t a marginal upgrade. It’s a replacement candidate.

Why Magnitude Heuristics Stuck Around This Long

Neural network pruning exists because bigger models cost more to run and more to store, and most of what makes them big turns out to be redundant once you look closely. Cutting parameters without wrecking accuracy has been a research goal since long before the current wave of massive models made it urgent again.

The trouble is that standard heuristics, whether magnitude-based or local curvature-based, measure something local and assume it tells you about global behavior. A weight can look tiny and still sit at a point where the model’s output is extremely sensitive to it. A weight can look large and turn out to barely matter once you account for how the whole network compensates. Magnitude and local curvature just don’t capture that kind of global displacement.

Fisher information geometry offers a different lens because it treats the model as sitting on a curved surface within probability distribution space, and it measures distance across that surface rather than distance in raw parameter values. That’s a fundamentally different question than “how big is this number,” and based on this paper’s results across two architectures and two datasets, it’s apparently a better one to ask.

Three Things That Will Decide If This Sticks

The obvious next step is peer review, since this is a fresh arXiv submission and hasn’t gone through that filter yet. Papers that win every comparison in a small benchmark suite sometimes wobble once independent labs try to reproduce the numbers, so that’s worth tracking closely.

The second thing to watch is scale. MNIST and CIFAR-10 are small, well-worn datasets, and fully-connected networks and vision transformers at this size are a long way from the billion-parameter language models where pruning costs actually bite hardest. Nobody has shown yet whether computing full geodesic distances stays computationally reasonable once the parameter count explodes, and the paper doesn’t spell out that tradeoff.

Third: a code release. Right now there’s no public implementation to point to, and geometry-heavy methods like this one live or die on whether other researchers can actually run them without reinventing the math from scratch. If a working implementation shows up and someone tests it on a model with real production weight, that’s when this stops being an interesting arXiv result and starts being something compression engineers have to take seriously.

Source: arXiv