Nexus Skips the Tool-Schema Tax That’s Slowing Down Agentic LLMs

TL;DR

• A new research paper introduces Nexus, a framework that stops agentic LLMs from re-prefilling entire tool schemas on every turn.
• Nexus reaches first-argument generation 1.66x faster than standard full-schema re-prefill and saves roughly 80% of main-context tokens.
• Routing accuracy holds near 89% even as the tool registry grows to 250 tools, a point where standard concatenate-all baselines overflow the context window.
• The team tested Nexus on Qwen2.5-14B-Instruct (Q4_K_M) running locally on Apple silicon unified memory.

The Schema Tax Nobody Talks About

Every time an agentic LLM wants to call a tool, it typically has to read the tool’s entire schema first: arguments, types, description, all of it, prefilled into context before a single output token appears. Do that for a handful of tools and nobody notices. Do it for a registry of 250, the scale real Model Context Protocol (MCP) deployments are heading toward, and the model spends more time reading manuals than doing anything useful.

Is reading the whole manual every single time really the only option? That’s the question a new arXiv paper, “Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory,” sets out to answer. The team built an INT8 semantic lookaside buffer (SLB) paired with a calibrated cross-encoder margin gate. Instead of re-encoding full schemas, the system routes by retrieval and generates arguments over a compressed textual signature, a median of just 19 tokens per tool, rather than over spliced key/value cache.

The numbers are the whole pitch here. Nexus hits first-argument tokens 1.66x faster than a full re-prefill approach, while trimming main-context token use by about 80%. Routing accuracy stays near 89% all the way out to 250 tools, the exact point where a standard concatenate-everything baseline overflows the context window and simply stops working.

Why RoPE Phase Drift Almost Broke This

Here’s the honest part of the paper, and it’s the part that makes me trust it more, not less. The team tried the obvious shortcut first: splice cached schema key/value blocks straight into context. It didn’t work cleanly. Splicing KV blocks off their original anchor position causes rotary position embedding (RoPE) phase drift, basically scrambling the model’s internal sense of where it sits in the sequence.

Fixing that required depth-adaptive suffix redecoding, and the fix has a cost. During that redecoding window, latency speedups can temporarily drop to 0.98x, essentially breaking even, before converging back toward the reported 1.66x figure (the paper cites a broader 1.1x to 1.7x band depending on setup). That’s not a detail you’d expect in a paper trying to sell you a speedup. It’s the kind of thing that survives peer review because it’s true, not because it’s flattering.

I’ve watched enough agent framework demos built around three cherry-picked tools to know they rarely survive contact with a real registry. I don’t think Nexus fixes everything here, but the willingness to publish a failure mode is rare enough that it’s worth more than ten flashier papers without a caveat section.

Think of the standard approach like handing a new hire the entire 400-page employee handbook every single morning, just so they can find out what time the coffee machine gets restocked. It technically works. It also guarantees they’re exhausted before lunch. Nexus’s compressed signature is different. Not a handbook. A laminated index card taped near the machine: same information, a fraction of the reading, and the person still shows up in time to make coffee.

The competitive framing matters too. Standard tool-use baselines concatenate every schema definition into the context window up front, which means prefill computation scales quadratically as the registry grows. That approach doesn’t just get slower with scale, it eventually overflows the context limit entirely and stops functioning, long before it gets anywhere near 250 tools. Nexus sidesteps that failure mode by never asking the model to hold every schema in its head at once.

Where MCP Agents Actually Hit This Wall

The Model Context Protocol has become a default way agentic LLMs talk to external tools, and it’s genuinely useful for that. But MCP’s flexibility comes with a cost most demos never surface: verbose schema descriptions that get reprocessed constantly, because the model has no persistent memory of what it read last turn.

That’s a fine trade when you’re wiring up three or four tools for a weekend project. How many teams have already hit this ceiling without realizing what was actually slowing them down? It becomes a real infrastructure problem once companies build agents against registries with dozens or hundreds of internal APIs, each with its own schema, each competing for the same limited context window. Time-to-first-token, the delay before an agent even starts responding, is already one of the more visible pain points in production agent deployments, and schema prefill is a big, largely invisible chunk of it.

Nexus was tested specifically on Qwen2.5-14B-Instruct in a quantized Q4_K_M format, running on Apple silicon unified memory rather than a data center GPU cluster. That’s a deliberate choice. Unified memory setups are becoming a real target for local and edge agent deployments, and a framework that saves 80% of main-context tokens matters a lot more when your context budget is genuinely scarce, not just theoretically expensive.

What to Watch Next

Keep an eye on whether Nexus’s numbers hold up outside a single model family. The paper tests Qwen2.5-14B-Instruct specifically, so it’s worth watching whether the same INT8 lookaside approach generalizes cleanly to other open models, or whether the cross-encoder margin gate needs retuning for each architecture.

Watch the registry ceiling too. An 89% accuracy rate at 250 tools is solid, but production MCP deployments at larger organizations could plausibly push past that number within a year or two. Whether accuracy degrades gracefully or falls off a cliff past 250 tools is an open question this paper doesn’t answer yet.

And watch for code. A paper with numbers this specific, 1.66x, 80%, 19 tokens, invites scrutiny, and the agent infrastructure community tends to move fast once something looks reproducible. If Nexus or something like it ships as an open tool within MCP-compatible frameworks, that’s the real signal this research mattered beyond arXiv.

Source: arXiv