NVIDIA Retires GenAI-Perf, Ships AIPerf to Fix a Benchmarking Tool That Was Lying to Engineers

TL;DR

• NVIDIA released AIPerf on September 18 as the official successor to GenAI-Perf, built specifically for testing LLM inference under heavy concurrency.
• The tool splits load generation and result processing into separate worker processes coordinated over ZMQ, sidestepping the Python GIL bottleneck that distorted single-process load testers.
• AIPerf supports more than 15 endpoint types and can replay real production traffic traces from Mooncake, Baseten, and WEKA’s AgentX, alongside datasets like ShareGPT.
• It tracks Time to First Token, inter-token latency, request latency, and output throughput, and pulls GPU power and utilization straight from DCGM or pynvml when available.

Why GenAI-Perf Started Lying to Engineers

NVIDIA isn’t shy about what it just retired. “AIPerf is the designated successor to GenAI-Perf and is a ground-up rewrite,” the company said in the developer blog post announcing the tool on September 18. That’s not a minor version bump talking. That’s a full replacement, and the reasoning behind it explains a lot about why so many teams have been quietly distrusting their own load test numbers.

GenAI-Perf ran on top of NVIDIA’s Perf Analyzer using a single-process architecture. Fine for light traffic. Under the kind of concurrency modern LLM serving stacks actually see in production, it started hitting Python’s Global Interpreter Lock, a hard ceiling on how fast one process can generate and log requests at the same time.

AIPerf’s fix is architectural, not cosmetic. “AIPerf is a multiprocessed system: worker processes generate load, separate record-processor services handle results, and everything is coordinated over ZMQ,” according to NVIDIA. Load generation and result logging no longer fight each other for the same interpreter thread.

The tool now supports more than 15 endpoint types, covering chat, responses, NIM rankings, and image generation. It also integrates trace replay formats from Mooncake, Baseten, and WEKA’s AgentX, so teams can throw actual production-shaped traffic at a server instead of a synthetic approximation. On the metrics side, AIPerf collects latency percentiles from p25 through p99, plus GPU power draw, utilization, and memory figures when DCGM or pynvml is present on the box.

What This Means

Picture a single stopwatch operator trying to time ten thousand marathon runners crossing a finish line at once, clicking a button by hand for every single one. Past a certain crowd size, the operator’s own clicking speed becomes the number on the results sheet. Not the runners’ actual pace. That’s roughly what was happening to teams running GenAI-Perf against modern LLM clusters: the single Python process generating and logging every request hit its own ceiling before the GPUs ever broke a sweat, and the latency chart ended up describing the test harness instead of the model serving stack.

I’ve sat through enough capacity planning meetings where someone points at a chart built on a load generator that was never stress tested itself, and the numbers get treated as settled fact anyway. A vendor admitting its own tool had a bottleneck, rather than quietly patching around it, is rarer than it should be.

Does one open source tool fix a decade of sloppy load testing habits across the industry? Of course not. But it does raise the floor for what a credible benchmark run is supposed to look like. NVIDIA is positioning AIPerf against a field of ad hoc scripts and the benchmark utilities bundled with vLLM and SGLang. Those work fine for one engineer stress testing one endpoint from a laptop. They get shakier once a team wants to simulate thousands of concurrent sessions, replay a real trace pulled from Mooncake or Baseten, and pull GPU telemetry off DCGM in the same run.

AIPerf’s split between load-generating workers and a separate result-recording service doesn’t just add speed headroom. It adds structural honesty to the measurement itself: the thing doing the counting is no longer the thing that’s out of breath. For any team using inference benchmarks to justify a hardware purchase or a deployment config, that distinction is the whole ballgame.

The Backstory: GenAI-Perf and the GIL Wall

GenAI-Perf wasn’t a bad idea when it shipped. It gave teams a standard way to measure LLM serving performance instead of everyone hand-rolling their own curl loops and calling it a benchmark. The problem showed up later, as concurrency requirements climbed and organizations started testing serving stacks under conditions that looked more like real production traffic than a demo.

Single-process load generators built on Python inherit a known limitation: the Global Interpreter Lock only lets one thread execute Python bytecode at a time. For light workloads that’s invisible. Once you’re trying to simulate hundreds or thousands of simultaneous clients, the interpreter itself becomes the slowest part of the pipeline, and the latency numbers it reports stop reflecting what the GPU-backed server is actually doing.

AIPerf’s answer, splitting the work across coordinated worker processes and a dedicated record-processor tier over ZMQ, is a fairly standard fix for GIL-bound systems in other domains. Applying it specifically to inference benchmarking, with support for the endpoint variety and trace formats teams are already using in production, is what turns it from a technical patch into something closer to an industry-standard tool.

Where AIPerf Goes From Here

Watch whether other serving stack maintainers start treating AIPerf as the default comparison tool rather than their own bundled scripts, since a shared benchmark only means something once multiple parties trust the same yardstick. Watch how the trace replay integrations with Mooncake, Baseten, and WEKA’s AgentX get used in practice, because real production traces tend to expose gaps that synthetic datasets never catch. And keep an eye on whether GPU telemetry through DCGM and pynvml becomes a standard line item in published benchmark results, or stays an optional extra that most reports quietly skip.

Source: NVIDIA