Argo-Bench Puts AI Data Agents Inside a 7.5-Billion-Row Warehouse, and Most of Them Get Lost

TL;DR

• Argo-Bench tests 14 frontier and open-weight models on 210 enterprise analytics and operations tasks, built around a simulated NYC food delivery platform that logs 81 million orders in 2024.
• The underlying ERP warehouse spans 235 tables and 7.5 billion rows, modeled on Oracle E-Business Suite, a scale that dwarfs the single-table datasets most text-to-SQL benchmarks use.
• The best-performing model averaged just 59.5 points and only hit a score of 95 or higher on 34.8% of tasks.
• Unlike older benchmarks that just grade whether a query returns the right rows, Argo-Bench scores agents on real consequences in simulation, like banning a fraudulent account or issuing backpay correctly.

Inside the 7.5-Billion-Row Stress Test

Researchers from TextQL built Argo-Bench to answer a question that’s been bugging anyone who’s watched a demo video of an ‘autonomous data agent’ closing tickets in seconds: can these things actually survive contact with a real enterprise database? Their answer, in their own words: “We introduce Argo-Bench, an evaluation framework comprising 210 data science and analytics tasks.”

The setup isn’t subtle. Argo-Bench simulates a NYC food delivery business that processed 81 million orders across 2024, then exports that world into an ERP warehouse styled after Oracle E-Business Suite. That warehouse holds 235 tables and 7.5 billion rows. Compare that to the public datasets most text-to-SQL leaderboards still lean on, where a handful of tables usually covers the whole business logic, and you start to see why this benchmark exists.

The tasks themselves go past query generation. Agents aren’t just asked to pull a number from a table; they’re asked to act, things like flagging and banning a fraudulent account, allocating a budget across departments, or issuing correct backpay to drivers. Every action gets graded on what actually happens inside the simulator afterward, not just whether the SQL parsed cleanly. And that distinction turns out to matter a lot.

The headline number is blunt. “The strongest of 14 frontier and open-weight models scores 95 or higher on only 34.8% of tasks and averages 59.5 points.” Fourteen models were tested. One came out on top. That top model still failed to clear a near-perfect score on roughly two-thirds of the tasks thrown at it.

What This Means

Here’s the uncomfortable part for anyone selling ‘agentic’ data tools right now: 59.5 points is not a passing grade in any classroom I’ve ever sat in, and this is supposedly the best model available. I’ve sat through enough vendor pitches promising hands-off enterprise automation to know how wide the gap usually is between a polished demo and a warehouse full of messy, real-world data, but seeing it quantified at 34.8% is still sharper than I expected.

Think of it like handing a rookie delivery driver the keys to an eighteen-wheeler and telling them to navigate Manhattan at rush hour with no map and no GPS. They might know how to drive. They’ve just never driven anything this big, through streets this tangled, where one wrong turn doesn’t just cost time, it costs someone their backpay or lets a fraudulent account slip through. That’s roughly what a 235-table, 7.5-billion-row schema feels like to a model trained mostly on toy databases.

So what happens when the model responsible for banning a fraudulent account instead approves a budget nobody authorized? That’s not a hypothetical glitch, it’s the exact kind of consequential error Argo-Bench is built to catch, and the 59.5 average suggests it’s happening a lot. The benchmark’s own framing nails the stakes: Argo-Bench shifts evaluation from simple text-to-SQL generation toward consequential enterprise decision-making, and current frontier models are struggling with both the scale of the schema and the responsibility of autonomous action.

Worth asking too: if the strongest of 14 leading models can’t crack a 95 on a third of its homework, how much unsupervised autonomy should any enterprise actually be handing these systems today? Not much, based on this data. Not yet.

The competitive picture here isn’t about one vendor beating another so much as an entire category failing a harder test together. Fourteen models, frontier and open-weight alike, all ran into the same wall of relational complexity and consequential grading. Nobody cleared it comfortably. That’s a different kind of result than the usual benchmark story where one lab edges out another by a few percentage points; this is an entire field getting a reality check at once.

The Benchmark Problem Nobody Fixed Until Now

For years, text-to-SQL evaluation has run on a quiet assumption: if a model can write a correct query against a small public dataset, it understands databases. Audits of those older benchmarks found the opposite problem hiding underneath, widespread errors in the answer keys themselves and schema structures so simplified they barely resemble what a real company actually runs on.

That mismatch matters because real enterprise systems don’t fit on a napkin. Oracle E-Business Suite style ERPs, the kind Argo-Bench models its 235-table warehouse on, spread business logic across dozens of interconnected tables where a single customer order might touch billing, inventory, fraud flags, and payroll simultaneously. A model that’s only ever been graded on single-table lookups has no practice navigating that. Argo-Bench’s 7.5 billion rows and 81 million simulated orders aren’t just bigger numbers for the sake of scale; they’re an attempt to recreate the actual mess that enterprise software lives in every day.

And that’s the real value of this kind of benchmark. It’s not flashy, and it won’t produce a viral demo clip. But it exposes exactly where the confident-sounding agent falls apart once the database stops being tidy.

Three Things Worth Tracking From Here

Watch whether future model releases start quoting Argo-Bench scores directly, the way labs now routinely cite MMLU or SWE-bench results; that would signal the industry is treating consequential enterprise reasoning as a real competitive metric rather than a side note. Also keep an eye on whether any model manages to push past that 34.8% ceiling on 95-plus scores in the next evaluation cycle, since a meaningful jump there would be the first real evidence that scale alone, bigger context windows, more parameters, is closing the gap rather than just better prompting tricks. Finally, it’s worth tracking whether other research groups build their own Argo-Bench-style enterprise simulations, because one benchmark from one team is a data point, not yet a consensus.

Source: Hugging Face