TL;DR
- ParseBench tests whether document parsers preserve the structure an AI agent needs to act correctly, not just whether the output looks right to a human eye.
- The benchmark covers 2,078 human-verified pages across 1,211 enterprise documents, graded against 169,011 evaluation rules.
- LlamaParse Agentic Plus tops the leaderboard with a 90.20 overall score, ahead of LlamaParse Agentic (87.01), Pulse Ultra 2 (81.60), and Anthropic’s Opus 5.5 (79.85).
- Five capability dimensions get scored separately: Tables, Charts, Content Faithfulness, Semantic Formatting, and Visual Grounding.
Inside The 2,078-Page Stress Test
ParseBench landed on GitHub under the run-llama organization, and it’s built around a question that most parsing benchmarks never bother asking. Does the output just look correct, or can a downstream agent actually do something with it?
That distinction sounds academic until you picture the failure mode. A parser can render a table that’s pixel-perfect on screen while quietly dropping a header row, and an agent reading that table will confidently pull the wrong number into a decision. Nobody notices until the decision is already wrong.
The dataset behind ParseBench isn’t small. It draws on 2,078 unique pages spread across 1,211 documents pulled from insurance filings, financial statements, and government paperwork, the kind of dense, messy PDFs that enterprise agents actually have to chew through. Layered on top sits 169,011 evaluation rules, which is a lot of granularity for something that used to get judged by eyeballing a screenshot.
Scoring runs across five capability dimensions: Tables, Charts, Content Faithfulness, Semantic Formatting, and Visual Grounding. As the project puts it, the goal is to test “whether parsed output preserves the structure and meaning needed for autonomous decisions, not just whether it looks similar to a reference text.” Right now LlamaParse Agentic Plus sits at the top with a 90.20 overall score, followed by LlamaParse Agentic at 87.01, Pulse Ultra 2 at 81.60, and Anthropic’s Opus 5.5 at 79.85.
Our Take
I’ve watched enough agent pipelines fall over on document ingestion to know this benchmark is solving a real problem, not a hypothetical one. Structure is the part everyone assumes is fine until it isn’t, and by then the agent has already made a bad call based on it.
Think of it like a home inspection. A house can look completely move-in ready from the curb, fresh paint, clean windows, staged furniture, while the load-bearing framing inside the walls is quietly rotting. A visual similarity score is the curb view. ParseBench is the inspector who actually goes into the crawlspace and checks whether the thing standing up will still be standing up when someone leans on it.
Now the competitive context is worth sitting with for a second. The benchmark lives under the run-llama GitHub organization, and the tool sitting at the top of its own leaderboard is LlamaParse, made by the same company. Is that disqualifying? Not necessarily, and the methodology behind the 169,011 rules looks genuinely rigorous. But it’s the kind of thing a reader should notice before treating the ranking as neutral ground truth. A vendor grading its own product first and publishing the results is a pattern worth watching, even when the underlying test design holds up.
What actually matters longer term is whether other vendors show up and submit their parsers for independent scoring, because a leaderboard with one dominant submitter next to a handful of comparison scores isn’t quite the same thing as an open competition. Pulse Ultra 2 landing at 81.60 and a proprietary model like Opus 5.5 hitting 79.85 suggests the gap is real and not trivial. Nine points between first and second place, and another five before you hit a general-purpose model, tells you specialized agentic parsing tools are currently winning this particular fight by a real margin, not a rounding error.
Where Document Parsing Has Been Failing
For years, document parsing got graded the same way OCR got graded: render the text, compare it to a reference, call it a day. That approach works fine if a human is the one reading the output. It falls apart the moment an autonomous agent is the one reading it, because agents don’t skim for gist, they act on specific fields, specific rows, specific formatting cues like strikethroughs or nested headers that change what a clause actually means.
The background context here is pretty simple. Enterprise workflows have leaned on document parsing to turn multi-page PDFs into structured text for years, and that pipeline mostly worked because a person was still checking the output before anything important happened. Agentic systems remove that checkpoint. When a parser silently mangles a table header or drops a footnote, there’s no human in the loop to catch it before the agent files a claim, approves a loan, or flags the wrong compliance line.
So the shift ParseBench represents isn’t really about parsing getting harder. It’s about the cost of a parsing mistake going up, because the thing reading the output can no longer tell the difference between confident and correct.
What To Track From Here
Keep an eye on whether the leaderboard actually diversifies beyond its current small field. Right now the story is mostly LlamaParse variants at the top and a short list of comparison models underneath, and a benchmark only earns real authority once competitors outside its own orbit start submitting scores. Watch too for how the dataset evolves. Enterprise documents change, new formats show up, and a benchmark that freezes at 2,078 pages will age fast if it doesn’t keep adding fresh, messier examples. And it’s worth watching whether agent frameworks start citing ParseBench scores directly when they choose a parsing backend, because that’s the moment a benchmark stops being a GitHub curiosity and starts shaping actual purchasing decisions.
Editor's Note
What gets me about ParseBench isn't the benchmark design, it's the crawlspace problem it points at. Every agent team I've talked to assumes their parsing layer is solved because the output reads fine in a demo. It never is. I'm watching to see if a vendor outside the run-llama orbit submits a competing score, because that's the real test of whether this becomes a standard or stays a house benchmark for one company's own tool.
– Sanket Chaukiyal, founder, SmartChunks
Source: GitHub
