TL;DR
- Researchers Aayam Bansal and Keertan Balaji released OpenDiscoveryTrace, a public CC BY 4.0 dataset of 558 complete AI-scientist trajectories across 124 tasks, recording 9 structured fields per step instead of grading only the final answer.
- Seven models were tested: GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro each ran 124 trajectories, while four smaller open models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, Qwen2.5-1.5B) ran 30 each, plus 60 live-retrieval variant runs.
- All three frontier models land in the same 84-89% success band, but Claude Opus 4.6 racks up 30x more errors per trajectory than GPT-5.4 (2.5 versus 0.08, p < 0.0001).
- The error types do not match either: 66.7% of Claude Opus 4.6’s mistakes come from tool misuse, while 83.6% of GPT-5.4’s come from flawed reasoning, meaning two models with near-identical scorecards are failing for opposite reasons.
A Dataset Built to Watch AI Scientists Think, Not Just Finish
OpenDiscoveryTrace does something most AI-agent benchmarks skip entirely. Instead of checking whether a model’s final paper, hypothesis, or code snippet was correct, it logs every single step an agent takes to get there. Each step carries nine fields: the model’s internal thought, the tool call it made, the observation it received back, any error, whether it triggered a revision, and a self-reported confidence score.
The dataset spans 124 tasks pulled from drug discovery, materials science, genomics, and literature analysis, the kind of grunt work autonomous science agents are increasingly asked to do unsupervised. Bansal and Balaji did not just run the three current frontier heavyweights. They also threw in four much smaller open models at 30 trajectories apiece, plus 60 live-retrieval variant runs where agents pulled real-time data instead of working from static context. That range matters. It is the difference between judging a chef by the plated dish and by the security footage of the whole kitchen.
As the paper puts it: “Existing benchmarks for autonomous AI scientists evaluate only final outputs, generated code, hypotheses, or papers, yet discard the reasoning process by which those outputs were obtained. This makes it impossible to audit scientific methodology, diagnose failure modes, or distinguish systematic reasoning from fortunate guessing.”
What Same Scores Are Hiding
Here is the number that should make anyone deploying these agents sit up: three models finish in an 84 to 89 percent success band, close enough to call a statistical tie. And yet Claude Opus 4.6 generates 2.5 errors per trajectory on average while GPT-5.4 generates 0.08. That is not a rounding difference. That is a 30x gap hiding underneath what looks, from the outside, like near-identical performance.
Think of it like two mechanics who both get your car running again by the end of the day. One does it clean, tightens every bolt on the first try. The other fixes the engine fine but leaves wrenches scattered across the floor, grabs the wrong socket four times, and occasionally bangs on the wrong panel before getting to the right one. The car starts either way. But you would want very different things from each mechanic if you were handing them the keys to something more expensive than an oil change, and autonomous science agents are being handed exactly that kind of trust.
The competitive picture gets sharper once you look at error type, not just error count. Claude Opus 4.6’s mistakes are 66.7% tool misuse, meaning it tends to reach for the wrong function or misuse the right one. GPT-5.4’s errors are 83.6% reasoning errors, meaning when it fails, it is usually because its logic went sideways, not because it fumbled a tool call. That is the more troubling failure mode of the two, since a reasoning error can quietly produce a plausible-sounding wrong conclusion, while a tool misuse error is often loud and easy to catch mid-run. Would you rather your AI collaborator make visible mistakes it can self-correct, or invisible ones that only surface when someone checks the underlying logic weeks later?
Gemini 3.1 Pro sits in the same benchmark run but the dataset summary does not break out its individual error rate the way it does for the other two, which is itself a small signal about how uneven process-level reporting still is across labs. Nobody publishes this stuff by default. That is the gap OpenDiscoveryTrace is trying to force open.
Why Final-Answer Grading Was Never Enough
The problem OpenDiscoveryTrace is responding to is not new, it has just been mostly ignored. Standard evaluation for AI scientific agents has always checked the destination and skipped the road. Did the model produce a working piece of code? Did the generated hypothesis match the known answer? Fine, tick the box, move on.
But that approach cannot tell you if the model got there through actual scientific reasoning or through a lucky guess that happened to land on the right answer. It cannot tell you where an agent is likely to break down on a harder, unseen task, because you never saw the intermediate steps where the wobble started. And it definitely cannot help anyone building governance frameworks for autonomous research, since you cannot audit a process you never recorded in the first place.
OpenDiscoveryTrace’s 9-field structure directly targets that blind spot. Thoughts, tool calls, observations, errors, revision triggers, and confidence scores turn a black box into something closer to a lab notebook. That is a meaningful shift for a field that has, until now, mostly graded homework without ever seeing the scratch paper.
What To Watch From Here
The obvious next question is whether other labs start publishing their own trace-level data voluntarily, or whether it takes regulatory pressure before anyone beyond a research paper’s authors bothers to log a model’s tool-call failures in public. Watch for whether Anthropic or OpenAI respond directly to being singled out on error rates, even implicitly, in their next model release notes.
Also worth tracking: whether the 558-trajectory dataset gets adopted as a standard benchmark that other researchers build on, the way earlier agent benchmarks did, or whether it stays a niche academic artifact. And keep an eye on the smaller open models in the study. If Qwen2.5-7B or Phi-3.5-mini show error patterns that scale predictably with model size, that could tell the field something useful about whether tool misuse and reasoning errors are architectural quirks or just a function of raw capability.
Editor's Note
What got me here is how casually the industry has accepted final-answer grading as good enough for agents doing real scientific work. A 30x error gap hiding behind two nearly identical success rates is exactly the kind of thing a spec sheet will never show you. I am watching whether any major lab responds to being singled out on error type, because right now this kind of process-level transparency is voluntary, and voluntary rarely scales without pressure.
– Sanket Chaukiyal, founder, SmartChunks
Source: arXiv
