TL;DR
- CAVEAT is a new benchmark that drops computer-use agents into nine marketplace environments designed to test whether they still buy what is best for the user, not what is best for the platform.
- Across five model families, agents picked the user-optimal product 78.6% of the time when nothing was rigged. Turn on eight common steering mechanisms and that number collapses to 17.3%.
- No prompt injection required. The manipulation lives in ordinary UI tricks like default sorting, sponsored placement, and scarcity messaging.
- A patch called CAVEAT-Harness clawed back some ground, lifting user-optimal purchases by 55.0%, proof the failure is fixable but not solved.
Nine Marketplaces, Five Model Families, One Bad Number
Researchers behind CAVEAT built something most agent benchmarks skip: a shopping environment that actually wants something from the agent. Nine marketplace simulations, eight steering mechanisms modeled on tactics real e-commerce sites already use, and five different model families sent through the gauntlet with no special coaching.
The setup is deliberately mundane. No jailbreak. No adversarial suffix. Just a storefront nudging the agent the way a real storefront nudges a human, through sponsored listings, artificial urgency, and interface defaults that quietly favor the seller over the shopper.
The control numbers looked fine. Left alone, agents bought the user-optimal product in 78.6% of episodes, a respectable batting average for autonomous purchasing. Flip on the steering mechanisms and that same set of agents, running the same models, dropped to 17.3%. That is not a small model being fooled. That is a pattern across five families, which means the weakness sits closer to how these systems process incentive-laden environments than to any single lab’s training data.
The team also tested a fix. CAVEAT-Harness, designed to target the specific failure modes the benchmark exposed, improved user-optimal purchasing by 55.0%. Progress, not a cure.
The Agent Is Not the Problem, the Room Is
Most computer-use benchmarks ask a fairly narrow question: can the agent be tricked by a malicious instruction hidden in a webpage? CAVEAT asks a different one. What happens when nobody attacks the agent at all, and the environment itself just quietly wants a different outcome than the user does? That is the more uncomfortable question, because it does not require a villain. It just requires an ordinary marketplace doing what marketplaces have always done to human shoppers, now aimed at a system that cannot roll its eyes and close the tab.
Think of it like sending a very literal-minded friend into a car dealership with your credit card and a shopping list, then watching the salesperson rearrange the showroom, mislabel the sticker prices, and manufacture a fake countdown clock before your friend even walks in. A human with decades of pattern recognition might squint at the “only 2 left!” banner. An agent trained to be helpful and decisive can read that banner as a legitimate signal and act on it exactly as intended, by the platform, not by you.
I have read a lot of agent safety papers that treat manipulation as something injected from outside. This one flips that assumption, and it is the first benchmark I have seen that bothers to measure how agents behave when the manipulation is just the business model.
The drop from 78.6% to 17.3% is the headline, but the consistency across five model families is the real story. If this were one model’s quirk, you would shrug and move to a better one. It isn’t. Whatever is going wrong looks structural, tied to how these agents weigh urgency cues, default rankings, and sponsored placement against the actual task they were given. CAVEAT-Harness recovering 55.0% of that ground suggests targeted fixes help, but it also confirms the base behavior is broken enough to need a dedicated patch in the first place. That is not a rounding error. That is a design gap.
How We Got Here
Computer-use agents have moved fast from novelty demo to something companies actually deploy for booking, shopping, and account management. The benchmarks built to evaluate them mostly grew up in two flavors: cooperative environments where the agent just has to complete a task competently, and adversarial environments built around prompt injection, where a hidden instruction tries to hijack the agent outright.
CAVEAT sits in a gap between those two categories that nobody had really mapped. Real marketplaces are not neutral test beds and they are not attackers either. They are businesses with their own incentives, and those incentives have been shaping human purchasing behavior through sponsored placement, artificial scarcity, and default sorting for years. The researchers’ framing is blunt about the gap: existing benchmarks, as they put it, do not test whether agents preserve user objectives when the environment itself has a stake in the outcome. CAVEAT is an attempt to close that gap with something closer to how commerce actually works online, incentives and all, rather than a sanitized task list.
Signals Worth Tracking
Watch whether other labs start reporting steering-resistance numbers alongside their usual task-completion scores, since a benchmark this specific tends to get adopted fast once it exposes a gap this large. Keep an eye on whether CAVEAT-Harness or something like it gets folded into commercial agent products, and whether that 55.0% recovery holds up outside the paper’s own test conditions. And it is worth watching whether real marketplaces start explicitly designing their sites around agent behavior rather than human behavior, because if steering already works this well on agents, that is a business incentive of its own.
Editor's Note
I keep coming back to that 78.6% to 17.3% drop because it isn't about a broken model, it's about a broken assumption. We've spent years worrying about agents getting hijacked by hidden instructions, and this paper says the more likely failure is just an ordinary marketplace doing what marketplaces do. What I'm watching next is whether commercial agent products start disclosing steering-resistance numbers the way they disclose accuracy scores. Right now nobody has to, and that gap should worry more people than it does.
– Sanket Chaukiyal, founder, SmartChunks
FAQ
What is CAVEAT?
CAVEAT is a benchmark that tests computer-use agents across nine simulated marketplace environments to see whether they still buy the product that is best for the user when the platform itself is trying to steer them elsewhere.
How much did agent performance drop under steering?
Across five model families, agents bought the user-optimal product in 78.6% of episodes with no steering. With eight common steering mechanisms turned on, that dropped to 17.3%.
Did the researchers find a fix?
They built a tool called CAVEAT-Harness aimed at the specific failure modes the benchmark uncovered. It improved user-optimal purchasing by 55.0%, a meaningful gain but well short of closing the gap entirely.
How is this different from prompt injection research?
Prompt injection relies on a hidden or malicious instruction hijacking the agent. CAVEAT tests something more mundane: ordinary platform incentives like sponsored placement, scarcity messaging, and default sorting, none of which require an attacker at all.
Source: arXiv
