AI Models Ace the Early Game in Slay the Spire, Then Fall Apart Like Everyone Else

TL;DR

• A new benchmark called SpireBench tested 20 large language models on full runs of Slay the Spire at the game’s highest difficulty.
• Frontier models beat heuristic baselines and matched the average human player, but stayed far behind human experts.
• The strongest models win Act 1 as often as experts do, then their performance drops off as the run goes on.
• Every model tested misjudged risk, hoarding health instead of spending it to get stronger, and barely learned anything from its own past runs.

A Deck-Building Game Becomes an AI Report Card

Slay the Spire is not a typical AI benchmark. It’s a commercial roguelike deck-builder where you fight your way up a tower, choosing cards, relics, and fights across three increasingly brutal acts. There’s no single right move. There’s a web of tradeoffs that only pays off twenty decisions later.

That’s exactly why researchers picked it. SpireBench ran 20 LLMs through complete games at the hardest difficulty setting and compared them against heuristic bots, average human players, and genuine experts. The headline result, straight from the paper: “Across 20 large language models (LLMs) at the game’s highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts.”

Dig into where that gap opens up and the story gets more specific. “The strongest models win the first act as often as experts, then fall behind.” So it’s not that these models are bad at the game. They’re bad at staying good at the game once the stakes compound.

The researchers also ran a clever check: they renamed every card, relic, and enemy in a self-consistent way and reran the tests. Performance dropped. That tells you a chunk of what looked like strategic reasoning was actually the model recognizing Slay the Spire from its training data, not reasoning about the renamed game fresh. And when models were given written memories from their own previous runs to learn from, it didn’t help much. The memories were “often inaccurate and overly cautious,” according to the findings, which is a polite way of saying the models were bad historians of their own mistakes.

The Long Game Problem

Here’s the thing that stuck with me reading this paper: it’s not that these models can’t plan. It’s that they can’t hold their nerve. The finding that every single model “misjudge[s] risk over a run, preserving health rather than spending it to grow stronger” is, honestly, a more human failure than I expected from a machine. Spending health to gain power is the entire engine of Slay the Spire, you take damage now so you don’t die later, and the models flinched from that trade every time.

Picture a climber working a multi-pitch route. The first pitch goes perfectly, clean form, confident reaches, textbook footwork. But halfway up the second pitch, the exposure starts to feel real, and instead of committing to the next hold, the climber starts hugging the rock face, conserving energy that was supposed to be spent on the climb itself. That’s what these models are doing to their own health bars. They treat a resource meant to be used as a resource meant to be protected, and the run stalls exactly where the texture of the game gets harder to read.

Why does this matter past a card game? Because Slay the Spire is a clean proxy for any task that unfolds over many sequential, consequence-bearing steps: a multi-week research project, a drawn-out negotiation, a codebase refactor touching forty files. If a model can’t tell when spending a resource now is worth the payoff twenty steps later, how much should anyone trust it with a decision that actually compounds in the real world?

The competitive framing here isn’t company versus company, it’s capability class versus capability class. Heuristic baselines, the simple rule-based bots, got beaten outright. Average humans got matched. But the experts, the players who’ve internalized thousands of runs of risk calculus, still hold a real and measurable lead. That’s the gap that matters, and it’s not closing just because a model got bigger or newer.

Why Most AI Benchmarks Never Caught This

Most agentic benchmarks test short, episodic tasks: answer this question, complete this one tool call, solve this isolated puzzle. Those tests are useful, but they can’t separate three things that all blur together in a single long run: genuine long-horizon reasoning, leftover knowledge from pre-training, and plain luck.

SpireBench was built specifically to pull those threads apart. A full Slay the Spire run stretches across dozens of decisions that all feed into each other, card choices in Act 1 determine what strategies are even viable by Act 3. That structure is what exposed the risk-management failure in the first place. In a short task, a model never has to live with the consequences of hoarding a resource it should have spent. Over a full run, it does, and the cost shows up as a losing streak in the back half of the game.

The renaming experiment adds a second layer worth sitting with. If you strip away the model’s ability to recognize “Slay the Spire” by name and force it to reason about a game it’s never technically seen before, performance drops. That’s a useful, slightly uncomfortable data point for anyone assuming these models are reasoning from first principles rather than pattern-matching against something they memorized during training.

What to Watch From Here

The obvious thing to track is whether future model generations close the Act 1-to-Act 3 gap, or whether it’s a structural limit tied to how these models handle compounding risk over time. If the next wave of frontier releases gets tested on SpireBench and still falls off a cliff after the opening stretch, that’s a much bigger story than one benchmark paper.

Worth watching too: whether the cross-run memory problem gets fixed. A model that can’t learn from its own documented mistakes across repeated attempts at the same task is a model that’s going to struggle with any job that rewards iteration. And keep an eye on whether other researchers adopt long, commercial games as benchmarks generally. If SpireBench proves durable and hard to saturate, don’t be shocked if the next evaluation paper reaches for a strategy game instead of another static test set.

Source: arXiv