A 9,500-Token Chat Bills 59,000. Here’s Where the Rest Goes

Sanket Chaukiyal

November 18, 2026

Take one hypothetical ten-turn customer support chat, the most ordinary job an AI bot does, and compare what’s on screen with what’s on the invoice. Add up every word on screen, including the hidden instructions, and you get about 9,500 tokens of text. The API bill for that same chat counts roughly 59,000.

Nobody is overcharging you. AI token pricing bills every token the model reads and every token it writes, on every single call. A chat model rereads the whole conversation each time it replies. Once you see where the extra 49,500 tokens come from, you can estimate any AI feature’s cost before you build it, and often cut that cost in half.

What is a token, in plain numbers?

A token is a chunk of text the model processes as one unit. Sometimes it’s a whole word (“the”). Sometimes it’s a fragment (“unbeliev” + “able”). Spaces and punctuation get folded in too. The model never sees letters. It sees a sequence of token IDs.

For English prose, OpenAI’s rule of thumb is that one token is about four characters, or about three-quarters of a word. Flip that around for estimating:

  • English prose: tokens ≈ words × 1.33. A 1,500-word article is roughly 2,000 tokens.
  • Code, JSON and markup: brackets, indentation and symbols each eat tokens. Characters ÷ 3 is a safer working estimate than ÷ 4.
  • Other languages: many non-English languages, especially those with non-Latin scripts, need noticeably more tokens to say the same thing. If your users write in Hindi, Japanese or Arabic, count a real sample rather than trusting the English ratio.
  • Images and files: these are converted into tokens too, scaled by resolution or page count. Anthropic, for example, counts an image in 28-by-28-pixel patches, so a larger image costs more tokens.

One caveat. Each model family uses its own tokenizer, so identical text produces different counts across vendors, and sometimes across versions from the same vendor. Anthropic says the tokenizer in its newer models produces about 30% more tokens for the same text than the one before it. Rules of thumb get you to a budget. For a real number, paste sample text into a counter such as OpenAI’s tokenizer tool, or read the usage field the API returns with every response.

Why AI token pricing tracks compute, not words

Providers price per token because tokens are what cost them money. Every token passes through the full model. More tokens means more GPU time and more memory, almost linearly.

The split between input and output prices comes from how inference actually runs. Input tokens are processed in one parallel sweep, called prefill. The GPU handles thousands of them at once, efficiently. Output tokens are generated one at a time, called decode. Each new token requires another pass through the model, and the hardware spends much of that time waiting on memory rather than calculating. (The prefill/decode split is so central to serving costs that engineers build whole systems around it, as in FluidPD’s approach to reassigning GPU roles mid-flight.)

That’s why output tokens cost several times more than input tokens. On current list prices at Anthropic, OpenAI and Google (see Anthropic’s API pricing page for one example), output usually runs about five to eight times the input price. It’s also why, for most requests, your question is cheap to read and the answer is expensive to write.

Long contexts carry a cost too. The model keeps a memory structure (the KV cache) for everything it has read, and that grows with every token. Some providers charge a higher rate above a certain prompt length: at the time of writing, OpenAI’s newest models above 272,000 input tokens, Gemini 3.1 Pro above 200,000 and Claude Haiku 5.5 above 100,000. Price is not the only downside, either: long contexts also blur the model’s focus.

The anatomy of one API request

Here is a single request to a support bot, priced against a hypothetical model at $3 per million input tokens and $15 per million output tokens. These are illustrative round numbers, not any vendor’s current rate card.

Line itemTypeTokensRate per millionCost
System prompt (instructions, tone, rules)Input1,500$3$0.0045
Customer’s messageInput500$3$0.0015
Retrieved help-center articlesInput4,000$3$0.0120
Model’s answerOutput800$15$0.0120
Total6,800$0.0300
One hypothetical support-bot request. Output is 12% of the tokens and 40% of the cost.

Three cents. That seems trivial, and per request it is. Two things in the table matter more than the total.

First, the customer’s actual question is the smallest line. The scaffolding around it (instructions plus retrieved documents) is eleven times bigger. Second, the 800-token answer costs as much as 4,000 tokens of documents. If you only optimize prompt length, you’re working on the cheaper half of the bill.

The line items people forget

The single request above is the honest case. Real applications add costs that never appear on screen.

Conversation history is re-billed every turn

Language models are stateless. They don’t remember the previous message. To keep a conversation going, your app sends the entire history again with every new turn, and you pay input rates for all of it, every time.

Run the numbers on a ten-turn chat. Keep the 1,500-token system prompt, and assume each turn adds a 300-token customer message and a 500-token reply.

  • Turn 1 input: 1,500 + 300 = 1,800 tokens
  • Turn 5 input: 1,500 + (4 × 800) + 300 = 5,000 tokens
  • Turn 10 input: 1,500 + (9 × 800) + 300 = 9,000 tokens

Summed across all ten turns, that’s 54,000 input tokens plus 5,000 output tokens. That’s the 59,000 from the opening, against 9,500 tokens of unique text. At our hypothetical rates, the chat costs $0.162 for input and $0.075 for output, or about $0.24 in total.

The growth is quadratic. A 20-turn chat on the same pattern bills about 188,000 input tokens, not the 108,000 you’d get by simply doubling. Double the conversation and the input bill here more than triples, and the multiplier creeps toward four as chats get longer. This is why long-running agents get expensive fast.

Thinking tokens are invisible but billable

Reasoning models work through a problem before answering. Anthropic, OpenAI and Google all bill those intermediate thinking tokens as output tokens, even when the raw reasoning is hidden or only a summary comes back.

Say a reasoning model thinks for 3,000 tokens before writing a 500-token answer. You pay output rates on 3,500 tokens: $0.0525 at our hypothetical $15 per million, versus $0.0075 for the answer alone. That’s seven times the cost for a reply that looks identical in length. The major APIs let you rein this in with an effort setting or a cap on output tokens. For routine questions, use it.

System prompts, tools and retrieved documents

Every tool definition you give a model (its name, description and parameter schema) is input text, sent on every call. Ten tools with detailed schemas can add a few thousand tokens before the user types a word. The same goes for retrieval: pulling eight document chunks “to be safe” when three would answer the question multiplies the biggest line in the table above.

Retries and failures

Malformed JSON, a failed tool call, a user hitting “regenerate”. Each one is a full request, billed in full, history included. Agent loops that retry on errors can quietly double a task’s token count. A budget without a retry buffer is a budget you will miss.

What caching and batching actually save

The re-sent history problem has a direct fix. Prompt caching lets the provider store the processed version of a prompt prefix. When the next request starts with the same tokens, those cached tokens bill at a steep discount: Anthropic, OpenAI and Google all list cached input at 90% or more off the standard rate on current models. Writing to the cache can cost extra. Anthropic charges 1.25 times the input rate for a five-minute cache entry, and OpenAI does the same on GPT-5.6 and later models.

Apply it to the ten-turn chat. Assume about 80% of input tokens are cache hits (the system prompt plus earlier turns), billed at 10% of the input rate, and ignore the small write premium to keep the arithmetic clean:

  • Effective input rate: (0.2 × $3) + (0.8 × $0.30) = $0.84 per million, down from $3
  • Input cost: 54,000 tokens × $0.84 per million = $0.045, down from $0.162
  • Total chat cost: $0.045 + $0.075 = about $0.12, down from $0.24

Caching halves the bill, and notice what happened to the balance. Output is now more than 60% of the cost. Once caching is on, answer length becomes the main lever.

Caching only works if the start of your prompt stays byte-for-byte identical. Put stable content (instructions, tool definitions, reference documents) first and variable content (the user’s message, timestamps) last. A timestamp at the top of a system prompt breaks the cache on every call. This worked example of a support bot’s cache bill shows one stray timestamp more than quadrupling the monthly total.

Batching is the other discount. If results can wait hours rather than seconds, for jobs like tagging a product catalog, summarizing last week’s tickets or translating archives, batch APIs cut prices by about half. Anthropic, OpenAI and Google all price batch requests at roughly 50% of the standard rate.

How to estimate tokens before you build

The estimate takes five steps and fits on a napkin.

  1. Write out one realistic request. Real system prompt, real retrieved documents, a real user message. Count it with a tokenizer, or use the words × 1.33 rule for a rough pass.
  2. Estimate the output, then pad it. Generate ten sample answers and average them. Models run longer than you expect. Add thinking tokens if you’re using a reasoning model.
  3. Model the conversation, not the message. If users go back and forth, use the history math above. Average turns per session matter more than average message length.
  4. Multiply by volume. Requests per day × 30.
  5. Add 20-30% for retries, regenerations and the long tail of unusually long sessions.

The formula behind it:

monthly cost = sessions × [(input tokens × input rate) + (output tokens × output rate)] ÷ 1,000,000 × 1.25

Scale up the support bot. Say it handles 300 chats a day, or 9,000 a month, at $0.24 each uncached. That comes to $2,160 a month before buffer, or about $2,700 with it. Turn on caching and the same traffic costs about $1,080, or $1,350 buffered. Cap answers at 300 tokens instead of 500 and output falls to $0.045 per chat. Shorter replies also shrink the history re-sent on later turns, so input drops to about 45,000 tokens. The cached total lands near $0.083 a chat, or about $750 a month before buffer.

Each lever is a configuration change, not a rebuild. Without the breakdown, you wouldn’t know which one to pull.

One scope note: none of this applies to flat-rate chat subscriptions, where you pay a monthly fee and hit message caps instead. Token math matters when you’re calling an API. For where flat-rate and free plans make sense, see this breakdown of free versus paid AI models for website work.

Is the cheapest per-token model actually cheapest?

Often, but not reliably. Per-token price is a rate. What you pay is rate × tokens used, and models differ widely in how many tokens they burn on the same task.

A model at half the price that writes answers twice as long costs the same. One that needs a retry on a third of requests costs more. A reasoning model at a higher rate that solves a task in one pass can undercut a cheap model that takes four attempts plus your time reviewing the failures.

Small models have become strong enough that routing helps. Send classification, extraction and short replies to a small model, and reserve the frontier model for the hard minority. Price cuts in that tier make this pattern more attractive. Claude Haiku 5.5 lists at a tenth of Haiku 4.5’s per-token price, though it also counts the same text as roughly 30% more tokens, which is why the real saving is smaller than the rate card suggests (more on Haiku 5.5’s lower running costs).

The fair comparison is cost per completed task. Run 50 real tasks through two candidate models, total the usage fields, and divide by the number of tasks each one got right. That number beats any rate card.

When you price an AI feature, price the full conversation, measure real output length, and turn on caching before you decide anything else. If the cost still doesn’t work after those three steps, the problem is the feature design, not the model.

Sanket Chaukiyal — Editor at Smart Chunks

Sanket Chaukiyal

Technology editor • 12+ years in editorial

Sanket is the founder and editor of Smart Chunks. He spent over six years at Autocar India (Haymarket SAC Publishing) as Sub Editor and Senior Copy Editor, and later served as Account Director (Content) at Rite Knowledge Labs. He holds a Master's in Media and Communication from the Symbiosis Institute of Media and Communication.

All articles → • LinkedIn