Picture it: you paste a 300-page supplier contract into a chatbot, ask which clause covers late-delivery penalties, and it can confidently cite the wrong one. The model didn’t run out of room. The contract fit with space to spare. It just didn’t use what it could see.
That gap between what fits in an AI context window and what the model actually pays attention to is the most misunderstood idea in working with language models. The context window is the model’s working memory: every token it can see in one request. A bigger window lets you hand over more material. It doesn’t guarantee the model reads that material well, and past a certain point, more can make answers worse.
What an AI context window actually holds
A context window is measured in tokens, not words or pages. A token is a chunk of text, often a word or a piece of one. For English prose, a common rule of thumb is about four characters, or roughly three-quarters of a word, per token. Code, tables and many non-English languages usually take more tokens per word, so the same “page” costs more.
Frontier models now commonly offer windows from around 200,000 tokens up to 1 million. Anthropic’s current Claude models, for example, ship with a 1M-token context window as standard, while some older ones stop at 200,000. At three-quarters of a word per token, a million tokens is roughly 750,000 words. That is several long novels in a single prompt.
The number on the spec sheet is a shared budget, though, not free space for your documents. Everything in the request draws from it. Say you’re working with a 200,000-token window on a long research task:
- System prompt and instructions: 2,000 tokens
- Tool definitions (if the model can search, run code or call APIs): 6,000 tokens
- Conversation so far: 35,000 tokens
- Pasted documents and tool results: 120,000 tokens
- Room reserved for the answer: 8,000 tokens
That’s 171,000 tokens before you’ve typed your next question. The window isn’t full, but it’s crowded. And on most APIs, the tokens the model writes in its answer count against the same window, which is why a model can cut off mid-sentence when a prompt sits close to the limit.
The window is a desk, not a filing cabinet
The single most useful mental model: a context window is a desk, not a memory.
A language model has no recollection of your last request. Each call starts from nothing. When a chat app seems to remember the conversation, it is quietly resending the entire transcript every turn, so the model rereads everything from the top before it writes a single new word. “Memory” features work the same way under the hood: stored notes get pasted back into the context, where they take up desk space like anything else.
Two consequences follow from this.
First, when a conversation outgrows the window, something has to go. Apps handle this differently. Some drop the oldest messages, some summarize them, some refuse to continue. If an assistant suddenly forgets a constraint you set an hour ago, the likeliest explanation is that the message containing it is no longer on the desk.
Second, everything on the desk gets processed every time. A 150,000-token conversation means 150,000 tokens of reading on every turn, whether or not the new question needs any of it.
How a model reads everything at once
The mechanism that lets a model use its context is called attention. As the model works out each new token, attention lets it weigh every earlier token and decide how much each one matters right now. That’s how a model can connect a pronoun on page 80 to a name on page 2.
The detail that matters: within each attention step, those weights are normalized so they add up to a fixed total. Attention behaves less like a floodlight that can brighten everything and more like a fixed budget shared across everything in view. Add 100,000 tokens of loosely related material, and the relevant paragraph now competes with all of it for the same share.
This is a simplification. Real models stack many attention layers and heads, and they can learn to focus sharply. But it explains the core tension. A bigger window raises the ceiling on what the model can see. It doesn’t raise the amount of focus available per token.
There is also a physical cost. In a standard transformer, attention compute grows faster than prompt length: double the prompt and the attention work more than doubles. The memory needed to hold the model’s running notes on every token, the KV cache, grows with length too, which is why labs invest heavily in compressing it, as the DeepSeek-V4.1-Flash KV cache compression work shows.
Why a bigger context window can make answers worse
None of this means long context is broken. It means long context has failure modes that short prompts don’t, and they show up most when you fill the window just because you can.
The middle of the prompt gets less attention
A 2023 paper led by Stanford researchers, “Lost in the Middle,” gave models a question plus a set of documents, one of which held the answer. Accuracy was highest when the relevant document sat at the very start or very end of the context and dropped noticeably when it sat in the middle, producing a U-shaped curve.
Today’s models handle long inputs far better than the ones in that paper, but the problem hasn’t vanished. Anthropic’s own documentation warns that accuracy and recall degrade as token count grows, and calls the effect “context rot.” The practical upshot is simple: the more material you add, the more of it ends up in the middle.
Near-misses are worse than noise
Irrelevant filler costs you a little accuracy. The more dangerous material is the kind that looks like the answer but isn’t.
Chroma’s July 2025 “Context Rot” report tested 18 models and found that performance grew less reliable as input length increased, even on simple tasks. It also found that a single distractor, a passage on the same topic that doesn’t actually answer the question, was enough to lower accuracy, and that the damage grew as inputs got longer.
Think back to the contract. The late-delivery penalty clause is surrounded by a force-majeure clause, a warranty clause and a termination clause, all of which mention delays, penalties and dates. Paste in last year’s draft of the same contract as well, and every one of those clauses now has a near-twin with slightly different numbers. The problem is no longer just volume. It’s a set of plausible alternatives.
Advertised length is not usable length
A model accepting a million tokens tells you nothing about how well it reasons across a million tokens. NVIDIA’s RULER benchmark, published in 2024, tested 17 long-context models on harder tasks than simple retrieval, including multi-hop tracing and aggregation. Almost all of them showed large performance drops as context grew, and although every model claimed a window of at least 32,000 tokens, only about half still performed satisfactorily at that length.
The gap tends to widen with task difficulty. Finding one fact is easier than counting how many times something happens across a document, which is easier than reasoning over facts scattered through it.
Old instructions fade in long chats
In a long conversation, your opening instruction (“always answer in British English, never invent figures”) is one message among hundreds. Later turns may contradict it, intentionally or not, and if the app trims history, it may be gone entirely. Long chats also accumulate the model’s own earlier mistakes, which it can treat as established facts and build on.
That’s why a fresh conversation with a tight summary often outperforms the same conversation continued for another hour.
You pay for every token, every turn
Because the full context is resent on every turn, long contexts multiply cost. Say input costs $3 per million tokens (a hypothetical rate) and you run a 20-turn session:
| Context per turn | Input tokens over 20 turns | Input cost |
|---|---|---|
| 150,000 tokens | 3,000,000 | $9.00 |
| 20,000 tokens | 400,000 | $1.20 |
Some models also charge a higher per-token rate once a prompt crosses a length threshold. Google’s Gemini 3.1 Pro Preview, for example, doubles its input price above 200,000 tokens, and Anthropic’s Claude Haiku 5.5 charges more above 100,000. Prompt caching softens the hit when the same prefix repeats: major providers bill cached input tokens at a fraction of the normal rate, often around a tenth.
Latency follows the same curve. Before a model writes its first word, it has to process the whole prompt, a stage called prefill. Long prompts mean a longer wait for that first token, which is why research such as RBS-Attention’s claimed prefill speedup targets exactly that stage.
Why “perfect needle-in-a-haystack” scores mislead
Launch announcements love a green heatmap. The classic test, popularized by Greg Kamradt in November 2023, hides one sentence (the “needle”) at various depths in a long run of unrelated essays and asks the model to retrieve it. Many frontier models now score near-perfectly.
That’s real progress, but it measures the easiest version of the problem. The needle is usually unrelated to the surrounding text, and the question often shares exact words with it. Your actual work rarely looks like that. Your needle is a clause surrounded by similar clauses, and your question is phrased differently from the text that answers it.
Read a long-context score the way you’d read any benchmark: ask what it measures, and whether it resembles your task. The four-question method for reading AI benchmarks applies directly here.
When a huge window genuinely earns its keep
Long context is valuable when the task needs the whole picture at once, not just one fact from it.
- Cross-referencing within one document. “Do the payment terms in section 4 conflict with anything in the appendices?” requires seeing both at once. Chopping the document into pieces would hide the conflict.
- Code changes that touch many files. Renaming a data model or changing an API contract ripples across a codebase. A coding agent that can hold the relevant files makes fewer broken edits, which is a big part of why long-running builds like the SaaS MVP walkthrough with Claude Code depend on larger context.
- Synthesis across sources. Summarizing themes across 40 customer interviews works better when the model can compare them directly, rather than summarizing each one in isolation.
- One-off jobs. If you need an answer from a long document once, pasting it in beats building a search pipeline you’ll never reuse.
The common thread: in each case, most of what’s in the window is relevant to the answer. The trouble starts when most of it isn’t.
How to get better answers from long prompts
The fixes are mostly about curation and placement, and none require switching models.
Put documents first and the question last. Anthropic’s prompting guidance recommends placing long documents at the top of the prompt and the query at the end, and says queries at the end can improve response quality by up to 30% in its tests, especially with complex, multi-document inputs. Wrapping each document in clear labels (title, date, version) also helps the model tell near-twins apart.
Make the model find the evidence before answering. Asking for quotes first forces the model to locate the relevant passage instead of answering from a general impression:
Before answering, quote the exact passages from the contract
that are relevant to late-delivery penalties, with their clause
numbers. Then answer using only those quotes. If no passage
covers it, say so.As a bonus, the quotes let you check the answer in seconds.
Cut before you paste. Strip signature blocks, boilerplate, tables of contents, duplicate drafts and appendices you don’t need. Old versions of the same document are the worst offenders, because they’re the most convincing distractors.
Restart long chats on purpose. When a session runs long, ask the model for a summary of decisions, constraints and open questions, check it, and paste it into a fresh chat. You lose the clutter and keep the substance.
Use retrieval when most of your material is irrelevant to any given question. If you’re querying a knowledge base of 2,000 documents, search for the five that matter and pass only those. A big window is for material that belongs together, not a substitute for search.
Run your own needle test. Take a real document from your work, plant a specific fact you know (a fake invoice number, say) about halfway through, surrounded by similar-looking numbers, and ask for it with different wording. Try it at a few lengths. Ten minutes of this tells you more about a model’s usable context for your work than any spec sheet. If you’re comparing models for a specific job, the AI model selection framework covers how to fold tests like this into the decision.
Fill it with what the answer needs
Context window size tells you the ceiling. Answer quality depends on what you put under it.
Before pasting anything, apply one test: would a careful human expert need this page to answer the question? If yes, include it, and put it near the top or right before the question. If no, leave it out. It isn’t harmless padding. It’s a distractor you pay for on every turn.
