Consider a typical failure. An employee asks the company’s AI chatbot a simple question with one right answer sitting in the handbook: how much parental leave she gets. It answers instantly, cites the handbook, and gets it wrong. It quoted last year’s policy, which still sits in the shared drive next to the new one. The AI model did nothing wrong. The search step failed.
That search step is the core of retrieval-augmented generation, or RAG. Here is RAG explained in one line: the AI doesn’t learn your documents. It looks things up in them every time you ask, then writes an answer from the handful of passages it found. Once you understand that, you understand why docs chatbots are brilliant on some questions and confidently wrong on others.
RAG explained as an open-book exam
A large language model learns from a huge pile of public text, up to a training cutoff. It has never seen your refund policy, your onboarding wiki or the PDF contract you signed in March. Ask it about them and it has two options: admit it doesn’t know, or guess. Guessing sounds fluent, which is the dangerous part.
There are two broad ways to close that gap. You can change the model itself by training it further on your material. Or you can leave the model alone and hand it the relevant pages at the moment it answers. RAG is the second approach.
Think of it as an open-book exam. The student is clever and writes well but has never read your binder. Before each question, an assistant flips through the binder, finds the three or four most relevant pages and places them on the desk. The student answers from those pages.
The three letters map onto that picture:
- Retrieval: search your documents for passages related to the question.
- Augmented: add those passages to the request sent to the model.
- Generation: the model writes an answer based on them.
The term comes from a 2020 research paper led by Patrick Lewis and a team at Facebook AI Research, with collaborators at University College London and New York University. Almost every “chat with your documents” product since then, from customer-support bots to internal knowledge assistants, runs some version of this loop.
Why not just give the AI all your files?
It is the obvious question, and there are real alternatives. Each one wins somewhere.
Training the model on your documents
Fine-tuning means running extra training on a model using your own material. It works well for teaching a style, a format or a specialized vocabulary. It works poorly for facts. The model absorbs your documents the way you absorb a book you skimmed once: roughly, with gaps, and with no way to point to the page. Every time a policy changes, you would need to train again. And when a fine-tuned model is wrong, it is wrong with the same confidence as when it is right.
Pasting everything into one giant prompt
Context windows have grown enormously. Some models now accept around a million tokens in a single request, as with NaiveAI’s open-weight model with a native 1M-token context and Anthropic’s current Claude models. A token is a chunk of text a bit shorter than a word; OpenAI’s rule of thumb is roughly 75 English words per 100 tokens, or about 750 words per 1,000.
So why not dump the whole handbook in? Run the numbers with a hypothetical setup. Say your company knowledge base is 1,000 pages at about 500 words a page. That is 500,000 words, or roughly 670,000 tokens. Now say your model charges $2 per million input tokens. Every single question costs about $1.33 just to send the documents. At 2,000 questions a month, that is about $2,670, mostly spent re-sending the same pages over and over.
With RAG, each question sends the question plus maybe five short passages, say 4,000 tokens. Same hypothetical price: less than a cent per question.
Cost is only half of it. Models also tend to pay less attention to information buried in the middle of very long inputs, a pattern a 2023 Stanford-led study named “lost in the middle.” A short, focused prompt gives the model less to get distracted by.
Long context still wins in one common case: a single document. If you want to question one 60-page contract, paste the contract. No search system needed, and nothing can be missed.
RAG: look it up each time
RAG trades those problems for a new one. The answer can only be as good as the search. If the right passage isn’t retrieved, the model cannot use it. That single fact explains most of what follows.
Before anyone asks a question: chopping documents into chunks
The work starts long before the first question. Your documents go through a preparation step, often called ingestion:
- Text is extracted from each file.
- That text is split into chunks, usually a few paragraphs each.
- Each chunk is labeled and stored in a searchable index.
Step one sounds trivial and isn’t. A clean Word document extracts fine. A scanned PDF is just a picture of text until something reads it. Tables are worse: a pricing table can come out as a jumble of numbers with the column headers detached. If the extraction garbles a page, the chatbot inherits garbage it will quote with a straight face. Document parsing is a whole problem of its own, which is why benchmarks like ParseBench, which tests whether your PDF parser is lying to your AI agent, exist at all.
Step two involves a real trade-off. Imagine a handbook paragraph that states a rule, followed by a sentence: “This does not apply to contractors.” If the chunk boundary falls between them, the search can retrieve the rule without the exception. The model then gives a contractor the wrong answer, and nobody can tell why from the outside.
Make chunks too small and they lose their context like that. Make them too large and each chunk covers several topics, so it matches many questions loosely and none precisely. Most systems split at a few hundred words, overlap neighboring chunks slightly so sentences at the edges appear in both, and try to respect headings rather than cutting mid-section.
Step three is where good systems quietly earn their keep. Each chunk gets labels: document title, section, date, maybe department or access level. Those labels let the system do things plain text search can’t, such as “only search policies marked current” or “never show this chunk to people outside finance.” Remember the parental-leave bot from the top of this article? A date label and one filter would have saved it.
How does RAG search for meaning, not just words?
Old-school search matches words. Type “parental leave” and you find pages containing “parental leave.” The employee from our opening, though, typed “how much time off do I get after my baby is born?” Not one important word overlaps with the handbook’s heading.
Most RAG systems solve this with embeddings. An embedding model turns any piece of text into a long list of numbers. You can think of those numbers as coordinates on an enormous map of meaning. Texts about similar things land near each other on the map, even when they share no words. “Time off after my baby is born” and “parental leave entitlement” end up as neighbors. “Parking permit renewal” ends up somewhere else entirely.
When a question arrives, it gets turned into coordinates too. The system then grabs the chunks closest to it on the map, typically the top five or so. That is the “search picks top chunks” step in the diagram.
Meaning-based search has a blind spot: exact strings. Product codes, error numbers, invoice IDs and people’s names don’t carry much “meaning” in the map sense. Ask about “error E-4021” and the closest chunks might be about other errors that sound similar. Plain keyword search would have found the exact match instantly.
That is why many serious systems run both searches and merge the results, an approach usually called hybrid search. Some add a second pass, a reranker, which takes the top 20 or 30 candidates and reorders them by reading each one against the question more carefully. If a vendor’s search is meaning-only, test it with your product codes before you sign anything.
What the model actually sees
Here is the part most explanations skip. After retrieval, the system assembles a prompt and sends it to the model. A simplified version looks something like this:
You are the HR assistant for Acme Ltd. Answer using ONLY the
sources below. If the sources do not contain the answer, say
you don't know. Cite source numbers.
[Source 1] Employee Handbook 2026, Section 4.2 (updated March 2026)
Employees with 12 months' service receive 16 weeks of paid
parental leave. This applies to all parents regardless of gender...
[Source 2] Employee Handbook 2026, Section 4.3
Requests for parental leave should be submitted to your manager
at least eight weeks before the expected start date...
[Source 3] Benefits FAQ (2025)
...
Question: How much time off do I get after my baby is born?That is the whole world, as far as the model is concerned. It never saw the other 999 pages. It doesn’t know Source 3 is stale unless the label tells it. If the policy on adoptive parents lives in a chunk that wasn’t retrieved, it doesn’t exist for this answer.
Notice the instructions at the top too. “Use only the sources” and “say you don’t know” are the main guardrails against the model filling gaps with general knowledge. They work most of the time, not all of the time. A model asked about leave in a UK company might still blend in what it knows about UK statutory leave, especially when the retrieved passages are vague.
Where RAG answers go wrong, and how to tell which half failed
Every bad RAG answer comes from one of two places: the search fetched the wrong material, or the model mishandled the right material. The fixes are completely different, so diagnosis matters.
Search failures
- The right passage never surfaced. The wording didn’t match, the PDF didn’t parse, or the answer sat in a chunk ranked sixth when only five were sent.
- An outdated version won. Old and new policies look almost identical on the meaning map. Without date labels or cleanup, the search has no reason to prefer the current one.
- Half a rule came back. The chunking problem from earlier: rule retrieved, exception left behind.
- The answer is spread across everything. “How many of our client contracts renew in January?” No single passage contains that. Answering requires reading every contract and counting. RAG pulls a few chunks, so the model counts the three contracts it was shown and reports a number that looks plausible and is wrong.
Model failures
- Blending in outside knowledge when the sources are thin, as in the UK leave example.
- Misreading a table that survived extraction in a mangled form.
- Answering anyway when it should have said the sources don’t cover the question.
The ten-second diagnosis
Look at what was retrieved. Most decent tools can show you the passages behind an answer. If the correct passage isn’t among them, you have a search problem: fix the documents, the chunking, the labels or the search method. Swapping in a smarter model won’t help, because no model can quote a page it never received.
If the correct passage is there and the answer is still wrong, you have a model or prompt problem. That is where a stronger model or tighter instructions pay off.
Check search first. It fails silently, it is usually cheaper to fix, and teams tend to blame the model by default.
Citations are what make RAG trustworthy
The single biggest advantage RAG has over a plain chatbot is that it can show its sources. Each claim can link back to the passage it came from, so a reader can check the answer in seconds instead of trusting it.
That changes how you should use these tools. For low-stakes questions, such as where the brand guidelines live, a quick glance is enough. For anything with consequences, such as legal terms, prices quoted to customers or medical or financial information, the citation is the actual product. The generated sentence is just a convenient summary of it.
A docs chatbot that won’t show its sources is like a confident new hire who never brings notes to a meeting. They might be right. You have no fast way to know.
Five questions to ask before you buy or build a docs chatbot
Every vendor demo uses clean documents and friendly questions. These five questions get past the demo.
- What happens to old versions of a document? Look for automatic replacement on re-upload, date labels, or a way to mark documents as superseded. “You should delete them yourself” is an honest answer, but it is also a job someone now owns.
- Can I see exactly which passages each answer used? Without this, you can’t run the ten-second diagnosis, and every failure turns into guesswork.
- How does it handle scanned PDFs, tables and slide decks? Ask them to ingest your three ugliest files during the trial, not their samples.
- Does retrieval respect permissions? If the salary spreadsheet is in the index, a sales rep’s innocent question can surface it. Access control must apply at the search step, before anything reaches the model.
- Which model writes the answers, and can I change it? Models improve and prices drop. Being locked to one is a cost you’ll feel later. Our framework for choosing which frontier model fits which job applies here too. If documents can’t leave your servers, look at how far open-weight models have caught up with closed ones, because a self-hosted model plus self-hosted search is a fully private RAG setup.
Then run your own test. Write down 30 real questions your team actually asks, with the correct answers and the page each one lives on. Include a few with exact codes, a couple that need an exception clause, and one or two whose honest answer is “the documents don’t say.” Run them through every option you’re considering. That small homegrown test set will tell you more than any vendor’s accuracy claim, for the same reasons covered in how to read an AI benchmark without being fooled: it matches your workload and nobody has trained on it.
When RAG is the wrong tool
RAG is a lookup machine. It fits best when the answer sits in one place, or a few places, that someone could point to. It struggles whenever answering means reading everything.
- One or two documents? Paste them into a long-context model. Simpler, and nothing can be missed.
- Want the AI to write in your house style? That is a prompting or fine-tuning job, not retrieval.
- Counting, totaling or comparing across hundreds of records? Put the data in a spreadsheet or database and query it. Some tools combine this with an AI layer that writes the query for you, which is a different design from RAG.
- Answers live in hundreds of documents that change often, and people need to check sources? This is exactly the job RAG was built for.
One last decision rule, and it costs nothing. Before you pick any tool, open the folder you plan to feed it and delete everything superseded, duplicated or wrong. A search engine can only rank what you give it, and the parental-leave bot didn’t need a better model. It needed last year’s handbook gone. The cheapest RAG upgrade you will ever make is the delete key.
