What is RAG?
RAG has a language model retrieve relevant documents from your own sources before it answers. How it works, when it beats fine-tuning, and where it breaks in production.
Last updated:
RAG — retrieval-augmented generation — has a language model fetch relevant documents from external sources and answer from those documents rather than from its own recall. The consequence that matters commercially: the model can answer using information it has never seen, information that is confidential, or information that changed yesterday.
How it works
Four steps, each carrying its own quality question.
- Chunking. Documents are split into retrievable pieces. Too small and context is severed; too large and irrelevant text drowns the answer.
- Embedding. Each chunk becomes a vector in a database. The embedding model matters more than it looks — performance on Turkish content differs from English.
- Retrieval. When a question arrives, the nearest chunks are pulled. This step decides system quality more than any other: given the wrong passage, the best model available still cannot answer correctly.
- Generation. The retrieved chunks go to the model as context, and citation becomes possible here.
In practice most of the engineering time goes into steps one and three. Model choice, contrary to expectation, is the least decisive call.
When RAG, and when not
| Situation | Right approach |
|---|---|
| The knowledge changes often | RAG |
| Answers must cite a source | RAG |
| Data is confidential and must stay in-house | RAG, self-hosted |
| A particular tone or format must be taught | Fine-tuning |
| General knowledge answers the question | Plain LLM, no retrieval |
| One database query would answer it | A query, not RAG |
That last row gets skipped often. "Which customer has an unpaid invoice" is a SQL question, not a retrieval problem. RAG is for answers buried inside unstructured text.
Where it breaks in production
Systems that work in a pilot and degrade in real use tend to fail in the same three places.
Document hygiene. The most common problem we meet is not technical: three versions of the same policy with no indication which is current. The system answers from whichever it retrieves. Cleaning the source corpus before building is frequently the most valuable part of the project.
Retrieval quality. Users do not phrase questions in the document's language. "How do I request leave" may live inside an "Annual Paid Absence Procedure" that never uses the word. Fixing this means rewriting queries, combining keyword search with vector search, and reranking what comes back.
Permission boundaries. In an organisation where not everyone may read every document, retrieval must respect that too. Access control belongs inside the retrieval step from the start — added later, the system leaks in the interval.
Where it is used
The strongest fits are internal documentation assistants, customer support knowledge bases and contract analysis. From our own work, RAG support in healthcare training shows the pattern built inside a regulated domain.
Pairing RAG with an AI agent is common: the agent decides which source a question belongs to, RAG grounds the answer in the document. They are complementary rather than competing.
Scope, timelines and price bands are on the knowledge assistant page.
Frequently asked
- What is the difference between RAG and fine-tuning?
- Fine-tuning changes the model's weights and suits teaching a style or a format. RAG leaves the model alone and hands it documents at answer time. Corporate knowledge almost always belongs to RAG: when a document changes you update the source rather than retrain the model.
- Does RAG eliminate hallucination?
- It reduces it rather than removing it. Given the right passage, a model is far more likely to ground its answer in it — but a wrong passage, or a question outside the corpus, can still produce invention. The most effective safeguards are showing the user which document an answer came from and instructing the model to say it does not know when the source is silent.
- Can RAG run without our data leaving our infrastructure?
- Yes. The vector database and the orchestration layer can run on your own servers. The decision point is the language model: a self-hosted model keeps data entirely in-house, while an external API means limiting what enters the prompt and reading the provider's data processing terms.
- How many documents does RAG need?
- There is no lower bound; a few hundred pages of handbook is already worth retrieving over. What decides quality is not volume but currency and consistency. Put two contradictory procedures in the same corpus and the system learns the contradiction too.
More insights
- What is AI workflow automation?Adding a decision layer to classic automation. How it differs from rule-based flows, the architecture it usually takes, and the processes where it does and does not pay.
- What is an AI agent?An AI agent decides for itself which tool to use and when, in service of a goal. How it differs from a chatbot and from rule-based automation, when it earns its cost and when it does not.
- What is n8n?n8n automates business processes through a visual workflow editor and can run on your own servers. Why its licence is not open source, how its billing unit differs, and where it wins.
Let's apply this to your own processes
If you want to move from definition to implementation, we'll look at your current process and work out where to start.
Tell us about your project