Retrieval-augmented generation (RAG) is a way of building AI assistants that look up relevant information in a trusted source, such as your company documents, before writing an answer. Instead of relying only on what a language model memorised during training, a RAG system retrieves passages first and uses them as context, so answers can be more specific, more current and easier to check.

For businesses, RAG offers a practical way to connect large language models (LLMs) to their own knowledge. This guide explains how it works, when to choose it over fine-tuning, the risks to manage and a practical checklist for getting started.

Where does the term RAG come from?

The term was introduced in a 2020 research paper by Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”. The authors combined two kinds of memory: the knowledge stored inside a pre-trained language model (its “parametric” memory) and an external, searchable index of documents (a “non-parametric” memory), in their case a dense vector index of Wikipedia accessed by a neural retriever.

The paper reported that, for language generation tasks, RAG models produced more specific, diverse and factual language than a comparable model without retrieval. It also showed that a model’s knowledge could be updated by swapping the document index, without retraining the model.

How does retrieval-augmented generation work?

Modern RAG systems vary in detail, but most follow the same basic steps.

1. Prepare the knowledge base

Documents such as policies, product manuals, help-centre articles or reports are cleaned and split into smaller chunks. Each chunk is converted into an embedding, a list of numbers that captures its meaning, and stored in a search index, often a vector database.

2. Retrieve relevant passages

When a user asks a question, the question is also turned into an embedding. The system searches the index for the chunks that are closest in meaning, and some systems combine this with traditional keyword search.

3. Generate a grounded answer

The most relevant passages are added to the prompt alongside the user’s question. The language model is instructed to answer using that context, and ideally to cite which documents it used and to say when the answer is not in the sources.

When should you use RAG instead of fine-tuning?

Fine-tuning means training a model further on your own examples so that its behaviour changes. RAG leaves the model as it is and changes what it can read at the moment of answering. They solve different problems, and some teams use both.

RAG is usually a better fit when:

  • your information changes often, such as prices, policies or stock levels, because you can update documents without retraining;
  • users need answers they can verify, because the system can point to its sources;
  • different users are allowed to see different information, because access rules can be applied at retrieval time;
  • you want to start quickly with existing documents.

Fine-tuning is usually a better fit when:

  • you need the model to follow a specific style, tone or output format consistently;
  • you are teaching a skill or a pattern of reasoning rather than facts;
  • you have a good set of high-quality examples of the behaviour you want.

A useful rule of thumb: use RAG to give a model the right knowledge, and consider fine-tuning to shape its behaviour.

What are the risks of RAG?

RAG reduces some problems but does not remove them. Plan for these risks from day one.

Hallucination does not disappear

The US National Institute of Standards and Technology calls false but confident AI output “confabulation”, noting in its Generative AI Profile (NIST AI 600-1) that such outputs are colloquially known as “hallucinations”. A RAG system can still misread a passage, blend two sources incorrectly or answer from the model’s own memory when retrieval finds nothing useful.

Stale or poor-quality data

A RAG system is only as good as what it retrieves. Outdated documents, duplicates, contradictory versions and badly scanned files lead to wrong answers delivered with confidence. If nobody owns the content, quality will drift.

Access control and data leakage

The OWASP Top 10 for LLM Applications lists vector and embedding weaknesses as a security risk for RAG systems. It warns that inadequate access controls can let a model retrieve and disclose sensitive information, and that where several user groups share one vector database, information can leak between them. OWASP recommends permission-aware vector stores and strict partitioning of data.

Poisoned or manipulated content

OWASP also describes data poisoning, including an example of hidden text in a submitted document that tries to give the AI instructions. Validating documents before they enter the knowledge base, accepting data only from trusted sources and keeping logs of retrieval activity all help.

Checklist: is your business ready for RAG?

  • Define the use case: write down the questions the assistant should answer and who will use it.
  • Choose trusted sources: start with a small, well-maintained set of documents and name an owner for each.
  • Clean and version content: remove outdated and duplicate files, and record when each document was last reviewed.
  • Enforce permissions: make sure retrieval respects the same access rules as your existing systems.
  • Validate inputs: check new documents before indexing, especially anything submitted by outsiders.
  • Show sources: display citations so users can verify answers.
  • Allow “I don’t know”: instruct the system to say when the answer is not in the sources.
  • Test and monitor: build a set of real test questions, review answers regularly and log retrieval activity.
  • Keep a human in the loop for decisions with legal, financial or safety consequences.

Grounding matters even more as AI moves from answering questions to taking actions; see our guide to agentic commerce and how AI agents pay.

Frequently asked questions

What does RAG stand for in AI?

RAG stands for retrieval-augmented generation: an AI system retrieves relevant information from an external source and uses it to generate its answer.

Does RAG stop AI hallucinations?

No. It can make answers more grounded and easier to check, but a model can still misinterpret or ignore retrieved text. Citations, testing and human review remain important.

Do I need a vector database for RAG?

Not always. Vector search is a common choice, but retrieval can also use keyword search or a combination of both. The right choice depends on your data and the questions users ask.

Is RAG better than fine-tuning?

Neither is better in general. RAG suits changing, verifiable knowledge, while fine-tuning suits consistent style and behaviour. Some teams combine them.

At Codelude, we build AI-powered products and think carefully about how they find, use and protect information.

Leave a Reply