RAG (Retrieval-Augmented Generation)

RAG is a technique that combines an LLM with an external knowledge retrieval system. Instead of relying solely on its training data, the model retrieves relevant documents at query time and uses them as context to generate more accurate, up-to-date responses.

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) is an AI architecture that enhances large language models by connecting them to external knowledge sources. Rather than generating answers purely from memorized training data, a RAG system first retrieves relevant chunks of information from a database, then augments the model's prompt with that context before generating a response.

How RAG Works

  1. Query encoding — The user's question is converted into a vector embedding.
  2. Retrieval — The embedding is used to search a vector database (e.g. Pinecone, ChromaDB) for the most semantically similar document chunks.
  3. Augmentation — The retrieved chunks are injected into the LLM's prompt as context.
  4. Generation — The LLM generates a response grounded in the retrieved context.

Why RAG Matters

RAG solves three critical problems with standalone LLMs:

  • Knowledge cutoffs — LLMs have a training cutoff date. RAG gives them access to current information.
  • Hallucination — Grounding responses in retrieved facts significantly reduces confabulation.
  • Private data — You can connect the model to your own proprietary documents without fine-tuning.

RAG vs Fine-tuning

Fine-tuning bakes knowledge into model weights permanently. RAG keeps knowledge external, versioned, and updatable. For most enterprise use cases, RAG is faster, cheaper, and more maintainable than fine-tuning.

Common RAG Frameworks

  • LangChain — Most popular; has extensive RAG chain abstractions.
  • LlamaIndex — Optimised specifically for document retrieval pipelines.
  • Haystack — Production-grade, especially for search-heavy use cases.

Example

A customer support bot uses RAG to retrieve the 5 most relevant paragraphs from your internal documentation before answering. This prevents hallucinated answers and keeps responses grounded in your actual product docs.

Related terms

  • Embedding An embedding is a numerical vector representation of data (text, image, audio) that captures semantic meaning. Similar concepts have embeddings that are close together in vector space, enabling AI systems to measure similarity, cluster data, and retrieve relevant information.
  • Vector Database A vector database stores and retrieves high-dimensional vector embeddings efficiently. It enables semantic similarity search — finding documents, images, or records that are semantically 'nearest' to a query vector. Examples include Pinecone, Weaviate, Qdrant, and ChromaDB.
  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • Hallucination Hallucination is when an AI language model generates plausible-sounding but factually incorrect or entirely fabricated information. The model 'hallucinates' facts, citations, names, or events that do not exist, presenting them with false confidence.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.