LangChain RAG Tutorial in Python

A practical LangChain RAG tutorial in Python covering document loading, chunking, embeddings, vector storage, retrieval, and answer generation.

By AI Mastermind Lab · Published 8 Mar 2026

TL;DR: Retrieval-Augmented Generation (RAG) connects an LLM to your own documents so it can answer questions with accurate, cited information instead of hallucinating. This tutorial walks you through building a complete RAG pipeline in Python using LangChain v0.3 and LCEL — from loading documents to generating grounded answers.

Building a LangChain RAG tutorial in Python is one of the most practical skills in modern AI engineering. RAG lets you give any LLM access to your private documents — PDFs, code repos, knowledge bases — without fine-tuning. The model retrieves relevant chunks at query time and uses them as context to generate accurate, verifiable answers.

By the end of this tutorial, you will have a working RAG application that loads documents, chunks them, embeds them into a vector store, retrieves relevant passages, and generates answers with citations.

Prerequisites

  • Python 3.10+
  • Basic familiarity with Python and LLMs
  • An OpenAI API key (or a running Ollama instance for local inference)
pip install langchain langchain-openai langchain-community chromadb

What Is RAG and Why Use It?

Retrieval-Augmented Generation (RAG) is a pattern where an LLM's response is grounded in retrieved documents rather than relying solely on its training data. RAG solves three critical problems with vanilla LLMs:

  1. Knowledge cutoff — LLMs do not know about events after their training date. RAG gives them access to current information.
  2. Hallucination — Without source material, LLMs confidently fabricate facts. RAG provides verifiable context.
  3. Private data — Your internal docs never need to leave your infrastructure. RAG queries them on demand.

LangChain RAG Architecture Overview

The RAG pipeline has five stages:

  1. Load — Read documents from files, URLs, or databases
  2. Chunk — Split documents into smaller, semantically meaningful pieces
  3. Embed — Convert each chunk into a numeric vector (embedding)
  4. Store — Save embeddings in a vector database for fast similarity search
  5. Retrieve + Generate — At query time, find the most relevant chunks and pass them to the LLM as context

Step 1: Load Your Documents

LangChain provides dozens of document loaders. Here we load a directory of text files:

from langchain_community.document_loaders import DirectoryLoader, TextLoader

# Load all .txt files from a directory
loader = DirectoryLoader(
    './docs/',
    glob='**/*.txt',
    loader_cls=TextLoader
)
documents = loader.load()
print(f'Loaded {len(documents)} documents')

For PDFs, swap in PyPDFLoader. For web pages, use WebBaseLoader. LangChain supports 50+ loaders out of the box.

Step 2: Chunk Documents

LLMs have limited context windows, and retrieval works best on focused passages. Split your documents into chunks of 500–1000 tokens with slight overlap to preserve context across boundaries:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # Characters per chunk
    chunk_overlap=200,    # Overlap between chunks for context continuity
    length_function=len,
    separators=['\n\n', '\n', ' ', '']  # Split on paragraphs first, then sentences
)

chunks = text_splitter.split_documents(documents)
print(f'Split into {len(chunks)} chunks')

Why RecursiveCharacterTextSplitter? It tries to split on natural boundaries (paragraphs, sentences) before falling back to raw character splits. This preserves semantic coherence within each chunk — critical for retrieval quality.

Step 3: Create Embeddings and Vector Store

Convert each chunk into a vector embedding and store it in ChromaDB (a lightweight, local vector database):

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Initialize the embedding model
embeddings = OpenAIEmbeddings(model='text-embedding-3-small')

# Create a persistent vector store
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory='./chroma_db'  # Saves to disk
)
print(f'Stored {len(chunks)} chunks in ChromaDB')

Critical rule: The embedding model used for indexing MUST be the exact same model used for querying. Mixing models (e.g., indexing with text-embedding-3-small but querying with text-embedding-ada-002) will produce garbage results because the vector spaces are different.

Using Ollama for Free, Local Embeddings

If you want to avoid API costs entirely, use a local embedding model via Ollama:

from langchain_community.embeddings import OllamaEmbeddings

embeddings = OllamaEmbeddings(model='nomic-embed-text')

Step 4: Build the Retriever

A retriever searches the vector store and returns the most relevant chunks for a given query:

# Create a retriever that returns the top 4 most similar chunks
retriever = vectorstore.as_retriever(
    search_type='similarity',
    search_kwargs={'k': 4}
)

# Test it
results = retriever.invoke('What is the refund policy?')
for doc in results:
    print(doc.page_content[:200])
    print('---')

Tuning k: Start with k=4. Increase to 6–8 for complex questions that need more context. Decrease to 2–3 if you have a small context window or find irrelevant chunks creeping in.

Step 5: Build the RAG Chain with LCEL

LangChain Expression Language (LCEL) lets you compose the full RAG pipeline as a declarative chain. This is the modern, recommended approach in LangChain v0.3:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# Initialize the LLM
llm = ChatOpenAI(model='gpt-4o', temperature=0)

# Define the RAG prompt
prompt = ChatPromptTemplate.from_template("""
Answer the question based ONLY on the following context. 
If the context does not contain the answer, say 'I don't have enough information to answer that.'
Always cite which part of the context supports your answer.

Context:
{context}

Question: {question}

Answer:
""")

# Helper to format retrieved documents
def format_docs(docs):
    return '\n\n'.join(doc.page_content for doc in docs)

# Build the LCEL chain
rag_chain = (
    {'context': retriever | format_docs, 'question': RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Run it
answer = rag_chain.invoke('What is the return policy for electronics?')
print(answer)

How the chain works: The user's question is passed to the retriever (which returns relevant documents) AND forwarded as the question variable. The retrieved docs are formatted into a string for the context variable. Both are injected into the prompt template, which is sent to the LLM. The output parser extracts the text response.

Using Ollama Instead of OpenAI

To run the entire pipeline locally for free, replace the LLM:

from langchain_community.llms import Ollama

llm = Ollama(model='llama3.3')

Combined with OllamaEmbeddings, you get a fully local, zero-cost RAG system.

Step 6: Evaluate Retrieval Quality

Do not guess — measure. Build a small evaluation set with known questions and expected answers:

eval_set = [
    {'question': 'What is the refund policy?', 'expected_source': 'refund_policy.txt'},
    {'question': 'How do I reset my password?', 'expected_source': 'account_help.txt'},
]

for item in eval_set:
    results = retriever.invoke(item['question'])
    sources = [doc.metadata.get('source', '') for doc in results]
    hit = any(item['expected_source'] in s for s in sources)
    print(f"Q: {item['question']} | Hit: {hit} | Sources: {sources}")

Track these metrics:

  • Hit@k — Did the correct document appear in the top-k results?
  • MRR (Mean Reciprocal Rank) — How high was the correct document ranked?
  • Answer quality — Did the LLM produce a correct, grounded answer?

For deeper evaluation methodologies, see our guide on Evaluating Retrieval Quality in RAG.

Common Mistakes and How to Avoid Them

Chunks too large (2000+ chars): Retrieval returns bloated, semi-relevant passages. The LLM has to sift through noise. Keep chunks at 500–1000 characters.

Chunks too small (< 200 chars): Context is lost. A chunk like "See section 3 for details" is useless without section 3. Use overlap to mitigate this.

No overlap between chunks: Sentences that span a chunk boundary are split and lost. Always use 10–20% overlap.

Embedding model mismatch: Re-embedding your corpus when you switch models is expensive. Plan your embedding model choice upfront and version your embeddings.

Skipping evaluation: Without a golden evaluation set, you are guessing. Build one with 50–100 examples and run it on every pipeline change.

When to Use LangGraph Instead

This tutorial covers a standard RAG chain — one retrieval step, one generation step. For more complex patterns, consider LangGraph:

  • Agentic RAG: The LLM decides whether to retrieve more documents, reformulate the query, or use a tool
  • Multi-step retrieval: First retrieve from source A, then use the results to query source B
  • Human-in-the-loop: Pause the pipeline for human review before generating the final answer
  • Conditional routing: Route different query types to different retrievers or knowledge bases

FAQ

Q: What is the best embedding model for RAG? A: For most English-language use cases, OpenAI's text-embedding-3-small offers the best balance of quality and cost. For local or free embeddings, nomic-embed-text via Ollama is excellent. For multilingual needs, use bge-m3.

Q: How much does it cost to run a RAG pipeline? A: Embedding costs are one-time per document (fractions of a cent per page with OpenAI). LLM generation costs depend on usage. For zero cost, run everything locally with Ollama and ChromaDB.

Q: Can I use RAG with a local model instead of OpenAI? A: Yes. Replace both the embedding model and the LLM with Ollama-backed versions. The pipeline works identically — just point to localhost:11434.

Q: RAG vs fine-tuning — when do I use which? A: Use RAG for fresh, fast-changing knowledge and verifiable answers. Use fine-tuning when you need the model to adopt a new behavior, output format, or domain style. Most production systems in 2026 use both — see RAG vs Fine-Tuning.

Next Steps