Local RAG Stack: Ollama + ChromaDB + LangChain

TL;DR: You can build a fully functional Retrieval-Augmented Generation (RAG) system that runs entirely on your local machine. By combining Ollama (for local LLM inference), ChromaDB (for local vector storage), and LangChain (for orchestration), your data never leaves your computer, and you pay zero API fees.

Running a RAG pipeline locally is the ultimate solution for privacy-sensitive AI applications. Whether you are analyzing proprietary codebases, confidential legal documents, or internal company wikis, a local RAG stack ensures your data is never sent to OpenAI, Anthropic, or any cloud provider.

In this tutorial, we will build a complete document Q&A application from scratch using Python.

The Stack at a Glance

Prerequisites

1. A Mac with Apple Silicon (M1+) or a PC with a decent GPU.
2. Python 3.10+ installed.
3. Ollama installed on your system.

First, start your terminal and pull the models we need using Ollama:

# Pull the language model for generating answers
ollama pull llama3.3

# Pull the embedding model for vectorizing documents
ollama pull nomic-embed-text

Next, install the required Python libraries:

pip install langchain langchain-community chromadb beautifulsoup4

Step 1: Load and Chunk Your Documents

For this example, let's pretend we have a directory of text files containing sensitive company policies. We need to load them and split them into smaller "chunks" so the LLM can digest them.

from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load documents from a local folder
loader = DirectoryLoader('./private_docs', glob='**/*.txt', loader_cls=TextLoader)
documents = loader.load()
print(f"Loaded {len(documents)} documents.")

# 2. Split them into manageable chunks (1000 characters each)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200, # Overlap prevents cutting sentences in half
)
chunks = text_splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks.")

Step 2: Generate Embeddings and Store in ChromaDB

Now we take those text chunks, convert them into mathematical vectors using Ollama's nomic-embed-text model, and save them to an embedded ChromaDB database on our hard drive.

from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

# Initialize the local embedding model
local_embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Create a local ChromaDB vector store and feed it the chunks
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=local_embeddings,
    persist_directory="./chroma_store" # Saves to disk so you don't have to re-embed later
)
print("Vectors stored successfully in ChromaDB.")

Because this runs locally, embedding thousands of documents might take a few minutes depending on your hardware, but you only have to do it once.

Step 3: Build the Retriever

The retriever's job is to take a user's question, turn it into a vector, and find the most mathematically similar document chunks in ChromaDB.

# Configure the retriever to return the top 4 most relevant chunks
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}
)

Step 4: Construct the RAG Chain

Finally, we use LangChain to wire everything together. We'll capture a user's question, retrieve the context, format it, and pass it to our local llama3.3 model to generate the final answer.

from langchain_community.llms import Ollama
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# Initialize the local LLM
llm = Ollama(model="llama3.3")

# Define the prompt template
template = """
Answer the question based ONLY on the following context. 
If the answer is not in the context, say "I don't know based on the provided documents."

Context:
{context}

Question: {question}

Answer:
"""
prompt = PromptTemplate.from_template(template)

# Helper function to format the retrieved documents into a single string
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Build the LangChain Expression Language (LCEL) chain
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Step 5: Ask Questions!

Your completely local, private, zero-cost AI is ready. Let's ask it a question:

question = "What is the company policy on remote work?"

print(f"\nThinking about: '{question}'...\n")
answer = rag_chain.invoke(question)
print(answer)

When you run this, you'll see the power of local AI in action. The LLM will use the precise text from your ./private_docs folder to formulate its answer, without a single byte of data leaving your machine.

Performance Tuning Tips for Local RAG

Running models locally requires managing your hardware resources:

1. RAM Limits: llama3.3 (8B parameters) requires about 8GB of RAM. If your system has 8GB totally, it will struggle. Switch to a smaller model like gemma3:4b (ollama pull gemma3:4b).
2. Chunk Size: If your answers feel "dumb" or lack context, try increasing the chunk_size in Step 1 to 1500 or 2000.
3. Context Window: Local models have finite context windows (often 8k tokens). If you retrieve too many documents (e.g., setting k=10 in Step 3), you might crash the model or cause it to hallucinate. Stick to k=3 or k=4.