# Local RAG Stack: Ollama + ChromaDB + LangChain

> Build a completely private, zero-cost Retrieval-Augmented Generation pipeline using Ollama for local LLMs, ChromaDB for vector storage, and LangChain.

Published: 2026-03-08  
Canonical: https://aimastermindlab.com/tutorials/local-rag-stack-ollama-chromadb-langchain

---
> **TL;DR:** You can build a fully functional Retrieval-Augmented Generation (RAG) system that runs entirely on your local machine. By combining **Ollama** (for local LLM inference), **ChromaDB** (for local vector storage), and **LangChain** (for orchestration), your data never leaves your computer, and you pay zero API fees.

Running a RAG pipeline locally is the ultimate solution for privacy-sensitive AI applications. Whether you are analyzing proprietary codebases, confidential legal documents, or internal company wikis, a local RAG stack ensures your data is never sent to OpenAI, Anthropic, or any cloud provider.

In this tutorial, we will build a complete document Q&A application from scratch using Python.

## The Stack at a Glance

| Component | Tool | Purpose |
|-----------|------|---------|
| **LLM Inference** | [Ollama](/tutorials/run-llm-locally-mac-ollama) | Runs the massive language model (e.g., Llama 3) locally |
| **Embeddings** | Ollama (`nomic-embed-text`) | Converts your text documents into mathematical vectors locally |
| **Vector Database** | [ChromaDB](/knowledge-base/chromadb-tutorial-python) | Stores the vectors and performs ultra-fast similarity search |
| **Orchestration** | LangChain | Ties everything together (loading docs, chunking, prompting) |

## Prerequisites

1. A Mac with Apple Silicon (M1+) or a PC with a decent GPU.
2. Python 3.10+ installed.
3. **Ollama** installed on your system.

First, start your terminal and pull the models we need using Ollama:

```bash
# Pull the language model for generating answers
ollama pull llama3.3

# Pull the embedding model for vectorizing documents
ollama pull nomic-embed-text
```

Next, install the required Python libraries:

```bash
pip install langchain langchain-community chromadb beautifulsoup4
```

## Step 1: Load and Chunk Your Documents

For this example, let's pretend we have a directory of text files containing sensitive company policies. We need to load them and split them into smaller "chunks" so the LLM can digest them.

```python
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load documents from a local folder
loader = DirectoryLoader('./private_docs', glob='**/*.txt', loader_cls=TextLoader)
documents = loader.load()
print(f"Loaded {len(documents)} documents.")

# 2. Split them into manageable chunks (1000 characters each)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200, # Overlap prevents cutting sentences in half
)
chunks = text_splitter.split_documents(documents)
print(f"Split into {len(chunks)} chunks.")
```

## Step 2: Generate Embeddings and Store in ChromaDB

Now we take those text chunks, convert them into mathematical vectors using Ollama's `nomic-embed-text` model, and save them to an embedded ChromaDB database on our hard drive.

```python
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

# Initialize the local embedding model
local_embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Create a local ChromaDB vector store and feed it the chunks
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=local_embeddings,
    persist_directory="./chroma_store" # Saves to disk so you don't have to re-embed later
)
print("Vectors stored successfully in ChromaDB.")
```

Because this runs locally, embedding thousands of documents might take a few minutes depending on your hardware, but you only have to do it once.

## Step 3: Build the Retriever

The retriever's job is to take a user's question, turn it into a vector, and find the most mathematically similar document chunks in ChromaDB.

```python
# Configure the retriever to return the top 4 most relevant chunks
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4}
)
```

## Step 4: Construct the RAG Chain

Finally, we use LangChain to wire everything together. We'll capture a user's question, retrieve the context, format it, and pass it to our local `llama3.3` model to generate the final answer.

```python
from langchain_community.llms import Ollama
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

# Initialize the local LLM
llm = Ollama(model="llama3.3")

# Define the prompt template
template = """
Answer the question based ONLY on the following context. 
If the answer is not in the context, say "I don't know based on the provided documents."

Context:
{context}

Question: {question}

Answer:
"""
prompt = PromptTemplate.from_template(template)

# Helper function to format the retrieved documents into a single string
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Build the LangChain Expression Language (LCEL) chain
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
```

## Step 5: Ask Questions!

Your completely local, private, zero-cost AI is ready. Let's ask it a question:

```python
question = "What is the company policy on remote work?"

print(f"\nThinking about: '{question}'...\n")
answer = rag_chain.invoke(question)
print(answer)
```

When you run this, you'll see the power of local AI in action. The LLM will use the precise text from your `./private_docs` folder to formulate its answer, without a single byte of data leaving your machine.

## Performance Tuning Tips for Local RAG

Running models locally requires managing your hardware resources:

1. **RAM Limits**: `llama3.3` (8B parameters) requires about 8GB of RAM. If your system has 8GB totally, it will struggle. Switch to a smaller model like `gemma3:4b` (`ollama pull gemma3:4b`).
2. **Chunk Size**: If your answers feel "dumb" or lack context, try increasing the `chunk_size` in Step 1 to 1500 or 2000. 
3. **Context Window**: Local models have finite context windows (often 8k tokens). If you retrieve too many documents (e.g., setting `k=10` in Step 3), you might crash the model or cause it to hallucinate. Stick to `k=3` or `k=4`.
---

Source: https://aimastermindlab.com/tutorials/local-rag-stack-ollama-chromadb-langchain — AI Mastermind Lab (https://aimastermindlab.com)