# Document Chunking Strategies for RAG: Which Works Best? > The size and strategy of your document chunks define the quality of your RAG pipeline. Compare fixed-size, sentence-aware, markdown-structure, and semantic chunking. Published: 2026-03-08 Canonical: https://aimastermindlab.com/knowledge-base/document-chunking-strategies-rag --- In a Retrieval-Augmented Generation (RAG) pipeline, **chunking** is the process of breaking down massive documents (like a 100-page PDF) into smaller, bite-sized pieces before turning them into mathematical embeddings. If you get chunking wrong, your RAG pipeline will fail conceptually. - **Chunks too large**: You dilute the semantic meaning of the text, causing search to fail and eating up your LLM's context window. - **Chunks too small**: The isolated sentence "Yes, it is covered" means nothing if the preceding chunk holding the context ("Does the warranty cover water damage?") isn't retrieved alongside it. Here is a breakdown of the primary chunking strategies in 2026, when to use them, and how they impact vector retrieval. ## 1. Fixed-Size Chunking (The Default) This is a brute-force approach. You split the text every N characters or tokens. To prevent cutting sentences in half, you implement an **overlap**. - **How it works**: Chunk 1 is tokens 0-500. Chunk 2 is tokens 400-900. (Overlap of 100). - **Pros**: Computationally cheap. Easy to implement (`RecursiveCharacterTextSplitter` in LangChain). Guarantees you never exceed an LLM's context limit. - **Cons**: completely ignores the actual structure of the document. A chunk might start in the middle of a paragraph and end halfway through a bulleted list. - **When to use**: Plain text files, transcripts without clear logical boundaries, or when you are building a quick prototype. ## 2. Structural Chunking (Markdown / HTML) Instead of counting characters, this strategy respects the physical layout of the document. It splits based on Headers (`#`, `##`) or specific HTML tags. - **How it works**: Everything under '## Pricing' becomes one chunk. Everything under '## Requirements' becomes another. - **Pros**: Hugely improves retrieval accuracy. The chunk retrieved is a complete, logically sealed thought. - **Cons**: Requires your source documents to be cleanly formatted. Messy PDFs parsed via OCR will fail spectacularly here. - **When to use**: Parsing Notion wikis, Github Readmes, and strongly formatted documentation sites. LangChain provides the `MarkdownHeaderTextSplitter` specifically for this. ## 3. Semantic Chunking (The State of the Art) Semantic chunking doesn't look at character counts or markdown syntax. It uses a lightweight embedding model to measure the mathematical shift in meaning between sentences. - **How it works**: 1. The system reads Sentence 1 and Sentence 2. 2. It checks their cosine similarity. 3. If they are similar (e.g., both talking about "account setup"), they are grouped into the same chunk. 4. When it reads Sentence 5, and the topic suddenly shifts from "account setup" to "billing", the cosine distance spikes, and the system creates a hard boundary, starting a new chunk. - **Pros**: Perfectly encapsulates single ideas regardless of length. Produces the highest fidelity retrieval results in modern RAG. - **Cons**: Slow. It requires running an embedding model *during* the indexing phase just to figure out where to split the text, before you even do the final embedding for the vector database. - **When to use**: When retrieval accuracy is critical and you have the compute time to process the initial ingest. (See `SemanticChunker` in LlamaIndex or LangChain). ## 4. Parent-Child Chunking (The Hybrid Approach) Also known as "Small-to-Big Retrieval." - **How it works**: You chunk the document into very small, highly specific pieces (e.g., 100 tokens). These are the "Child" chunks that you embed in the vector DB. However, each Child holds a metadata link back to its much larger "Parent" chunk (e.g., 1000 tokens). - When a user searches for a specific keyword, the Vector DB finds the tiny Child chunk perfectly (high precision). But instead of sending that tiny chunk to the LLM, the system intercepts it, looks up the Parent chunk, and sends the massive Parent context to the LLM instead. - **Pros**: You get the extreme precision of tiny vectors, combined with the massive context of large chunks. - **When to use**: Complex enterprise RAG systems, legal documents, or highly technical manuals where both extreme precision and broad context are required simultaneously. ## What should you start with? If you are building your first application, don't overcomplicate it. Use **Fixed-Size Chunking** with a chunk size of `1000` characters and an overlap of `200` characters. When you notice your RAG failing to return the right context, upgrade to **Parent-Child Chunking** before migrating to full Semantic Chunking. --- Source: https://aimastermindlab.com/knowledge-base/document-chunking-strategies-rag — AI Mastermind Lab (https://aimastermindlab.com)