Hybrid Search: BM25 + Vector Similarity (With Code)

TL;DR: Pure vector search is amazing at understanding concepts, but terrible at finding exact matches (like serial numbers, specific names, or error codes). Hybrid search solves this by querying both a vector database and a traditional keyword index (BM25), then intelligently merging the results.

If you've built a basic RAG application, you've likely encountered this problem:

User query: "How do I fix Error Code XJ-881?"

A pure vector database will embed that query and look for things conceptually similar to fixing errors. It might return completely irrelevant documents about "Troubleshooting network issues" because "XJ-881" didn't carry enough semantic weight in the embedding model.

To build production-grade RAG, you must implement Hybrid Search.

The Two Pillars of Hybrid Search

1. Semantic Search (Vector Similarity): "Understands" meaning. A search for "canines" will find documents talking about "dogs" and "wolves."
   • Tool: ChromaDB, Pinecone, FAISS.
2. Lexical Search (BM25): "Understands" keywords. A search for "XJ-881" looks precisely for the frequency of that exact string in your documents and penalizes overly common words (the TF-IDF approach).
   • Tool: Elasticsearch, BM25 algorithm.

Hybrid means doing both simultaneously, then fusing the results using an algorithm like Reciprocal Rank Fusion (RRF) or a weighted sum.

Implementation with LangChain

We can build this easily in Python using LangChain's EnsembleRetriever.

Prerequisites

pip install langchain langchain-openai langchain-community rank_bm25 chromadb

(Note: rank_bm25 is necessary for the local lexical search implementation.)

Step 1: Set up the Documents

from langchain_core.documents import Document

# Sample dataset showing the weakness of pure vector search
docs = [
    Document(page_content="To fix the network router, reboot the system."),
    Document(page_content="Error Code XJ-881: The thermal paste has degraded."),
    Document(page_content="General overheating issues can cause system failure."),
    Document(page_content="For support regarding part XJ-900, call 555-0199.")
]

Step 2: Create the Vector Retriever (Semantic)

First, we build the standard dense embedding retriever using OpenAI (or Ollama locally).

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(docs, embeddings)

# Retriever 1: Vector Search
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

Step 3: Create the BM25 Retriever (Lexical)

Next, we use LangChain's built-in BM25 interface to index the exact same documents.

from langchain_community.retrievers import BM25Retriever

# Retriever 2: Keyword Search
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 2 # Return top 2 results

Step 4: The Ensemble Retriever (Hybrid)

Now we fuse them. The EnsembleRetriever runs both retrievers in parallel and merges their results.

We provide weights to determine which is more important. [0.5, 0.5] means equal weighting. If your application relies heavily on specific product names, you might lean towards [0.3, 0.7] (30% Semantic, 70% Keyword).

from langchain.retrievers import EnsembleRetriever

# Combine them!
hybrid_retriever = EnsembleRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    weights=[0.5, 0.5]
)

Step 5: Test the Difference

Let's search for an exact error code and see why Hybrid is necessary.

query = "I have Error Code XJ-881"

# 1. PURE VECTOR SEARCH
print("=== VECTOR ONLY ===")
vector_results = vector_retriever.invoke(query)
for doc in vector_results:
    print(f"- {doc.page_content}")

# Depending on the embedding model, this often returns the "overheating" 
# or "router" generic IT text because "XJ-881" is ignored mathematically.

# 2. HYBRID SEARCH
print("\n=== HYBRID SEARCH ===")
hybrid_results = hybrid_retriever.invoke(query)
for doc in hybrid_results:
    print(f"- {doc.page_content}")

# The BM25 algorithm immediately catches the "XJ-881" string, pulling the correct
# document to the top of the ensemble rankings.

Re-Ranking: The Next Step

While this EnsembleRetriever uses simple weight mathematics, production RAG systems (like those used by Perplexity) take Hybrid search a step further.

They retrieve ~20 documents using Hybrid search, and then pass those 20 documents to a Cross-Encoder Model (like Cohere Rerank or bge-reranker). The cross-encoder is an AI model that explicitly scores exactly how well a specific document answers a specific query, resorting the Top 20 into a highly precise Top 3 to pass to the final LLM.

Implementing Hybrid Search + Re-ranking is the single most effective way to eliminate hallucinations in a RAG pipeline.