Vector search feels like the default answer for retrieval in 2026. Embed your chunks, run cosine similarity, return the nearest neighbors, done. It works beautifully — until it doesn't. Ask your RAG system about "error code 0x80070005" or "ISO 27001 clause 6.1.2" and pure vector search may confidently return semantically adjacent paragraphs about Windows errors or risk management while missing the exact document that contains the string you typed.

The failure is structural. Embeddings compress meaning into dense vectors, and exact tokens — identifiers, serial numbers, proper nouns, version strings — often have weak signals in that compressed space. Keyword search, specifically BM25, has the opposite profile: it's superb at matching exact rare terms and terrible at understanding that "outage" and "downtime" mean the same thing.

Hybrid search is the boringly effective answer: run both, fuse the results, return one ranked list. This tutorial shows how it works, why the naive way of combining scores breaks, and how to build a working pipeline.

The two halves of retrieval#

Before fusing anything, it helps to be precise about what each half does.

BM25 (sparse retrieval). BM25 scores documents by term frequency (how often your query terms appear in the document) adjusted by inverse document frequency (how rare those terms are across the corpus), with saturation and document-length normalization. It is the default text-scoring algorithm in Elasticsearch and most keyword engines. Its superpowers: exact-token precision, rare-term sensitivity, no model required, and speed. Its blind spots: vocabulary mismatch (the user says "crash," the doc says "segfault"), stemming artifacts, and zero understanding of meaning.

Dense vector search. Each chunk is embedded into a few hundred to a few thousand dimensions by a model like bge-m3 or a text-embedding API, and retrieval is a nearest-neighbor lookup — usually approximate (HNSW) for speed. Its superpowers: paraphrase and synonym matching, cross-lingual overlap, and tolerance for sloppy queries. Its blind spots: exact strings and identifiers, short queries where semantic drift dominates, and a hard dependency on embedding-model quality, cost, and version compatibility.

In practice, query traffic splits across these profiles. Support queries mix conceptual descriptions with ticket numbers. Enterprise docs mix prose with clause references. E-commerce queries mix intent ("weatherproof commuter backpack") with hard constraints (brand, model number). Neither half alone covers that mix — which is why hybrid is the sensible default for production RAG, not an advanced upgrade.

Why you can't just add the scores#

The tempting first implementation looks like this:

final_score = alpha * bm25_score + (1 - alpha) * vector_score

This is a trap. BM25 scores and cosine similarities live on completely different scales and distributions. A BM25 score of 3.2 and a cosine similarity of 0.87 cannot be combined arithmetically without careful, dataset-specific normalization — and that normalization shifts every time your corpus changes. Any alpha you pick is a guess, and it decays silently as documents accumulate.

That leaves two families of fusion that actually hold up:

  1. Rank-based fusion (Reciprocal Rank Fusion, RRF) — ignores raw scores entirely and combines only rank positions.
  2. Normalized score fusion — rescales each side's scores onto a common range (typically 0–1) before combining them.

Weaviate ships both: rankedFusion (RRF-style) and relativeScoreFusion (normalize-then-sum), with relative score fusion as the newer default. Elasticsearch offers RRF natively for combining its BM25 retriever with kNN vector search. LangChain's EnsembleRetriever lets you attach weights to a BM25 retriever and a vector-store retriever. We'll build the fusion ourselves first, so you know exactly what these wrappers are doing.

Reciprocal Rank Fusion, in detail#

RRF is the most parameter-free fusion method worth knowing. The formula:

score(d) = Σ over lists of  1 / (k + rank(d))

where rank(d) is the 1-indexed position of document d in each result list and k is a smoothing constant, conventionally 60. That's the whole algorithm. A document at rank 1 in the BM25 list and rank 3 in the dense list gets 1/61 + 1/63 ≈ 0.0323.

Why k = 60? It dampens the influence of top positions: a document at rank 1 contributes 1/61 ≈ 0.0164 per list while one at rank 100 contributes 1/160 = 0.00625 — only a 2.6× gap instead of a 100× gap. This gives consistent cross-list appearances a chance against a single lucky top hit. As the Elasticsearch docs note, citing the original Cormack et al. paper: "RRF requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results."

The important design property: RRF needs no score normalization and no training data. It sidesteps the scale-mismatch problem by construction, and documents that appear in both lists get naturally boosted — a built-in reward for cross-signal agreement.

Building the pipeline: a minimal working example#

Here's a complete, dependency-light hybrid search you can run locally. It uses rank_bm25 for the keyword side and a sentence-transformers model for the dense side — the same pairing that powers many production prototypes:

from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np

docs = [
    "Error 0x80070005: access denied when the service account reads the config file.",
    "To grant permissions, add the service account to the local Administrators group.",
    "Authentication failures usually indicate expired credentials, not network problems.",
    "The timeout setting controls how long the client waits before retrying a request.",
]

# Keyword side
tokenized = [d.lower().split() for d in docs]
bm25 = BM25Okapi(tokenized)

# Dense side
model = SentenceTransformer("all-MiniLM-L6-v2")
doc_vectors = model.encode(docs, normalize_embeddings=True)

def reciprocal_rank_fusion(ranked_lists, k=60):
    scores = {}
    for ranked_list in ranked_lists:          # ranked_list: [(doc_id, score), ...] best-first
        for rank, (doc_id, _) in enumerate(ranked_list, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

def hybrid_search(query, top_k=3, overfetch=3):
    # 1. BM25 ranking
    bm25_scores = bm25.get_scores(query.lower().split())
    bm25_ranked = sorted(enumerate(bm25_scores), key=lambda x: x[1], reverse=True)

    # 2. Dense ranking
    q_vec = model.encode([query], normalize_embeddings=True)[0]
    dense_scores = doc_vectors @ q_vec          # cosine, since normalized
    dense_ranked = sorted(enumerate(dense_scores), key=lambda x: x[1], reverse=True)

    # 3. Fuse and trim
    fused = reciprocal_rank_fusion(
        [bm25_ranked[: top_k * overfetch], dense_ranked[: top_k * overfetch]]
    )
    return [(docs[i], round(s, 4)) for i, s in fused[:top_k]]

for doc, score in hybrid_search("0x80070005 access denied"):
    print(f"{score:.4f}  {doc}")

Three things to notice:

  • Overfetch before fusing. Each side retrieves top_k × overfetch candidates (here 9), because the fused top-3 can legitimately include a document that was rank 6 on one side. Fusing only the top-3 from each side starves the merge.
  • Normalization before cosine. normalize_embeddings=True turns the dot product into true cosine similarity. Small detail, easy to forget.
  • No weights anywhere. RRF needs none. That is the point.

Try the query "expired credentials login problem" against this corpus and you'll see the dense side carry the result even though no document contains the word "login" — exactly the paraphrase case BM25 fumbles.

The production layer: tuning knobs that matter#

The toy pipeline teaches the mechanics. Production adds four controls.

1. The alpha knob (where your engine has one)#

Engines that do normalized score fusion, like Weaviate's relativeScoreFusion, expose an alpha that balances keyword vs. vector weight: 0 is pure BM25, 1 is pure vector, with Weaviate defaulting to 0.75 (leaning semantic). A common practical split is around 0.6–0.7 vector for conceptual query traffic, shifting toward BM25 when your users search identifiers, SKUs, or error codes. Treat alpha as data, not dogma — set it with an evaluation set, not a guess.

2. Filters belong before fusion#

Tenancy, permissions, stock status, dates — these are hard constraints, not ranking signals. Apply them to both candidate lists before fusing. Fusing first and filtering after can produce an empty-looking result page even when matching documents exist, because the candidates you fused were never eligible.

3. Rerank the fused shortlist#

RRF gives you better recall — more of the right documents in the top 50. For the final ordering of the top 5–10 that actually reach your LLM's context window, a cross-encoder reranker (e.g., bge-reranker variants) scores each (query, chunk) pair jointly and is noticeably more accurate than any fusion of independent retrievers. The catch is cost: a cross-encoder is one model call per pair, so run it only over the fused shortlist, never the corpus. The standard production shape is: hybrid retrieval → top 50 → cross-encoder rerank → top 5–8 for generation.

4. Watch latency and the embedding dependency#

Hybrid retrieval is two retrieval calls plus an embedding call per query. That's the real cost of the dense side, and it's why some architectures keep BM25 as the resilient default: if the embedding provider is down, hybrid mode can degrade to BM25-only rather than failing outright. Make that fallback explicit, not accidental.

When hybrid isn't the answer#

Hybrid search fixes a specific failure — mixed query traffic with both exact terms and conceptual language. It does not fix:

  • Bad chunking. If the answer is split across two chunks or buried in a 2,000-token slab, better retrieval ranking can't save you.
  • Missing content. No fusion algorithm retrieves a document that doesn't exist. The fallback prompt — "answer only from the provided context, say you don't know otherwise" — still belongs in every RAG system.
  • Vocabulary mismatch within the keyword side. If users consistently use terms your corpus never contains, add query rewriting or expansion before reaching for a different fusion method.

Takeaway#

The mental model to keep: BM25 is for the words, vectors are for the meaning, and RRF is the glue that needs no calibration. Start with reciprocal rank fusion at k = 60, overfetch 2–3× per side, and apply hard filters before the merge. Add normalized-score weighting only when an eval set says a tuned alpha beats rank fusion on your data, and put a cross-encoder reranker on the fused shortlist when precision at the top matters more than raw latency.

Pure vector search was a useful simplification while the tooling matured. But search traffic is hybrid by nature — part identifiers, part questions — and your retrieval pipeline should be too.