Late-interaction retrieval with ColBERT: beat BM25 and retire your reranker — measured end-to-end

Your RAG pipeline has a retrieval problem, and it is not your chunking strategy. It is the thirty-year-old keyword matcher sitting at the front of it: BM25 cannot match a query to a document that says the same thing in different words. Dense bi-encoders fix the vocabulary mismatch but throw away token-level detail — one vector per passage, take it or leave it. ColBERT's late interaction is the third option: keep every token's embedding, and let each query token find its best match at search time. I built a 48-passage test corpus with deliberately paraphrased queries, ran BM25 against ColBERT end to end on a CPU, and measured exactly where each one wins. Here is the full loop, reproducible.

Why this matters now

Most production RAG stacks in 2026 look the same: BM25 for cheap first-pass retrieval, then a cross-encoder reranker (monoT5, a Cohere-style rerank API, or a fine-tuned MiniLM) rescoring the top 50, then the LLM. That reranker stage exists for one reason — BM25's vocabulary mismatch. A user asks "how do I shrink a model to four bits without wrecking it" and the passage that answers it talks about "GPTQ weight quantization with calibration data." Zero shared keywords, zero BM25 score, and the reranker never even sees the passage because it was cut at the first stage.

Cross-encoders are accurate because they run full self-attention over the query and passage together — but that is also why they are expensive: you cannot run one over a million passages. So you use them as rerankers on a shortlist produced by something dumber. ColBERT (Khattab & Zaharia, 2020; v2 by Santhanam et al., 2021) breaks the dilemma with late interaction: encode the query and the document independently into per-token embeddings, store the document token embeddings in the index, and only at search time compute fine-grained token-to-token matches. You get cross-encoder-grade matching quality with retriever-grade economics — the documents are encoded once, offline, and the expensive interaction happens over a handful of vectors per candidate, not over raw text.

The practical consequence: a ColBERT index can replace the BM25-plus-reranker two-stage pipeline with a single stage. That is the claim I am testing below — measured, not asserted.

Late interaction in sixty seconds

Three retrieval architectures, one query, one passage:

• BM25 (sparse): bag of words on both sides. Score = weighted keyword overlap. Fast, exact, and blind to paraphrase.
• Bi-encoder (dense): one embedding vector per query, one per passage, cosine similarity. Handles paraphrase, but compresses a 200-token passage into a single vector — token-level evidence is lost.
• ColBERT (late interaction): one embedding vector per token on both sides. Score = for each query token, take its maximum similarity over all document tokens, then sum. This is MaxSim:

score(query, doc) = sum(
    max(cosine_sim(q_i, d_j) for d_j in doc_tokens)
    for q_i in query_tokens
)

Each query token independently finds its best-matching document token — "shrink" can match "quantization," "four bits" can match "4-bit" — and the sum rewards passages that cover all of the query's aspects. Because documents are encoded offline and only the lightweight MaxSim runs at query time, you keep the per-token fidelity of a cross-encoder without paying cross-encoder prices at retrieval.

Two details that make it work in practice. First, query augmentation: ColBERT pads short queries with [MASK] tokens up to a fixed length, which gives the model spare token slots to "expand" the query semantically during encoding. Second, ColBERTv2 added residual compression of the token embeddings (roughly 6–10× smaller index) and denoised training supervision — the checkpoint you will use below, colbert-ir/colbertv2.0, is the v2 model. For large-scale search, the PLAID engine (2022) clusters the token embeddings so MaxSim only runs against candidate clusters instead of the full index.

What you will need

Python 3.10+, about 2 GB of disk for the model and dependencies, and no GPU — everything below ran on a plain CPU. I used RAGatouille (Answer.AI's high-level ColBERT wrapper) because it reduces indexing and search to two method calls, with PLAID indexing under the hood.

Install CPU-only PyTorch first — the default wheels bundle CUDA and download ~2.5 GB you will never use:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install "transformers<5" ragatouille rank-bm25 psutil ninja

Four notes on that install line, all earned the hard way:

• transformers<5: RAGatouille 0.0.9 vendors ColBERT code written against the transformers 4.x model-loading API. With transformers 5.x, checkpoint loading crashes inside _finalize_model_loading (AttributeError on the renamed tied-weights attribute). Pinning 4.x — I used 4.56.2 — fixes it. RAGatouille has announced a 0.0.10 release migrating to a PyLate backend; if you are reading this later, try unpinned first.
• ninja: ColBERT JIT-compiles a small C++ MaxSim kernel with torch.utils.cpp_extension at import time, which requires Ninja and a C++ compiler. Without it you get RuntimeError: Ninja is required to load C++ extensions.
• psutil: a missing transitive dependency of fast_pytorch_kmeans (used by PLAID clustering). Without it, indexing dies on import.
• langchain: RAGatouille 0.0.9 imports langchain.retrievers.document_compressors.base at module top level, but langchain is only an optional extra — and langchain 1.x removed that module path entirely. The snippet below shims it from langchain_core (RAGatouille only uses it as a base class). Twelve lines, then you never think about it again.

Step 1 — Build a test corpus that punishes keyword search

To measure anything honestly, you need queries where BM25 should fail. I wrote 48 passages — six per topic across eight ML topics (attention, optimization, RAG, quantization, diffusion, RLHF, vector databases, prompting) — and 20 queries with known gold passages. Eight queries share vocabulary with their gold passage; twelve are deliberately paraphrased to avoid the passage's key terms. That 12-query paraphrase set is the whole experiment: it simulates real users, who never word things the way your documents do.

A paraphrase example from the set — query first, then the gold passage it must find:

Query: "What optimizer adapts a separate step size for every weight
        and fixes its own startup bias?"

Gold passage (opt-2): "Adam keeps running averages of the gradient (first
moment) and the squared gradient (second moment), then corrects both for
initialization bias. It adapts the learning rate per parameter..."

Shared content words between query and passage: "adapts"/"adapts", "weight"/"parameter" if you are generous — and the passage never says the word "optimizer" at all. A keyword matcher has almost nothing to grip. Keep this example in mind; we will come back to it.

Step 2 — The BM25 baseline

Baseline first, always. Tokenize, score, rank — rank-bm25 in a few lines:

from rank_bm25 import BM25Okapi

tokenized = [text.lower().split() for text in passages]
bm25 = BM25Okapi(tokenized)

def bm25_search(query, k=5):
    scores = bm25.get_scores(query.lower().split())
    ranked = sorted(range(len(scores)), key=lambda i: -scores[i])
    return [doc_ids[i] for i in ranked[:k]]

Nothing to tune, runs in microseconds. This is the bar ColBERT has to clear — and on the eight vocabulary-overlap queries, it is a high bar.

Step 3 — Index with ColBERT

Here is the complete indexing script. Note the langchain shim at the top (see the install notes), and that everything runs under if __name__ == "__main__" — RAGatouille's multiprocessing indexing requires it:

import sys, types

# Compatibility shim: RAGatouille 0.0.9 imports
# langchain.retrievers.document_compressors.base at module level,
# which no longer exists in langchain>=1. Re-point it at langchain_core.
from langchain_core.documents.compressor import BaseDocumentCompressor
base_mod = types.ModuleType("langchain.retrievers.document_compressors.base")
base_mod.BaseDocumentCompressor = BaseDocumentCompressor
dc_mod = types.ModuleType("langchain.retrievers.document_compressors")
dc_mod.base = base_mod
r_mod = types.ModuleType("langchain.retrievers")
r_mod.document_compressors = dc_mod
sys.modules["langchain.retrievers"] = r_mod
sys.modules["langchain.retrievers.document_compressors"] = dc_mod
sys.modules["langchain.retrievers.document_compressors.base"] = base_mod

from ragatouille import RAGPretrainedModel


def main():
    passages = [...]       # your texts
    doc_ids = [...]        # stable IDs, one per passage

    RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
    index_path = RAG.index(
        collection=passages,
        document_ids=doc_ids,
        index_name="my_corpus",
        max_document_length=180,
        split_documents=False,
    )
    print("index at:", index_path)


if __name__ == "__main__":
    main()

What those arguments do: max_document_length=180 caps per-passage tokens (ColBERTv2's native limit is higher, but 180 covers these passages and keeps the index small); split_documents=False keeps each passage as one retrievable unit instead of auto-chunking. from_pretrained downloads the ~420 MB checkpoint on first run. Indexing my 48 passages took 136.5 seconds on CPU, and the index on disk is 376 KB.

Step 4 — Search and measure

Search is one call. It returns ranked hits with scores, and document_id maps back to your IDs:

hits = RAG.search(
    query="What optimizer adapts a separate step size for every weight "
          "and fixes its own startup bias?",
    k=5,
)
for h in hits:
    print(h["rank"], h["document_id"], round(h["score"], 2))
# 1 opt-2 13.82
# 2 opt-5 13.54
# 3 opt-6 13.07
# ...

That is real output from the experiment index: the Adam passage (opt-2) at rank 1, ahead of the other optimization passages. Each hit also carries content and passage_id if you need them.

To reload an existing index without re-encoding (the second run onward), use RAGPretrainedModel.from_index(index_path) instead of from_pretrained + index.

I scored both systems with recall@1/3/5 and MRR@5 across all 20 queries, then split the results by query type. Mean per-query latency was timed with time.perf_counter around each search call.

The measured results

Overall, across all 20 queries:

* ColBERT latency: ~1.1 s per query on CPU after warmup (measured probe of 6 searches); the 20-query experiment averaged 10 s per query with each search running cold. BM25 needs no warmup.

The aggregate hides the real story. Split by query type:

Read it as three findings:

1. On vocabulary-overlap queries, BM25 is genuinely competitive. BM25 reached 0.75 recall@1 with 0.875 MRR on the eight overlap queries; ColBERT was perfect (1.00 across the board), but the gap is small. If your users search with your documents' vocabulary — documentation search, log search — BM25 is not the problem and ColBERT is not the fix.
2. On paraphrased queries, BM25 collapses and ColBERT holds. On the twelve paraphrase queries, ColBERT scored 0.75 recall@1 and 0.826 MRR against BM25's 0.583 and 0.736 — and ColBERT's recall@5 was a perfect 1.00, meaning the gold passage was always in the shortlist a reranker or LLM would see. This is the vocabulary-mismatch problem, and it is the normal case for RAG: users describe problems, documents describe solutions, and the words rarely line up.
3. The latency cost is real but one-sided. BM25 answers in about half a millisecond; ColBERT takes roughly a second per query on CPU once warmed up (my 20-query experiment averaged 10 s per query because each search ran cold, paging the index in — a warmed-up probe averaged 1.1 s). That is three orders of magnitude slower than BM25, still far below a cross-encoder reranker pass or an LLM call. You pay it once per query at search time; indexing is the offline cost (136.5 s for 48 passages here, scaling roughly linearly).

And the Adam example from Step 1? BM25 ranked the gold passage opt-2 at position 9 — outside any usable top-5 shortlist, so a reranker would never have seen it. ColBERT put it at rank 1. That single query is the entire argument in miniature: the user described the behavior ("adapts a separate step size for every weight and fixes its own startup bias"), the passage described the mechanism ("adapts the learning rate per parameter... corrects both for initialization bias"), and only the system that matches meaning per token connected them.

Which approach should you use?

Four retrieval architectures, honest placement:

• BM25 — Use it when queries share vocabulary with documents (docs search, keyword-ish site search, tiny corpora), or as a near-free prefilter. It is deterministic, explainable, needs no GPU, and indexes in milliseconds. It cannot do paraphrase. Do not ask it to.
• Bi-encoder dense retrieval — Use it when you need semantic matching over a large corpus with a small index (one vector per passage) and millisecond ANN search. The price is the information bottleneck: everything about a passage must survive compression into one vector. Fine for short passages; lossy for long ones.
• ColBERT / late interaction — Use it when retrieval quality is the bottleneck and your queries paraphrase rather than quote. It keeps per-token fidelity, needs no reranker behind it, and the index stays on CPU-friendly infrastructure. Prices: larger index than bi-encoders (one vector per token, mitigated by v2 residual compression), slower queries than BM25, and a heavier indexing step. This is the sweet spot for RAG over technical/prose corpora up to millions of passages with PLAID.
• Cross-encoder reranker — Use it when you need the last few points of accuracy on a shortlist and can afford full query-passage attention per candidate — typically as a second stage over 20–100 candidates, not as a retriever. If ColBERT is your first stage, you may find the reranker adds little: late interaction already captures most of what cross-attention buys, which is exactly why the single-stage ColBERT setup can retire the two-stage pipeline.

A practical migration path if you run BM25 + reranker today: swap the BM25 stage for ColBERT first and measure whether the reranker still moves your metrics. In many RAG setups it stops earning its latency — that is the "retire your reranker" moment. If it still helps, keep it; ColBERT as a first stage feeds it better candidates than BM25 did.

Takeaway

Late interaction is the rare idea that is both elegant and immediately usable: per-token embeddings, MaxSim scoring, documents encoded once and matched finely at query time. With RAGatouille it is genuinely a few lines of code — the hard parts of this tutorial were all environment friction (transformers 5, Ninja, the langchain import), not the retrieval itself. Measure on your own queries before you commit: build a small gold set with paraphrased queries the way I did here, because the overlap queries will tell you BM25 is fine and the paraphrase queries will tell you the truth. If your users describe problems and your documents describe solutions, ColBERT will earn its index.

The full experiment scripts — corpus builder, query set, BM25 baseline, ColBERT indexing, and the scoring harness — are structured so you can drop in your own passages and queries and re-run the same comparison in an afternoon. That is the real deliverable: not my numbers, but a harness that produces yours.