RAG — retrieval-augmented generation — is the highest-ROI pattern in applied AI. The idea is simple: instead of hoping the model memorized your data during training (it didn't), you retrieve the relevant documents at query time and stuff them into the prompt before generating the answer.

It's how you build a chatbot over your docs, a support assistant over your knowledge base, or a research helper over your PDFs — without fine-tuning anything. This tutorial builds a complete working pipeline in one afternoon with open tools.

What you'll build#

A question-answering system over a folder of documents:

docs/  →  [chunk] → [embed] → [vector index] → query → [retrieve top-k] → [LLM + context] → answer with sources

Prerequisites: Python 3.10+, basic comfort with pip, and an LLM API key (any provider works — the code below is provider-agnostic at the generation step).

Stack:

  • sentence-transformers for embeddings (the all-MiniLM-L6-v2 model: small, fast, good enough to start)
  • faiss for the vector index (or Chroma if you prefer a managed-feeling API)
  • Your LLM provider's API for generation

Step 1: Ingest and chunk your documents#

Models don't read documents — they read token windows. So split your docs into overlapping chunks. This is the step beginners under-invest in, and it's where most RAG quality is won or lost.

def chunk_text(text, chunk_size=500, overlap=50):
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        chunks.append(chunk)
    return chunks

Practical rules of thumb:

  • 300–800 tokens per chunk is the sweet spot for most Q&A. Smaller chunks retrieve more precisely but lose context; larger chunks keep context but dilute relevance.
  • Overlap 10–20% so ideas split across boundaries survive.
  • Respect structure. If your docs are Markdown, split on headings first, then by size. A chunk that starts mid-sentence is a chunk the model will misunderstand.
  • Attach metadata — filename, page, section — to every chunk. You'll want it for citations and debugging.

Step 2: Embed the chunks#

Embeddings turn text into vectors where similar meanings sit close together. This is the "retrieval" in RAG.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(chunks, show_progress_bar=True)

all-MiniLM-L6-v2 produces 384-dimensional vectors and runs fine on CPU. It's not the best embedding model available — but it's fast, free, and good enough that your bottleneck will be chunking quality long before it's embedding quality. Upgrade later; start now.

Step 3: Build the vector index#

import faiss
import numpy as np

dim = embeddings.shape[1]
index = faiss.IndexFlatL2(dim)
index.add(np.array(embeddings).astype("float32"))
# persist it — rebuilding embeddings every run wastes your afternoon
faiss.write_index(index, "docs.index")

IndexFlatL2 is exact brute-force search — perfect up to hundreds of thousands of chunks. When you outgrow it, FAISS has approximate indexes (IVF, HNSW) that trade a little recall for a lot of speed.

Step 4: Retrieve at query time#

def retrieve(question, k=5):
    q_vec = model.encode([question]).astype("float32")
    distances, ids = index.search(q_vec, k)
    return [chunks[i] for i in ids[0]]

Start with k=5. More context isn't always better — every irrelevant chunk is noise the model must ignore, and attention dilutes across long contexts (see our attention explainer). Tune k on real questions, not vibes.

Two upgrades worth knowing about:

  • Hybrid search: combine vector similarity with keyword search (BM25). Vectors catch paraphrases; keywords catch exact terms, product names, and error codes. Together they beat either alone.
  • Reranking: retrieve 20 candidates cheaply, then rerank to the top 5 with a heavier cross-encoder model. This is the single biggest quality jump for the least effort.

Step 5: Generate with context#

def answer(question):
    context = "\n\n---\n\n".join(retrieve(question))
    prompt = f"""Answer the question using ONLY the context below.
If the answer isn't in the context, say so plainly — do not guess.

Context:
{context}

Question: {question}"""
    return llm.complete(prompt)  # your provider's API call

That instruction — say so plainly, do not guess — is doing real work. RAG reduces hallucinations only if the model is allowed to admit retrieval came up empty. A RAG system that never says "I don't know" is just a hallucination machine with footnotes.

Step 6: Evaluate before you celebrate#

Don't ship on vibes. Build a test set of 20–30 real questions with known answers, and check:

  • Retrieval recall: is the right chunk in the top-k? If not, fix chunking or embeddings — no prompt trick compensates for missing context.
  • Faithfulness: does the answer stick to the retrieved context? Have a second model (or a human) check a sample.
  • Abstention: ask questions the docs can't answer. The system should say so. If it invents answers, tighten the prompt or add a confidence threshold.

Log every query, retrieved chunk, and answer from day one. Your future debugging self will thank you.

Common pitfalls (learned the hard way)#

  • Garbage in, retrieved out. RAG over messy, outdated docs gives you a chatbot that confidently quotes last year's prices. Curate the corpus like it matters — because it does.
  • Stale indexes. Docs change; embeddings don't update themselves. Rebuild on a schedule or wire ingestion to your doc pipeline.
  • Ignoring metadata filters. "What was Q3 revenue?" needs the 2024 report, not 2021's. Filter by date/source before vector search.
  • No citations. Show which chunks the answer came from. It builds user trust and makes failures debuggable.

Where to go next#

Once the basics work: add hybrid search and reranking, experiment with larger embedding models, try query rewriting (have the LLM reformulate the question before retrieval), and consider a small fine-tuned embedding model on your domain if retrieval recall plateaus.

The takeaway#

RAG is five steps — chunk, embed, index, retrieve, generate — and most of the quality lives in step one. Build the simple version this afternoon, evaluate it on real questions tomorrow, and iterate on chunking before you touch anything else. It's the rare AI pattern where the boring engineering matters more than the model — which is exactly why it's the highest-ROI thing a builder can learn right now.