PHASE 02 · RAG
Goal: Answer questions over documents the model was never trained on.
RAG (Retrieval-Augmented Generation) is the standard way to give an LLM access to information beyond its training data — your company's docs, your personal notes, a PDF that was published yesterday.
The pipeline has three steps. Index time: split your docs into chunks, turn each chunk into a numerical embedding (a vector that captures meaning), and store them in a vector database like FAISS. Query time: embed the question, find the K nearest chunks by similarity, and stuff them into the prompt as context. The LLM then answers using that retrieved context.
The single biggest lever in RAG quality is chunking strategy — how you split the documents. Fixed-size, recursive (paragraph → sentence → token), and semantic (split where meaning shifts) all behave very differently. Most RAG failures trace back to bad chunks.
pdfplumber or pypdf for PDFs).text-embedding-3-small (OpenAI) or Voyage. Store vectors in a FAISS index (in-memory, no infra).rag.py module to plug into the agent in Phase 4.rag.py — Pass a query, get an answer grounded in your docs — with citations to the retrieved chunks.
Concepts unlocked: embeddings · vector search · chunking trade-offs · context window
Phase 1 ended on a warning: the model predicts what's likely, not what's true. Here's where that bites, before any jargon. Ask a model about something private — your company's internal rules, a document only your team has — and it can't help; it was never trained on your stuff. Worse, it'll often make something up that sounds right. That's a hallucination.
Let's prove it on a company the model can't possibly know — because we made it up:
The paragraph is everything: hand the model the right text and it answers perfectly; leave it out and it guesses. Now make the problem real. Zentara's shared drive holds about 100 documents — a 40-page HR handbook, an expense policy, a security guide, onboarding checklists, years of meeting notes. Somewhere in that pile sits the one paragraph about parental leave. So the real job is this: given a question, find the one right paragraph out of a huge pile of documents, automatically.
Why you can't just paste everything. You can't shove all your documents into every question — they won't fit, and almost all of it is irrelevant. The whole phase is about finding the few paragraphs that matter and pasting only those. The next section is the trick that makes it possible.
For now, picture the whole thing as one magic box — call it a retriever. It somehow reaches into all 100 documents, pulls out the single right paragraph, and hands it to the model along with your question:
That retriever is the only mystery here. Everything else — question in, the right paragraph out, the model reads it and answers — you already understand. The rest of this phase is opening that box and building what's inside. Keep this picture in mind; we'll fill it in piece by piece.
Before we find paragraphs, here's the idea that makes it possible — and it's simpler than it sounds. Picture a sheet of graph paper. Your job: place every word on it, giving each one an (x, y) coordinate, so that words with similar meaning land close together.
So cat, kitten, and dog cluster in one corner; invoice, payment, and tax in another; ocean, wave, and beach in a third — like this:
You'd lose your mind doing this by hand for every word. The good news: there's a special kind of model — an embedding model — that does exactly this, automatically. Give it a word and it hands back coordinates. Those coordinates are called an embedding, or a vector. Think of it as a street address for meaning: an address tells you where a building sits in a city, an embedding tells you where a piece of text sits on a map of meaning. (Real ones use hundreds of dimensions, not two — but the picture is identical.)
Now the payoff. Give it a new word, drop it onto the same map, and measure the distance to every existing point. The nearest points are the nearest in meaning:
One name to file away now, because you'll see it in real code: in practice that "how close" measurement is usually a number called cosine similarity — a single score for how close in meaning two points are, where 1 means pointing in exactly the same direction (same meaning) and around 0 means unrelated. The tools compute it for you; the ruler-distance-between-dots picture you already have is the same idea.
So far we've placed single words. Here's the leap that makes search possible: the embedding model doesn't care whether you hand it one word, a whole sentence, or a full paragraph — it's all just text. It reads the meaning of the whole thing and gives it one point on the map.
So the sentence "employees get 18 weeks of paid parental leave" becomes a single point. And it lands near other text about the same thing — close to "our maternity and paternity policy", and far away from "how to reset your wifi password." What decides where it goes is the meaning, not the exact words.
Now the piece that trips people up. A question is also just text — so it gets a point too, on the very same map. And a question about parental leave is about the same thing as the paragraph that answers it. Same topic, same neighborhood.
So the question "how much parental leave do I get?" lands right next to the paragraph "employees get 18 weeks of paid parental leave" — not because the words match (they barely do), but because the two are about the same thing. That's the whole reason this works.
Picture Zentara's HR documents as points — each dot is one paragraph — and drop the question onto the same map. It falls next to the paragraph that answers it:
And here's what those cosine-similarity scores look like with real values. Score Zentara's question "how much parental leave do I get?" against three pieces of its documents:
That's the entire move. To find the paragraph that answers a question, you don't read anything — you turn the question into a point, score every paragraph-point by closeness, and grab the nearest ones. This is retrieval: searching by meaning instead of by matching keywords.
One catch: with 100,000 paragraphs, measuring the distance to all of them for every question would be slow. A vector database (FAISS, Qdrant, Pinecone, pgvector) is built for exactly this — it stores millions of these vectors and returns the nearest ones in milliseconds. You hand it a question's vector; it hands back the closest paragraphs.
The one idea under all of RAG. An embedding turns any text into a point in space, placed by meaning, so similar meanings sit close. A vector database stores those points and finds the nearest ones fast. Search-by-meaning is the engine under everything that follows.
Remember the mystery retriever box from a moment ago? You can already open it. Inside, it does exactly what you just learned — embed the question into a point, then grab the nearest paragraph-points:
So the retriever isn't magic — it's just embed + find nearest. But we've been glossing over one detail, and it's big enough to deserve its own section: which piece of text gets its own point?
We just said every paragraph becomes a point on the map. But who decided it's paragraphs? Zentara's HR manual is 40 pages. Should the whole manual get one point? Each paragraph? Each sentence? Somebody has to choose — and the choice matters more than it looks.
Embed a whole document and its point blurs together dozens of unrelated topics — a question about parental leave might match a document that's mostly about expense reports, just because leave gets one paragraph in it. Embed single sentences and you lose the surrounding context that made an answer make sense. There's a middle ground, and it has a name: a chunk — a small piece of a document, usually a paragraph or two.
Here's why this little word is the most important one in the phase. The chunk is the unit everything else works on — every stage of the system handles one chunk at a time:
The unit of everything. One chunk = one point on the map = one row in the vector database = one search result = one block of text pasted into the prompt. Get the chunk right and the rest of RAG mostly takes care of itself; get it wrong and nothing downstream can save you. That's why a later section spends real time on how to cut them.
Now you have the whole plan, in one line: split your documents into chunks, embed each one into a point, and store them — done once, ahead of time. Then, every time a question comes in, embed it too, and grab the nearest chunks. Two separate paths. Let's build them.
Time to build the first path for real. Zentara has 100 HR documents, and an employee will eventually ask "what's the parental leave policy?" — but this path runs before that ever happens.
Take the first path. Before anyone asks anything, you prepare the documents in three moves — split, embed, store. Split each document into small chunks (a few paragraphs each). Run every chunk through the embedding model to turn it into a vector — its point in space. Drop every vector into the vector database. Watch a document split into chunks and land in storage:
Put real numbers on it for Zentara: 100 documents go in; the splitter turns them into roughly 2,000 chunks (the 40-page HR manual alone yields about 180); the embedding model turns each chunk into a vector; and the vector DB ends up holding 2,000 points — a searchable map of everything Zentara knows, each paragraph placed by meaning.
Ingestion is the slow, one-time part. You only re-run it when the documents change — add a new policy doc, embed just that one. Once it's done, the hard work is over and answering questions is fast.
The documents are prepped, so the second path is quick. When "what's the parental leave policy?" comes in, you do four moves — embed, search, retrieve, answer.
Embed the question into a vector, using the exact same embedding model so it lands in the same space as the chunks. Search the vector DB for the nearest chunk vectors. Retrieve those few chunks, paste them in front of the question, and let the model answer using only that text. Watch the question search, match, and get answered:
On the example: the question embeds to a point; the nearest chunk is the HR-handbook paragraph that says "18 weeks paid"; that chunk gets pasted in with the question; the model reads it and answers correctly, citing where it came from. (In practice you pull the nearest 3–5 chunks, not just one, to be safe.)
The word paste is meant literally, by the way. You never have to write any of this yourself — your coding assistant will — but here is the actual text the model ends up receiving for Zentara's question. It's just Phase 1's "send the model some text", with the retrieved chunks stacked on top:
Context:
[1] Parental leave: employees receive 18 weeks of fully paid
leave, plus a phased return to work.
[2] Vacation days: full-time employees accrue 20 days per year.
[3] Public holidays: Zentara observes 11 public holidays.
Question: how much parental leave do I get?
Answer using ONLY the context above, citing chunks by number.
If the answer is not in the context, say you don't know.No magic left: the model reads chunk [1] and answers "18 weeks of fully paid leave [1]" — the same trick as the pasted paragraph in the very first section, just automated.
Why it must be the SAME embedding model. Each embedding model draws its own map of meaning — its own coordinate system, laid out differently. Two models are like two different maps of the same city: both are fine maps, but a coordinate read off one points at nothing meaningful on the other. Embed your chunks with model A and your questions with model B, and the question lands somewhere random relative to the chunks — not slightly less accurate, broken outright. One map. Same model on both paths, always.
That's RAG. RAG — retrieval-augmented generation. Ingestion (once): split → embed → store. Query (every time): embed → search → retrieve → answer. Using the same embedding model on both sides is what makes the question and the right paragraph land near each other.
When is RAG the right tool?. Use RAG when the knowledge is large, changes often, or needs citations. If everything fits in the context window and you ask rarely, just paste it all — simpler. And fine-tuning is for changing the model's style, not for adding facts.
Back up to the very first step of ingestion — splitting each document into chunks. It sounds like boring plumbing, but it quietly decides whether RAG works at all. Here's why it matters so much.
Remember the unit-of-everything rule: each chunk becomes one point in the vector space, and retrieval hands the model whole chunks. So it's what gets embedded, what gets searched, and what gets pasted into the prompt. Get the chunk wrong and there's nothing the model can do to save it.
The classic failure is cutting a document in the wrong place, so the answer gets torn in half:
So the real skill is cutting at sensible boundaries — the end of a paragraph or a section — instead of blindly every N characters. There are three common ways to do it, from crudest to smartest:
Start with the simplest thing that works — fixed-size, but with a little overlap, where each chunk repeats the last sentence or two of the one before it. The overlap is a safety net: if the answer happens to land right on a boundary, the repeated text means it still shows up whole in one of the chunks. For scale: Zentara's 40-page manual is roughly 90,000 characters, so at 500 characters per chunk that's about 180 chunks.
Here's what the naive version looks like. Like every code block in this course, you're not expected to write or even parse it — your assistant does that — it's just here so you can see how little code the crude version is:
def chunk(text, size=500, overlap=50):
chunks, start = [], 0
while start < len(text):
end = start + size
chunks.append(text[start:end])
start = end - overlap # step back a little so chunks overlap
return chunksThe trade-off you'll actually feel. Chunks too big → each point covers many topics, so retrieval returns a wall of mostly-irrelevant text and the real answer gets diluted. Too small → you lose the surrounding context that made a sentence meaningful. Good starting point: ~300–500 characters for dense docs, larger for flowing prose — then let your Phase 3 evals tell you what actually works for your content.
You've now seen every piece — chunks, embeddings, the vector database, search, and the prompt. Time to put the whole thing in one picture. Here's the complete RAG pipeline:
Read it as the same two paths from before, and walk Zentara's running example through it one last time:
The mystery box, solved. Remember the retriever from the very start of this phase? It was the embed → search → vector DB part all along. You've now built every box in this diagram — and that's the whole of RAG.
Time to build the deliverable: a single module you pass a query and get back a grounded answer plus the chunks it used. As always, you describe it to your assistant and run the result — this is the thing Phase 4's agent will call as a tool.
Then read what it built — here's the shape of it and the moving parts:
One thing in the sample below surprises people: it talks to two different companies. An OpenAI embedding model turns text into points, and Claude — a chat model — writes the final answer. That's not a mistake: embedding models and chat models are two different kinds of product, each does exactly one job, and mixing vendors like this is completely normal. The only hard rule is the one you already know — the same embedding model on both paths.
# rag.py — pass a query, get a grounded answer with citations
import faiss, numpy as np
from openai import OpenAI
from anthropic import Anthropic
oai, claude = OpenAI(), Anthropic()
def embed(text):
r = oai.embeddings.create(
model="text-embedding-3-small", input=text)
return r.data[0].embedding
def build_index(chunks):
vecs = np.array([embed(c) for c in chunks]).astype("float32")
faiss.normalize_L2(vecs)
idx = faiss.IndexFlatIP(vecs.shape[1])
idx.add(vecs)
return idx
def answer(question, chunks, index, k=4):
q = np.array([embed(question)]).astype("float32")
faiss.normalize_L2(q)
scores, ids = index.search(q, k)
hits = [chunks[i] for i in ids[0]]
context = "\n\n".join(f"[{n+1}] {h}" for n, h in enumerate(hits))
msg = claude.messages.create(
model="claude-sonnet-4-6",
max_tokens=600,
messages=[{"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}"}])
return msg.content[0].text, hitsDeliverable — rag.py. Pass a query, get an answer grounded in your docs, with citations to the retrieved chunks. Keep it importable: Phase 4 wires answer() in as a tool the agent can call.When RAG gives a bad answer, the instinct is to blame the model. It's almost always retrieval. Separate the two so you know which to fix.
Evaluate retrieval and answers separately. First check retrieval: for each test question, is the gold chunk — the chunk that actually contains the correct answer, which you identified by hand ahead of time — in the top-K results? (Pure counting, no LLM needed.) Only once retrieval is good, judge answer quality. Phase 3 turns this into a real eval harness.
What you'll have when you're done: you can answer questions over your own documents; you've felt how chunk size moves quality more than the model does; you can explain embeddings as "vibes-as-vectors" to a friend; and you have a reusable rag.py ready for Phase 4.