Zucchetti LLM Course — Lecture 4 of 10

RAG Foundations
& Decision Matrix

From frozen knowledge to live data — and when to use what

Kamyar Zeinalipour · 90 minutes

Today’s Agenda

20 min The Retrieval Problem & The RAG Architecture

10 min ☕ Break

20 min From Keywords to Vectors — How Retrieval Evolved

15 min Document Processing & Chunking Theory

10 min Vector Databases — Similarity Search at Scale

15 min Decision Matrix: Prompt vs. RAG vs. Fine-Tune

5 min Key Takeaways & Q&A

Total: 90 minutes · Pure theory — hands-on lab in Lecture 5 (combined Lab 4 + Lab 5)

Section 1 of 5

The Retrieval Problem

Why LLMs need external knowledge

The Fundamental Problem

LLMs have two critical limitations:

🧊 Frozen Knowledge

Model knowledge is fixed at training cutoff.

GPT-4 does not know what happened yesterday.

It cannot access your internal documents.

It has never seen your company’s policies.

🎭 Hallucination Under Pressure

When the model lacks knowledge, it does not say “I don’t know.”

It generates a plausible-sounding answer.

The answer looks correct.

It is completely fabricated.

We need a way to inject real, verifiable knowledge into the model at query time.

Two Ways to Add Knowledge

Approach 1: Fine-Tuning

Bake knowledge into the weights.

Train on domain-specific data
Knowledge becomes “permanent”
Expensive, slow to update
Model may still hallucinate

Like memorizing a textbook — you might misremember.

Approach 2: RAG ⭐

Provide knowledge at query time.

Retrieve relevant documents
Inject into the prompt as context
Fast to update (re-index documents)
Answers are grounded in sources

Like an open-book exam — you can check your answers.

Fine-tuning changes behavior. RAG provides knowledge. They solve different problems.

Retrieval-Augmented Generation

The idea: instead of asking the model to remember, let it look things up.

📄 Your Documents → ✂️ Chunks → 🔢 Embed → 💾 Index

⬇ at query time ⬇

❓ Question → 🔍 Retrieve → 📝 Augment → 🤖 Generate → ✅ Grounded Answer

Two phases: Offline (index once) — Online (retrieve & generate per query)

P(y | x) = ∑_d∈D P(d | x) ⋅ P(y | x, d)

y = answer, x = query, d = retrieved doc — RAG marginalizes over retrieved documents (Lewis et al., 2020)

Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 2020

RAG in Action — Step by Step

A concrete example of the full pipeline.

👤 User asks: “What is the maternity leave policy?”

🔍 Step 1: Retrieve — system searches vector DB, finds 3 relevant chunks

Chunk 1 (hr_policy.pdf, p.12): “Employees are entitled to 5 months of mandatory maternity leave...”

Chunk 2 (hr_policy.pdf, p.13): “Optional parental leave can be extended up to 6 additional months...”

Chunk 3 (hr_policy.pdf, p.14): “During maternity leave, employees receive 80% of their salary...”

✅ Step 2: Augment & Generate — LLM reads chunks and produces a grounded answer

“Maternity leave is 5 months mandatory at 80% salary, with an option to extend up to 6 additional months. (Source: hr_policy.pdf, pages 12–14)”

The answer comes from your documents, not the model’s imagination. The user can verify by checking page 12.

Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 2020

Why RAG Works — The Grounding Principle

RAG transforms the generation task from recall to reading comprehension.

❌ Without RAG — Hallucination

“What is Zucchetti’s remote work policy?”

Model output: “Employees may work remotely up to 3 days per week, as per the 2022 Smart Working Agreement.”

⚠️ Completely invented. Confident. Wrong.

✅ With RAG — Grounded

“What is Zucchetti’s remote work policy?”

Model output: “According to hr_policy_2024.pdf p.8: employees may request up to 2 remote days per week after 6 months of employment.”

✅ From document. Verifiable. Cited.

LLMs excel at reading comprehension — RAG converts recall into comprehension.

RAG vs. Fine-Tuning — When to Use Each

	📚 RAG	🔧 Fine-Tuning
What it changes	Knowledge available	Model behavior / weights
Knowledge source	External documents	Baked into parameters
Update frequency	Fast (swap or re-index documents — minutes for typical corpora)	Retrain required (hours–days)
Cost	Low (embeddings + DB)	High (GPU compute)
Hallucination	Significantly reduced (answers grounded in text — but not eliminated: model can still ignore context)	Medium (may still confabulate)
Interpretability	High (can show sources)	Low (black box)
Best for	Facts, policies, Q&A	Style, format, behavior

RAG and fine-tuning are complementary, not competing. Many production systems use both.

☕

10-Minute Break

Stretch your legs — we are going deep next

Next up: How Search Evolved — from keywords to meaning — the math behind retrieval

Section 2 of 5

From Keywords
to Vectors

How information retrieval evolved over 50 years

The Evolution of Search

Era	Method	How It Works	Limitation
1970s	Boolean	Exact keyword match (AND, OR, NOT)	No ranking, no synonyms
1990s	TF-IDF	Rank by term frequency × rarity	No semantic understanding
1994+	BM25	Improved TF-IDF with saturation & length norm	Still keyword-based
2019+	Dense Retrieval	Embed query & docs as vectors, compare similarity	Needs good embeddings

RAG is retrieval-method-agnostic — it works with sparse (BM25), dense (embeddings), or hybrid search. Simple RAG systems still use BM25. Production systems typically use hybrid retrieval. Dense retrieval (embeddings & cosine similarity, from Lectures 1 & 2) is the modern standard — but not the only option.

Boolean Search — The First Era

Information retrieval before statistics. Simple logic, no ranking.

How It Works

Documents must contain exact keywords combined with logical operators.

AND → both terms must appear
OR → either term matches
NOT → exclude this term

Query: maternity AND leave AND policy NOT termination

Why It Failed

No ranking — every matching document is equal. 10,000 results, no idea which is best.

No synonyms — query “car” misses documents saying “automobile”.

Brittle — one typo, zero results. Expert users only.

No context — “python AND programming” matches a document about programming AND snakes separately.

Boolean search was revolutionary in the 1970s — MEDLINE (1971) and DIALOG (1972) made entire scientific databases searchable for the first time. But it requires users to think like a computer, not a human. TF-IDF replaced it by adding statistical ranking.

TF-IDF — The Classic Approach

The foundation of keyword search. Still used today in hybrid systems.

TF-IDF(t, d) = TF(t, d) × IDF(t)

TF (Term Frequency)

How often does the word appear in this document?

TF(t, d) = count(t in d) / |d|

More occurrences = more relevant

IDF (Inverse Document Frequency)

How rare is this word across all documents?

IDF(t) = log(N / df(t))

“the” appears everywhere = low IDF
“maternity” is rare = high IDF

TF-IDF works well for exact keyword matching. But it fails when the user says “parental leave” and the document says “maternity policy.”

Note: Modern implementations use smoothed IDF = log(N / (df+1)) + 1 to avoid division by zero.

Spärck Jones, “A Statistical Interpretation of Term Specificity and Its Application in Retrieval”, Journal of Documentation, 1972

BM25 — The Industry Standard

The default ranking in Elasticsearch, Solr, OpenSearch. Still the strongest sparse retriever.

BM25(q,d) = ∑_i IDF(t_i) · ^{TF(t_i,d) · (k₁+1)}⁄_{TF(t_i,d) + k₁·(1−b+b·|d|/avgdl)}

1. Term Saturation (k₁)

The 100th “policy” adds almost nothing to the score.

k₁ ∈ [1.2, 2.0] controls saturation speed. TF-IDF has no cap.

2. Length Normalization (b)

“policy” 3× in a 2-page memo vs. 50× in a 100-page manual.

b ∈ [0, 1] controls length penalty. b=0.75 is default. The memo wins.

⚙️ BM25 Scoring in Action — Query: “sick leave policy”

Inverted index finds 6 candidate chunks. BM25 scores each one:

Chunk	Has “sick”?	Has “leave”?	Has “policy”?	BM25 Score
chunk3	✅	✅	✅	6.48 🏆
chunk7	✅	✅	✅	5.90
chunk41	✅	❌	❌	3.36
chunk12	❌	✅	❌	1.68
chunk88	❌	✅	❌	1.68
chunk1	❌	❌	✅	1.44

Score = IDF(“sick”)×TF_norm + IDF(“leave”)×TF_norm + IDF(“policy”)×TF_norm — summed across query terms. “sick” is rare (IDF=4.2) so contributes more than “policy” (IDF=1.8).

BM25 rewards: (1) more query terms matched • (2) rarer words = bigger bonus • (3) more occurrences in a short chunk. chunk3 wins because it contains all 3 terms and “sick” is rare.

Robertson & Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond”, Foundations and Trends in IR, 2009

From Formula to Results — How Retrieval Works

In RAG, the unit being indexed and retrieved is always a chunk — not the original full document. Each method works on chunks.

📋 Sparse Retrieval (BM25)

Uses an inverted index — a lookup table: term → [chunk_ids that contain it]

1. Query: “sick leave policy”
2. Look up each term in index:
    “sick” → [chunk3, chunk7, chunk41]
    “leave” → [chunk3, chunk7, chunk12, chunk88]
    “policy” → [chunk1, chunk3, chunk7]
3. Candidate set: union = [chunk1, chunk3, chunk7, chunk12, chunk41, chunk88]
4. Compute BM25 score for each candidate chunk
5. Sort by score → return top-k chunks

⚡ Fast: only scores chunks that share at least one query term

🔢 Dense Retrieval (Embeddings)

Uses a vector index (HNSW) — stores chunk embeddings in a navigable graph

1. Query: “sick leave policy”
2. Embed query → [0.21, −0.45, 0.78, ...]
3. Search HNSW graph for nearest chunk vectors
(no word lookup — geometric search)
4. Candidate set: top-k chunks by cosine similarity
5. Return those chunks to the LLM

⚡ Fast: HNSW navigates from coarse to fine — O(log n)

In RAG: chunk = the unit of retrieval. Both BM25 and dense retrieval index chunks, score chunks, and return chunks — never the full source document. This is why chunking quality directly determines retrieval quality.

Dense Retrieval — The Semantic Leap

Instead of matching keywords, we match meaning. The key intuition: similar meanings point in the same direction in vector space.

Sparse (BM25)

Each chunk is a bag of words.

Vector has 50,000+ dimensions (one per word in vocabulary).

Most values are zero (sparse).

Match = shared words exist. No synonyms.

Dense (Embeddings)

Each chunk is a meaning direction in space.

Vector has 384–3072 dims. All non-zero.

Cosine measures angle between two directions.

cos(A,B) = ^A⋅B⁄_{||A|| · ||B||} ∈ [−1, 1]

1 = same direction = same meaning | 0 = unrelated | −1 = opposite

“vacation days” → embed → [0.21, −0.45, 0.78, ...]

“time off” → embed → [0.19, −0.41, 0.82, ...]

cosine similarity = 0.89 — BM25 score = 0 (no shared words)

💡 Unit-normalized vectors: &lVert;e&rVert; = 1 ⇒ cos(A,B) = A⋅B (how FAISS/ChromaDB compute similarity internally)

Dense retrieval finds meaning, not keywords. Two texts are similar if they point in the same direction — even when they share zero words.

The Scale Problem — Why We Need ANN

Cosine similarity gives the score. But how do you search 1 million vectors fast enough?

❌ Brute Force (KNN)

Compare query against every vector.

1M × 384 dims = 384M operations → ~300ms

Too slow for real-time use.

✅ Approximate Nearest Neighbor (ANN)

Build a smart index that enables shortcuts.

~20 comparisons instead of 1M → ~1ms

300× faster. Trades ~2–5% accuracy for speed.

All modern vector databases use ANN algorithms — most commonly HNSW (Hierarchical Navigable Small World). We’ll cover exactly how HNSW works in Section 4.

Two Retrieval Approaches — Which to Use?

Two fundamentally different ways to measure relevance. Different trade-offs.

⚡ Bi-Encoder (start here)

Query and chunk are embedded separately. Relevance = cosine similarity between their vectors.

sim(q, d) = cos( embed(q), embed(d) )

✅ All chunk embeddings precomputed offline — at query time: one embedding call + ANN search = milliseconds

✅ Works with any embedding model (MiniLM, BGE-M3, OpenAI, etc.)

⚠️ Query and chunk never "interact" — relevance is geometric distance only

🎯 Cross-Encoder (costly)

Query and chunk are processed together by the model. Output: one relevance score.

score = model( query + chunk ) → 0.94

✅ Model reads query against chunk — full interaction = higher accuracy

❌ Cannot precompute anything — must run the model once per candidate at query time

❌ Too slow to run on 100,000 chunks — only practical on a small candidate set

🔁 Combine Both — The Production Pattern

📦 All 100,000 chunks

→
Stage 1: Bi-encoder
ms — fast filter

Top 100

→
Stage 2: Cross-encoder
re-rank 100 only

Top 3–5

→

🤖 LLM

Cross-encoder runs 100 times — not 100,000. Bi-encoder eliminates 99,900 chunks first.

Start with bi-encoder only — it's fast, simple, and works well. Add a cross-encoder re-ranker only when you need higher precision. Deep dive in Lecture 5.

Embedding Models — The Current Landscape

The bi-encoder works with any embedding model. The field moves fast — here is where it stands in 2025.

Model	Dims	Tokens	Cost	Status / Use
MiniLM-L6-v2	384	256	Free	⚠️ Legacy — use for this lab only
Qwen3-Embedding-0.6B	1024	32768	Free (local)	✅ Modern lightweight — replaces MiniLM
BGE-M3	1024	8192	Free (local)	✅ Production workhorse — dense + sparse + hybrid
text-embedding-3-large	3072	8191	Paid API	☁️ Safe cloud default — supports MRL
Gemini Embedding 2	3072	32768	Paid API	☁️ Top-tier — long docs, multimodal

🏠 Self-hosted?

BGE-M3 — best open-source choice. Unique: one model for dense, sparse, AND hybrid retrieval. 100+ languages. Data stays local.

☁️ Cloud API?

text-embedding-3-large or Gemini Embedding 2. Easy, high quality. text-3-large supports Matryoshka — truncate dims without quality loss.

⚠️ Critical rule: index model = query model. Different models = different vector spaces = garbage similarity scores. Also: don't trust MTEB rankings alone — benchmark on your own data.

Hybrid Retrieval — Best of Both Worlds

Production systems combine keyword and semantic search.

BM25

Keyword precision

“Article 47” → exact match

Dense

Semantic recall

“time off” → “vacation”

Hybrid

Precision + recall

Ranked by combined score

score(q, d) = α · BM25(q, d) + (1 − α) · cos(embed(q), embed(d))

Example — query: “sick leave regulations”

BM25 (normalized): 0.72 | Dense: 0.85

Hybrid (α=0.4): 0.4×0.72 + 0.6×0.85 = 0.799

Why normalization is required

BM25 scores can be 0–20+. Cosine scores are 0–1. You cannot add them raw — BM25 would always dominate.

→ Normalize each to [0,1] first (min-max or softmax), then apply α.

Advanced alternative: Reciprocal Rank Fusion (covered in L5) — avoids score normalization entirely.

Craswell et al., “MS MARCO: Benchmarking Ranking Models in the Large-Data Regime”, SIGIR 2021

Section 3 of 5

Document Processing
& Chunking Theory

The hidden bottleneck of every RAG system

Why Chunking Matters

Documents are too long to embed as a single vector. We must split them into chunks.

A 50-page HR manual = ~25,000 tokens

Embedding models have a limit of 256 tokens (MiniLM-L6-v2) or 8192 tokens (BGE-M3)

Even if you could embed it all, one vector cannot capture 50 different topics

The embedding dilution problem

Long text → the embedding becomes an average of everything → it matches nothing well.

Analogy: mix all paint colors together → you get grey. The grey matches no specific color. Same with meaning: average of 50 topics = matches none of them.

The retrieval granularity problem

User asks about “sick leave” → you need just those 3 paragraphs, not the whole manual.

Returning the full 50-page manual as context would fill the entire LLM context window with noise.

Chunking is not just a technical step — it determines retrieval quality. Bad chunks = bad answers.

Three Chunking Strategies

1. Fixed-Size

Split every N characters. Simple but cuts mid-sentence. Not recommended.

2. Recursive / Structural ⭐

Split on natural boundaries: paragraphs → sentences → characters. Preserves semantic coherence.

Try \n\n (paragraph) first. If the piece is still longer than your target chunk size (e.g. 400 tokens), try . (sentence). Still too long? Try commas. Go coarse to fine until each piece fits.

3. Semantic

Use embeddings to detect topic shifts. Most accurate but most complex.

split at i ⇔ cos(emb(s_i), emb(s_i+1)) < θ

Compare adjacent sentence embeddings. When similarity drops below threshold θ, start a new chunk.

Recursive chunking is the best default for most RAG systems.

The Chunk Size Trade-Off

❌ Too Small (50–100 tokens)

High retrieval precision
But context fragmentation
“...must submit the request to...”
No surrounding context → useless

❌ Too Large (1000+ tokens)

Full context preserved
But retrieval dilution
Embedding averages over many topics
Returns irrelevant surrounding text

✅ Sweet Spot: 300–500 tokens

With 50–100 token overlap between consecutive chunks

Overlap = the end of chunk N is repeated at the start of chunk N+1

Chunk 1: “...mandatory maternity leave. During this period, they receive 80%”
Chunk 2: “During this period, they receive 80% of their salary. Optional parental...”
▲ overlap zone — appears in both chunks

Without overlap, a sentence at the boundary might be split across two chunks and lost to both. Overlap guarantees boundary context is always retrievable.

Chunking in Practice — All Three Approaches

Same source text. Target size: 400 tokens. | “Employees are entitled to 5 months of mandatory maternity leave. During this period, they receive 80% of their salary. Optional parental leave can be extended up to 6 additional months at reduced pay. Requests must be submitted to HR at least 30 days in advance.”

❌ 1. Fixed-Size

Split every 50 characters. No understanding of language.

Chunk 1: “Employees are entitled to 5 months of mand”
Chunk 2: “atory maternity leave. During this peri”
Chunk 3: “od, they receive 80% of their salary...”

Word “mandatory” split in half. Neither chunk is meaningful alone.

✅ 2. Recursive ⭐

Try paragraph → sentence → comma. Stop when ≤ 400 tokens.

Chunk 1: “Employees are entitled to 5 months of mandatory maternity leave.”
Chunk 2: “During this period, they receive 80% of their salary.”
Chunk 3: “Optional parental leave... up to 6 months at reduced pay. Requests must be submitted 30 days in advance.”

Each chunk = one complete idea. Clean boundaries, full sentences.

🧠 3. Semantic

Embed each sentence. Split where cosine similarity drops below θ = 0.7.

s1→s2: cos = 0.91 (same topic, keep)
s2→s3: cos = 0.88 (still related, keep)
s3→s4: cos = 0.41 → topic shift! split here

Chunk 1: sentences 1–3 (leave & salary)
Chunk 2: sentence 4 (HR process)

Most accurate. Needs an embedding model just to chunk — more complex.

Recursive is the best default. Add semantic chunking only when topics shift within paragraphs and retrieval quality is still poor after tuning chunk size.

Metadata — The Fourth Dimension

Every chunk carries structured data alongside its text and embedding.

A chunk is not just text. It is a tuple: (text, embedding, metadata)

Metadata Field	Example	Why It Matters
`source`	hr_policy_2024.pdf	Source attribution — users can verify
`page`	12	Precise reference
`section`	Maternity Leave	Hierarchical filtering
`date`	2024-01-15	Recency filtering
`department`	HR	Access control / routing

🤔 How does the system know to filter by 2024?

User asks: “What was the remote work policy in 2024?” — How does “in 2024” become date = 2024?

1. LLM parses query
An LLM reads the question first, extracts {"date": "2024"}, then the retriever uses that as a filter. Most common.

2. UI filter
User picks “Year: 2024” from a dropdown before asking. No extraction needed.

3. Rule-based
Regex detects date patterns (e.g. \b20\d{2}\b) and maps to metadata. Simple but brittle.

Metadata enables hybrid filtering: vector search + structured filter. But filters don’t appear magically — your system must extract or collect them before retrieval. Deep dive in Lecture 5 (query routing).

Section 4 of 5

Vector Databases

How similarity search works at scale

The Search Problem at Scale

You have 1 million chunks, each with a 384-dim embedding. A user asks a question. How do you find the most similar vectors?

❌ Brute Force (KNN)

Compare query to every vector.

O(n · d) — 1M × 384 = 384M ops

~300ms per query. Perfect recall, unusable speed.

✅ Approximate Nearest Neighbor (ANN)

Build an index that creates intelligent shortcuts.

O(log n) to O(n^0.5) — sublinear

~1ms per query. 95%+ recall, 300× faster.

Three families of ANN algorithms

Algorithm	Idea	Used By
HNSW	Multi-layer navigable graph — coarse to fine	ChromaDB, Qdrant, pgvector
IVF (Inverted File)	Partition space into clusters, search only nearby clusters	FAISS, pgvector
PQ (Product Quantization)	Compress vectors to reduce memory & speed up comparisons	FAISS (large-scale)

ANN trades 2–5% accuracy for 300× speed. HNSW is the most common. You don’t implement it — your vector database does it for you.

HNSW — How It Actually Works

Hierarchical Navigable Small World — the default algorithm in ChromaDB, Qdrant, and most vector databases.

🗺️ The City Navigation Analogy

HNSW builds multiple layers of a navigable graph. To find the nearest chunk:

🌍 Top layer
Few nodes, long jumps
World map → Europe

🗾 Mid layer
More nodes, medium jumps
Country map → Milan

🏙️ Bottom layer
All nodes, short steps
Street map → exact restaurant

🔍 Search algorithm: greedy navigation

1. Start at a random node on the top layer

2. Greedily move to whichever neighbor is closest to the query vector

3. When no neighbor is closer → drop to the next layer at the same node

4. Repeat until the bottom layer → return the nearest nodes as results

Each layer dramatically narrows the search space. Total: ~20 hops instead of 1M scans.

You never implement HNSW yourself. collection.query() in ChromaDB runs it automatically. Understanding the intuition helps you tune it (e.g., ef_search, M parameters).

Malkov & Yashunin, “Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs”, IEEE TPAMI, 2020

Vector Database Landscape (2025)

Three categories. The trend: every major database now supports vectors. Choose based on your existing stack.

🔷 Purpose-Built Vector Databases

Database	Type	ANN	Best For
ChromaDB	Embedded (Python)	HNSW	Prototyping, learning — our lab
Qdrant	Client-Server	HNSW	Production RAG, advanced filtering
Pinecone	Managed Cloud	Proprietary	Zero-ops enterprise
Weaviate	Client-Server	HNSW	Multi-modal, hybrid search built-in

🔶 Traditional DBs + Vector Extensions

Database	Extension	ANN	Best For
PostgreSQL	`pgvector`	IVFFlat / HNSW	Already on Postgres — no new infra
MongoDB	Atlas Vector Search	HNSW	Already on Mongo — document + vector
Elasticsearch	kNN search (8.x+)	HNSW	Already on Elastic — keyword + vector
Neo4j	Vector Index	HNSW	GraphRAG — relationships + vectors

🔧 Libraries (not databases)

Library	By	ANN	Best For
FAISS	Meta	IVF, PQ, HNSW	Research, billion-scale, GPU
Annoy	Spotify	Random projection trees	Read-heavy, memory-mapped

Decision guide: New project → ChromaDB (prototype) then Qdrant (production). Already have Postgres → pgvector. Already have MongoDB → Atlas Vector Search. Need graph relationships → Neo4j. Need billion-scale → FAISS.

The Complete RAG Pipeline

Five components, each with its own design decisions.

#	Component	Key Decision	Our Choice (Lab)
1	Document Loader	Format handling	`pypdf`
2	Text Splitter	Chunk size, overlap, strategy	Recursive, 400 / 80
3	Embedding Model	Quality vs. speed vs. cost	MiniLM (384d)
4	Vector Database	Scale, features, hosting	ChromaDB
5	Generator (LLM)	Model, prompt template	Gemini 2.0 Flash

The embedding model used at index time must be identical to the one used at query time — different models live in different vector spaces.

🔬 You will build this exact pipeline in the Lecture 5 combined lab.

Section 4 — Pipeline & Prompt Design

The RAG Prompt — Design Principles

You already master prompting from Lecture 3. Now apply it with grounding constraints specific to RAG.

The prompt template determines answer quality as much as retrieval does.

✅ Grounding Instruction

“Answer based ONLY on the provided context” — prevents hallucination

✅ Refusal Instruction

“If the answer is not in the context, say: I don’t have enough information”

✅ Source Citation

“Always cite which document the information comes from”

❌ Common Mistake

Not including grounding → model ignores context and uses its own knowledge → hallucination

prompt = f"""Answer ONLY from the context below. If not found, say: I don’t have this information. Cite the source.
Context: {context}
Question: {question}"""

Section 4 — Pipeline & Prompt Design

RAG Failure Modes

Understanding failures helps you debug and improve your system.

Failure	Symptom	Root Cause	Solution
Wrong chunks	Answer about wrong topic	Embedding model too weak	Better model; hybrid search → L5
Chunks too small	Incomplete answers	Context fragmented	Increase chunk size
Chunks too large	Irrelevant noise in answer	Diluted embeddings	Decrease chunk size
No relevant docs	Hallucinated answer	Missing refusal prompt	Add “I don’t know” instruction
Ignores context	Uses own knowledge	Weak grounding prompt	Stronger “ONLY context” instruction
Stale answers	Outdated information	Documents not re-ingested	Schedule periodic re-indexing

Most RAG failures are retrieval failures, not generation failures. Fix retrieval first.

Section 5 of 5

Decision Matrix

The most important decision you will make on every project

Three Tools, Three Problems

💬

Prompting

Change the instructions

Model stays unchanged

Zero data, zero compute

📚

RAG

Give the model knowledge

External docs at query time

Low cost, fast updates

🔧

Fine-Tuning

Change the model’s behavior

Retrain on your data

High cost, high customization

Each tool solves a different problem. Using the wrong tool wastes time and money.

The Decision Matrix

Criterion	💬 Prompting	📚 RAG	🔧 Fine-Tuning
Setup effort	Minutes	Hours	Days–Weeks
Data required	None	Documents	100–10K examples
Compute cost	$0	Low ($)	High ($$$)
Update speed	Instant	Minutes	Hours
Customization	Low	Medium	High
Hallucination	High risk	Reduced (grounded)	Medium
Best for	General tasks	Knowledge Q&A	Style / domain

Use Prompting first (minutes), then RAG if you need external knowledge (hours), then Fine-Tuning only if behavior/style needs changing (days). Each escalation multiplies cost by 10×.

Decision Flowchart

Does the model already know the answer well enough?

Yes → 💬 Prompting

↓ No

Is the knowledge in documents you can provide?

Yes → 📚 RAG

↓ No

Do you need specific behavior, format, or domain expertise?

Yes → 🔧 Fine-Tuning (Lecture 9)

↓ Still unsure?

Try RAG + better prompts first. Fine-tune only if that fails.

Cloud vs. Local Inference

	☁️ Cloud API	🏠 Local Inference
Setup time	5–10 min (API key + env)	15–30 min with Ollama / LM Studio 2–4 h only if full CUDA GPU drivers needed
Cost model	Per token ($)	Fixed (hardware cost)
Privacy	Data leaves premises	Data stays local
Models	GPT-4, Gemini, Claude	LLaMA, Mistral, Phi
Quality	Best available	Smaller, quantized
Tools	API calls	Ollama, vLLM, LM Studio, 🤗 HuggingFace

Start with cloud (development & prototyping). Move to local when privacy, cost, or latency requires it.

Apply the Matrix — 5 Real Scenarios

1. “Summarize this meeting transcript”

→ 💬 Prompting — general task, model already knows how. Just good instructions.

2. “Answer questions about our HR policies”

→ 📚 RAG — knowledge is in company documents. Ground answers in retrieved chunks.

3. “Customer support chatbot for our product manuals”

→ 📚 RAG (not prompting!) — domain-specific product knowledge. Prompting alone would hallucinate specs.

4. “Auto-categorize incoming support tickets”

→ 💬 Prompting first, then 🔧 Fine-Tune if accuracy isn’t enough. Start simple, escalate only if needed.

5. “Generate medical reports in our specific format”

→ 🔧 Fine-Tuning — needs specific style, clinical terminology, domain language.

Lecture 4 — Key Takeaways

1. RAG = Retrieve → Augment → Generate — give the model an open book

2. Search evolved from keywords (TF-IDF, BM25) to meaning (dense embeddings)

3. Chunking determines quality — 300–500 tokens, recursive, with overlap

4. Vector databases use HNSW for millisecond similarity search at scale

5. Prompt design matters — grounding, refusal, and citation instructions

6. Prompting → RAG → Fine-Tuning — always try the simpler approach first

7. Most RAG failures are retrieval failures — fix chunking and embeddings before touching the LLM

❓

Questions?

Thank you!

Kamyar Zeinalipour — Zucchetti LLM Course

Next → Lecture 5: Advanced RAG & Intelligent Routing

Theory: Re-ranking · HyDE · Query decomposition · Semantic routing

🔬 Combined Lab: RAG Pipeline (L4) + Query Router (L5) — back-to-back hands-on session

RAG Foundations& Decision Matrix

Today’s Agenda

The Retrieval Problem

The Fundamental Problem

🧊 Frozen Knowledge

🎭 Hallucination Under Pressure

Two Ways to Add Knowledge

Approach 1: Fine-Tuning

Approach 2: RAG ⭐

Retrieval-Augmented Generation

RAG in Action — Step by Step

👤 User asks: “What is the maternity leave policy?”

🔍 Step 1: Retrieve — system searches vector DB, finds 3 relevant chunks

✅ Step 2: Augment & Generate — LLM reads chunks and produces a grounded answer

Why RAG Works — The Grounding Principle

❌ Without RAG — Hallucination

✅ With RAG — Grounded

RAG vs. Fine-Tuning — When to Use Each

10-Minute Break

From Keywordsto Vectors

The Evolution of Search

Boolean Search — The First Era

How It Works

Why It Failed

TF-IDF — The Classic Approach

TF (Term Frequency)

IDF (Inverse Document Frequency)

BM25 — The Industry Standard

1. Term Saturation (k1)

2. Length Normalization (b)

⚙️ BM25 Scoring in Action — Query: “sick leave policy”

From Formula to Results — How Retrieval Works

📋 Sparse Retrieval (BM25)

🔢 Dense Retrieval (Embeddings)

Dense Retrieval — The Semantic Leap

Sparse (BM25)

Dense (Embeddings)

The Scale Problem — Why We Need ANN

❌ Brute Force (KNN)

✅ Approximate Nearest Neighbor (ANN)

Two Retrieval Approaches — Which to Use?

⚡ Bi-Encoder (start here)

🎯 Cross-Encoder (costly)

🔁 Combine Both — The Production Pattern

Embedding Models — The Current Landscape

🏠 Self-hosted?

☁️ Cloud API?

Hybrid Retrieval — Best of Both Worlds

BM25

Dense

Hybrid

Example — query: “sick leave regulations”

Why normalization is required

Document Processing& Chunking Theory

Why Chunking Matters

The embedding dilution problem

The retrieval granularity problem

Three Chunking Strategies

1. Fixed-Size

2. Recursive / Structural ⭐

3. Semantic

The Chunk Size Trade-Off

❌ Too Small (50–100 tokens)

❌ Too Large (1000+ tokens)

✅ Sweet Spot: 300–500 tokens

Chunking in Practice — All Three Approaches

❌ 1. Fixed-Size

✅ 2. Recursive ⭐

🧠 3. Semantic

Metadata — The Fourth Dimension

🤔 How does the system know to filter by 2024?

Vector Databases

The Search Problem at Scale

❌ Brute Force (KNN)

✅ Approximate Nearest Neighbor (ANN)

Three families of ANN algorithms

HNSW — How It Actually Works

🗺️ The City Navigation Analogy

🔍 Search algorithm: greedy navigation

Vector Database Landscape (2025)

RAG Foundations
& Decision Matrix

From Keywords
to Vectors

1. Term Saturation (k₁)

Document Processing
& Chunking Theory