Zucchetti LLM Course — Lecture 5 of 10

Advanced RAG
& Intelligent Routing

Beyond naive retrieval — making RAG systems that actually work on complex enterprise data

Kamyar Zeinalipour · 2 hours

Today’s Agenda

Section 1: Why Naive RAG Fails (15 min)

Six failure modes you’ll encounter in production

Section 2: Advanced Retrieval Strategies (40 min)

Re-ranking · Cross-encoders · HyDE · Parent-child chunking

Section 3: Query Decomposition (25 min)

Breaking complex questions into answerable sub-queries

Section 4: Semantic Routing (20 min)

Sending queries to the right data source automatically

Pure theory today. Lab is in Lecture 6 — combined mega-lab covering L4 + L5 + L6 content.

Section 1 of 4

Why Naive RAG Fails

Diagnosing the six failure modes

The Naive RAG Pipeline

Query

→

Embed

→

Top-K Search

→

Stuff into Prompt

→

LLM

→

Answer

This is what we built in L4. It works for simple cases. It breaks on everything else.

If your RAG system returns “I don’t know” when the answer is clearly in your documents — the problem is almost always retrieval, not the LLM.

Six Ways Naive RAG Fails

1. Semantic Mismatch

Query uses different words than the document.
Q: “sick leave policy” → Doc says “absence due to illness”

2. Chunk Boundary Split

The answer spans two chunks, but only one is retrieved.
Table header in chunk 1, data rows in chunk 2.

3. Context Dilution

Top-K returns 10 chunks but only 1 is relevant. The signal drowns in noise.

4. Multi-Hop Failure

Answer requires connecting info from 2+ documents.
“Compare policy A with policy B”

5. Recency Blindness

Old and new versions of a document both match. System returns outdated info.

6. Wrong Granularity

Chunks are too big (diluted) or too small (no context). Neither gives a good answer.

The Fix Map

Failure Mode	Fix	Section
Semantic mismatch	HyDE — generate a hypothetical answer, embed that instead	S2
Wrong chunks retrieved	Re-ranking — cross-encoder precision pass after coarse retrieval	S2
Chunk boundary splits	Parent-child chunking — retrieve child, pass parent context	S2
Context dilution	Re-ranking + fewer chunks — quality over quantity	S2
Multi-hop questions	Query decomposition — break into sub-queries	S3
Wrong data source	Semantic routing — route to the right system	S4

Each technique in today’s lecture solves a specific failure mode. Don’t apply them all blindly — diagnose first, then fix.

Section 2 of 4

Advanced Retrieval

Re-ranking · HyDE · Parent-child chunks

What Is Re-ranking?

The single highest-ROI improvement for any RAG system

The Problem

Standard vector search returns the most similar documents — but similar ≠ relevant.

You search for "sick leave policy" and get 10 chunks about leave policies. But chunks #7 and #3 are the ones that actually answer the question. Chunks #1-2 are just vaguely similar.

The Fix: Re-ranking

After initial retrieval, pass each result through a more powerful model that scores how well it actually answers the query.

Result: the truly relevant chunks move to positions #1 and #2. Noise drops to the bottom.

Re-ranking = a second pass with a smarter model. First pass casts a wide net (recall). Second pass picks the best fish (precision). +15–30% precision improvement.

Bi-Encoder vs Cross-Encoder

Why we need two different models

Bi-Encoder STAGE 1

Query → [Encoder] → vec_q
Doc → [Encoder] → vec_d

score = cos(vec_q, vec_d)

Encoded separately. Doc vectors stored in DB.
Search = nearest neighbor lookup.
Fast: ~1ms per 1M docs

Cross-Encoder STAGE 2

query + doc → [Encoder] → score

Both texts fed together into one encoder.
Full cross-attention between query and doc.
Output: a single relevance score (0–1).

Cannot precompute. Must run for every pair.
Slow but much more accurate

Bi-Encoder: O(1) per query | Cross-Encoder: O(N) per query
→ Use bi-encoder for speed (Stage 1), cross-encoder for accuracy (Stage 2)

Two-Stage Retrieval

Putting it together: fast recall + precise re-ranking

Query

→

Stage 1: Bi-Encoder
Fast, approximate
Retrieve top-50

→

Stage 2: Cross-Encoder
Slow, precise
Re-rank to top-5

→

LLM

Stage 1: Recall RECALL

Recall = “don’t miss the right answer”

Encode query and docs independently
Cosine similarity for ranking
~1ms per 1M docs (HNSW)
Retrieves many candidates (high recall)

Stage 2: Precision PRECISION

Precision = “only return the right answers”

Encodes query + doc together
Full attention between them
~100–200ms total for top-50
Much more accurate (high precision)

Embedding Model ≠ Re-ranker

Two different models, two different jobs — don’t confuse them

Stage 1: Embedding Model

What it does: Converts text into a vector

How it searches: Cosine similarity between vectors

Speed: ~1ms per 1M docs

Accuracy: Good (approximate)

Examples:
MiniLM · BGE-M3 · OpenAI text-embedding-3

Stage 2: Re-ranker Model

What it does: Reads query + doc together

How it scores: Direct relevance score (0–1)

Speed: ~100–200ms for 50 docs

Accuracy: Excellent (precise)

Examples:
Cohere Rerank · BGE Reranker · ms-marco

Analogy: The embedding model is like a librarian who points you to the right shelf. The re-ranker is like an expert who reads each book and tells you which one actually answers your question.

Re-ranking in Practice

Re-rankers are cross-encoder models — not embedding models. They score (query, doc) pairs directly.

Cross-Encoder Models for Stage 2

Model	Type	Quality	Notes
Cohere Rerank	Cloud API	Excellent	Easiest to integrate, pay per call
BGE Reranker v2	Local (GPU)	Very good	Open source, best for on-prem
cross-encoder/ms-marco	Local (GPU)	Good	HuggingFace classic, well-tested
ColBERT v2	Local (GPU)	Good	Late interaction — faster than cross-encoder

Don’t Confuse Them

Stage 1 (embedding model): MiniLM, BGE-M3, OpenAI → produces vectors, search by cosine similarity
Stage 2 (re-ranker): Cohere, BGE Reranker → reads query + doc together, outputs a relevance score

Rule of thumb: Re-ranking improves precision by 15–30% on most benchmarks. It’s the single highest-ROI improvement you can add to a naive RAG pipeline.

HyDE — The Problem It Solves

When the question and the answer use completely different words

Semantic Mismatch

User asks:

“What is our policy on working from home?”

Real document says:

“Remote work arrangements are governed by the Telecommuting and Hybrid Work Guidelines, Section 4.2...”

“working from home”

≠ “remote work arrangements”
≠ “telecommuting”
≠ “hybrid work guidelines”

Embedding similarity: LOW — different vocabulary

The Insight

Questions and answers use different language. But answers and other answers use similar language.

What if we generate a fake answer first?

HyDE idea: Instead of searching with the question, generate a hypothetical answer and search with that instead.

HyDE — How It Works

Two LLM calls: one to search better, one to answer

Step 1 — LLM Call #1: Generate Fake Answer EXTRA

User Query

→

LLM

→

Hypothetical Answer
(may be wrong — that’s OK)

Step 2 — Search with the Fake Answer

Hypothetical Answer

→

Embed

→

Vector Search

→

Real Documents

Step 3 — LLM Call #2: Generate Real Answer STANDARD RAG

User Query

Real Documents

→

LLM

→

Final Answer

Cost: 2 LLM calls (1 extra vs normal RAG) · +0.5–2s latency · 📄 Gao et al. (2023), ACL

HyDE — Walk-Through

1. User Query

“What is our policy on working from home?”

2. LLM generates hypothetical answer (NOT from your docs — made up)

“Employees may work from home up to three days per week with manager approval. Remote work arrangements must be documented in the HR system and reviewed quarterly...”

3. We embed this fake answer and search — it matches real docs about:

“remote work arrangements” · “telecommuting policy” · “hybrid work guidelines”

Terms the original question never used — but the fake answer did!

4. Real docs + original query → LLM → correct final answer

The fake answer is thrown away. Only the real docs are used for the final answer.

The fake answer doesn’t need to be correct. It just needs to be in the same semantic neighborhood as the real answer. We never show it to the user.

Query Rewriting — What & Why

Users type bad queries. Fix them before they reach retrieval.

The Pipeline

Bad Query

→

LLM
“Rewrite as a clear search query”

→

Clean Query

→

Embed & Search

→

Docs

→

LLM → Answer

Examples

✗ Raw User Query	✓ Rewritten Query
“that leave thing for new people”	“What is the annual leave policy for newly hired employees?”
“how fix error 500”	“How to troubleshoot HTTP 500 internal server error?”
“can I do the thing remotely”	“Is remote access to the internal ERP system available?”

Cost: 1 extra LLM call (very small output — just one sentence). Cheapest query transformation technique.

Query Rewriting vs HyDE

Both transform the query — but they solve different problems

Query Rewriting

LLM generates: a better question

Solves: vague, informal user input

When: chatbots, search bars, internal tools

Output size: 1 sentence (cheap)

HyDE

LLM generates: a fake answer

Solves: vocabulary mismatch (question ≠ document language)

When: domain-specific jargon, technical docs

Output size: full paragraph (more expensive)

When to Use Which?

Situation	Use
Users type vague, informal queries	Query Rewriting
Question & document use different vocabulary	HyDE
Both problems at once	Rewrite first, then HyDE

Use Rewriting always (cheapest, rarely hurts). Add HyDE only when vocabulary mismatch is the problem. We’ll add a third technique — Decomposition — in Section 3.

Parent-Child Chunking — The Problem

The fundamental trade-off in chunk size

Small Chunks (100–200 tokens)

✓ Precise retrieval — the right chunk matches exactly
✗ Lost context — missing surrounding sentences, section headers, the paragraph before

LLM gets the right sentence but can’t understand the full picture

Large Chunks (500–1000 tokens)

✓ Rich context — full sections with headers and transitions
✗ Diluted relevance — the matching sentence is buried in noise

LLM gets too much text, the answer is lost inside it

You can’t win with a single chunk size. Small = precise but no context. Large = context but imprecise. What if you could have both?

Parent-Child Chunking — The Solution

Use two levels: search on small, read on big

How It Works

Document

→

Split into
Parent Chunks
(500–1000 tokens)

→

Split each parent
into Children
(100–200 tokens)

→

Index the
Children only

At Search Time

Step 1: Query matches a child chunk (precise)
Step 2: Look up which parent that child belongs to
Step 3: Send the parent to the LLM (full context)

Why It Works

Search accuracy: small chunks match precisely
LLM context: parent gives full section with headers
Cost: ~2× index size (storage is cheap)

Search on small, read on big. Best of both worlds. Like finding a paragraph using a book’s index (precise), then reading the whole page (context).

Parent-Child Chunking — Example

PARENT CHUNK (sent to LLM) — 800 tokens

Section 3.2: Remote Work Policy

Full section with eligibility, process, exceptions, and appeals...

Child 1
eligibility requirements

Child 2
← SEARCH MATCHED THIS

Child 3
exceptions & appeals

What Happened

Step	Detail
User asks:	“How do I apply for remote work?”
Search matches:	Child 2 — the application process paragraph (precise!)
Sent to LLM:	Parent — the full Remote Work Policy section (eligibility + process + exceptions)
Result:	LLM gives complete answer including eligibility criteria the user didn’t ask about

Without parent-child: LLM sees only the application process. With parent-child: LLM sees the full policy and can say “you’re eligible if...”

Hybrid Search — The Problem We Left

In L4, we introduced: score = α · BM25 + (1−α) · Dense. But there’s a catch.

The Normalization Headache

Ranker	Score Range	Example
BM25	0 to 20+	12.4
Dense (cosine)	-1 to 1	0.87

If you add them raw: 12.4 + 0.87 = 13.27 — BM25 always dominates. Dense is meaningless.

Min-max normalization helps, but is fragile: depends on the batch, changes between queries, hard to tune α.

What If We Didn’t Need Scores At All?

What if we only used the rank positions — “this doc was #1 in BM25 and #3 in Dense” — instead of raw scores?

That’s exactly what Reciprocal Rank Fusion (RRF) does.

Reciprocal Rank Fusion (RRF)

Combine rankings without scores — just positions

RRF(d) = Σ_r∈rankers 1 / (k + rank_r(d)) where k = 60

Example — Merging BM25 and Dense results

Document	BM25 Rank	Dense Rank	RRF Score	Final Rank
Doc A	#1	#3	1/61 + 1/63 = 0.0323	#1
Doc B	#4	#1	1/64 + 1/61 = 0.0320	#2
Doc C	#2	#5	1/62 + 1/65 = 0.0315	#3

Documents that rank high in both rankers rise to the top. No normalization needed.

When to Use Score Fusion (α)

You control both rankers
Scores are already on similar scales
You want fine-grained control via α

When to Use RRF

Different rankers with different scales
3+ rankers (BM25 + Dense + re-ranker)
Simple, robust, no tuning needed

Hierarchical Indexing — The Problem

Flat search breaks down at scale

Flat Search: 500,000 chunks

10,000 documents × ~50 chunks each = 500K chunks to search through

User asks: “What is the sick leave policy?”

What Flat Search Returns

Rank	Chunk From	Why It Matched	Correct?
#1	HR Policy Manual	Contains “sick leave”	✓
#2	Employee Handbook Intro	Mentions “policies” generically	✗
#3	IT Security Policy	Contains “leave your device...”	✗
#4	Finance Handbook	Mentions “sick day deductions”	~

The answer exists, but it’s competing with 500K chunks from unrelated documents that use similar words.

Hierarchical Indexing — The Solution

Search in layers: broad first, then narrow

Three Levels of Search

Level 1
Document Index
10,000 → top 5

→

Level 2
Section Index
~50 sections → top 10

→

Level 3
Paragraph Index
~30 paragraphs → top 3

→

LLM

The Library Analogy

Level 1: Find the right shelf — HR, not IT
Level 2: Find the right book — Leave Policy, not Hiring
Level 3: Find the right page — Sick leave, not vacation

When to Use

Large collections (1,000+ docs)
Structured docs (manuals, policies, legal)
Similar vocabulary across many docs

Skip if < 100 docs — flat search is fine

Hierarchical Indexing — Example

Query: “What is the sick leave policy?”

Level 1: Which document? (10,000 → 5)

Embed document summaries. Match: HR Policy Manual, Employee Handbook, Benefits Guide, Payroll FAQ, Onboarding Guide

Level 2: Which section? (5 docs × ~10 sections → 10)

Search within those 5 docs. Match: Sick Leave, Annual Leave, Parental Leave, Public Holidays...

Level 3: Which paragraph? (10 sections × ~5 paragraphs → 3)

Match: Sick leave entitlement, Sick leave process, Doctor’s note requirements

Flat Search

500,000 chunks → 3 results
Many near-misses from wrong documents

Hierarchical Search

10K → 5 → 10 → 3 results
All from the right document and section

Contextual Retrieval — The Problem

When you chunk a document, each chunk loses its identity

This chunk is ambiguous

“Employees are entitled to 15 days of paid leave per year.”

Questions the chunk can’t answer

Question	The chunk doesn’t say
Which employees?	Full-time? Part-time? Interns? Contractors?
Which document?	HR Policy Manual? Italian labor law? Company handbook?
Which section?	Annual leave? Sick leave? Parental leave?
Which company?	Parent company? Subsidiary? Joint venture?

The context was in the document header and section title — but they got lost during chunking. The embedding captures “leave” but not “full-time annual leave in HR policy.”

Contextual Retrieval — The Solution

Prepend context to each chunk before embedding

The Pipeline

Document
+ Chunk

→

Generate Context
“Where does this chunk fit?”

→

Context + Chunk
enriched text

→

Embed & Index

Option A: Rule-Based FREE

Prepend: “Document: {title} | Section: {heading}”

Works when docs have clear titles and section headers

Option B: LLM-Generated BETTER

Ask LLM: “Given the full doc and this chunk, write 2–3 sentences explaining where this chunk fits.”

Cost: 1 LLM call per chunk at index time (not query time)

You do this once when building the index, not on every search. The enriched chunks produce better embeddings permanently.

Contextual Retrieval — Before & After

✗ Naive Chunk

“Employees are entitled to 15 days of paid leave per year.”

Matches any query about “leave” — annual, sick, parental, unpaid...

✓ Contextual Chunk

[HR Policy Manual > Section 4: Annual Leave > Full-Time Employees]
“Employees are entitled to 15 days of paid leave per year.”

Matches only “full-time annual leave” queries — precise!

Results

Metric	Without Context	With Context
Retrieval failures	Baseline	−49% (Anthropic, 2024)
When to apply	—	Always — cheapest win
Cost	—	Rule-based: free / LLM: 1 call per chunk at index time

The simplest technique with the biggest payoff. If you do only two things from this section: contextual retrieval + re-ranking.

Lost in the Middle

Even correct retrieval can fail at generation time

The Finding Liu et al. 2023

LLMs pay most attention to information at the beginning and end of the context window. Information in the middle is often ignored.

Why It Matters

You retrieve 10 chunks
The answer is in chunk #5
The LLM focuses on chunks #1-2 and #9-10
Result: wrong answer despite correct retrieval

✓ The Fixes

Fewer chunks — K=3–5 not K=20
Re-rank first — best chunk goes to position #1
Summarize — compress chunks before injecting
Cite position — put key info first in prompt

This is why re-ranking matters beyond precision. It doesn’t just find better chunks — it puts the best chunk first, where the LLM actually reads it.

Section 2: Summary — Your Retrieval Toolkit

Technique	What It Does	When to Use	Cost
Re-ranking	Precision pass after coarse retrieval	Always (highest ROI)	+100–200ms
HyDE	Bridges vocabulary gap	Domain-specific jargon mismatch	+1 LLM call
Parent-child chunking	Search small, read big	Structured docs with sections	2× index size
Hybrid search	BM25 + dense combined	Mixed exact + semantic queries	+BM25 index
Query rewriting	Fix bad queries before search	Always (cheapest win)	+1 LLM call
Hierarchical indexing	Doc → Section → Paragraph	Large collections (1000+ docs)	3× index
Contextual retrieval	Prepend doc context to chunks	Always (cheapest win)	~Free

Stack them: Contextual chunks → Hybrid search → Re-rank → Parent expansion. Each fixes a different failure mode.

☕

Break

10 minutes

Section 3 of 4

Query Decomposition

Breaking complex questions into answerable sub-queries

The Multi-Hop Problem

User asks:

“Compare the sick leave policies for full-time vs part-time employees hired after 2022.”

This single query requires information from at least 3 different places:

Sub-query 1

Sick leave policy for full-time employees

Sub-query 2

Sick leave policy for part-time employees

Sub-query 3

Rules specific to employees hired after 2022

No single chunk contains the full answer. Naive RAG retrieves one partial chunk and gives an incomplete or wrong response.

Strategy 1: Sequential DEPENDENT

Each sub-query depends on the answer to the previous one

The Flow

Sub-query 1

→

Answer 1

→

Sub-query 2
(uses Answer 1)

→

Answer 2

→

Final Answer

Example

“What is the sick leave policy for Mario’s department?”

Step	Sub-query	Answer
1	“What department does Mario work in?”	Engineering
2	“What is the sick leave policy for Engineering?”	15 days/year

Step 2 can’t run until Step 1 finishes — it needs the department name.

Use when: chain of reasoning where each step builds on the previous. Trade-off: slower (sequential), but handles dependent information.

Strategy 2: Parallel INDEPENDENT

All sub-queries run at the same time — no dependencies

The Flow

Original
Query

→

Sub-query 1 → Docs

Sub-query 2 → Docs

Sub-query 3 → Docs

→

LLM
Synthesize

→

Final
Answer

Example

“Compare sick leave for full-time vs part-time employees”

Sub-query	Runs	Result
“Sick leave policy for full-time employees”	Simultaneously	15 days/year
“Sick leave policy for part-time employees”	Simultaneously	8 days/year

Both queries are independent — they can run at the same time.

Use when: comparison queries, multi-entity lookups. Advantage: fast (concurrent). Most common in enterprise use cases.

Strategy 3: Conditional BRANCHING

The next sub-query depends on the answer — like a decision tree

The Flow

Sub-query 1

→

Answer

→

If YES → Sub-query 2a

If NO → Sub-query 2b

Example — IT Troubleshooting

“My application keeps crashing”

Step	Sub-query	Branch
1	“Is this a known bug in version 3.2?”	—
2a	If YES: “What is the fix for bug #3847?”	Known bug path
2b	If NO: “Common crash causes for this application?”	General troubleshooting

Use when: diagnostic flows, troubleshooting, eligibility checks. Trade-off: most complex to implement, but handles “it depends” queries.

Implementation: LLM as Decomposer

The Prompt ONE CALL DOES BOTH

Analyze the user's question.
If it can be answered with a single search, return it
unchanged as a list with one item.
If it requires information from multiple sources, break
it into 2-5 independent sub-queries.

User question: "{original_question}"

Return a JSON list of queries:
[
  "query 1",
  "query 2 (only if needed)",
  ...
]

What happens

Input	LLM returns	Result
“What is sick leave?”	1 query (same)	Direct search — no extra cost
“Compare FT vs PT after 2022”	3 sub-queries	3 parallel searches → synthesize

One LLM call handles both: if the query is simple, it passes through unchanged. If complex, it decomposes. No separate classification step needed.

Section 4 of 4

Semantic Routing

Sending queries to the right data source

The Multi-Source Challenge

Real enterprise data lives in many places. A single vector DB isn’t enough.

📚

Vector DB

Policy documents, manuals, FAQs

“What is our vacation policy?”

🗃️

SQL Database

Employee records, transactions, metrics

“How many employees joined in Q3?”

🌐

API / Live Data

Stock prices, weather, real-time status

“What is the current system status?”

A semantic router classifies the query and sends it to the right system before any retrieval happens.

Approach 1: Embedding-Based Router FAST

Use cosine similarity to match queries to routes — no LLM needed

How It Works

Define Route
Descriptions

→

Embed Each
Route Description

→

At Query Time:
Embed Query

→

Cosine Similarity
Pick Nearest Route

Example

Route	Example Descriptions (embedded at setup)
Vector DB	“company policy”, “employee handbook”, “HR guidelines”
SQL	“how many employees”, “total revenue”, “average salary”
API	“current system status”, “live stock price”, “weather now”

route* = argmax_r cos(embed(query), embed(route_r))

Speed: ~1ms. No LLM call. Just cosine similarity. Best when routes are well-defined and don’t overlap.

Approach 2: LLM-Based Router FLEXIBLE

Ask the LLM to classify the query — handles edge cases

The Prompt

Classify this query into one of:
- VECTOR_DB (policy/document questions)
- SQL (data/numbers/counting questions)
- API (real-time/live data)

Query: "{query}"
Return: route name + confidence (0-1)

When to Use Which?

	Embedding Router	LLM Router
Speed	~1ms	~500ms
Cost	Free (no LLM)	1 LLM call
Edge cases	Struggles	Handles with reasoning
Ambiguous queries	Picks closest	Can output confidence + reason
Best for	Clear-cut categories	Complex/overlapping categories

Start with embedding router (fast, free). Switch to LLM router when you see misrouting on ambiguous queries.

Router Architecture

The full picture: classify → route → retrieve → answer

Route	Pipeline	Example Queries
📚 Vector DB	Embed → Search → Re-rank → LLM	“What does the policy say about...”
🗃️ SQL	Text-to-SQL → Execute → LLM	“How many / total / average...”
🌐 API	API Call → Parse JSON → LLM	“Current status of / right now...”

Fallback Strategies IMPORTANT

Multi-route

Confidence low? Try multiple routes in parallel, LLM picks best answer

Threshold

Confidence < 0.6? Ask user to clarify: “Policy or data?”

Default route

Can’t decide? Fall back to Vector DB — safest default

Don’t let the router silently pick the wrong route. A wrong route is worse than no routing at all.

Routing Walk-Through

Same topic, different queries — different routes

Topic: Employee Leave

User Query	Route	Why
“What is our vacation policy?”	Vector DB	Document question — search policy text
“How many sick days were taken in Q3?”	SQL	Counting/aggregation — needs database query
“Who is on leave right now?”	API	Real-time data — needs live HR system

✗ Misrouting Example

“What is the average sick leave per department?”

Embedding router says: Vector DB
(matched “sick leave” → policy docs)

Correct route: SQL
(needs aggregation: AVG + GROUP BY)

This is when you switch from embedding router to LLM router — the LLM understands “average per department” means data, not documents.

Routing + Decomposition

Some queries need BOTH — split AND route to different sources

The Query

“Compare our vacation policy with how many days employees actually took last year.”

Sub-query 1 → Vector DB

“What is the company vacation policy?”

Search policy documents → “20 days per year”

Sub-query 2 → SQL

“Average vacation days taken per employee last year”

SELECT AVG(days_taken) → “14.2 days”

LLM Synthesizes

“The policy allows 20 days. On average, employees used 14.2 days (71%). Engineering used the least at 11.3 days.”

This is the full picture: Decompose → Route each sub-query → Retrieve from different sources → Synthesize. Neither Section 3 nor Section 4 alone could answer this.

Lecture 5 — Key Takeaways

1. Most RAG failures are retrieval failures — diagnose the failure mode first

2. Re-ranking with cross-encoders is the highest-ROI improvement (+15–30% precision)

3. HyDE bridges vocabulary gaps by generating a hypothetical answer to search with

4. Parent-child chunking = search on small, read on big. Best of both granularities

5. Query decomposition breaks multi-hop questions into independent sub-queries

6. Semantic routing sends queries to the right data source (vector DB, SQL, API)

7. Stack techniques: Hybrid search → Re-rank → Parent expansion → Route

❓

Questions?

Thank you!

Kamyar Zeinalipour — Zucchetti LLM Course

Next → Lecture 6: Graph RAG & Knowledge Graphs

Theory: Entity extraction · Graph traversal · Community detection

🔬 Mega-Lab: RAG Pipeline (L4) + Advanced Retrieval (L5) + Graph RAG (L6)

Advanced RAG& Intelligent Routing

Today’s Agenda

Section 1: Why Naive RAG Fails (15 min)

Section 2: Advanced Retrieval Strategies (40 min)

Section 3: Query Decomposition (25 min)

Section 4: Semantic Routing (20 min)

Why Naive RAG Fails

The Naive RAG Pipeline

Six Ways Naive RAG Fails

1. Semantic Mismatch

2. Chunk Boundary Split

3. Context Dilution

4. Multi-Hop Failure

5. Recency Blindness

6. Wrong Granularity

The Fix Map

Advanced Retrieval

What Is Re-ranking?

The Problem

The Fix: Re-ranking

Bi-Encoder vs Cross-Encoder

Bi-Encoder STAGE 1

Cross-Encoder STAGE 2

Two-Stage Retrieval

Stage 1: Recall RECALL

Stage 2: Precision PRECISION

Embedding Model ≠ Re-ranker

Stage 1: Embedding Model

Stage 2: Re-ranker Model

Re-ranking in Practice

Cross-Encoder Models for Stage 2

Don’t Confuse Them

HyDE — The Problem It Solves

Semantic Mismatch

“working from home”

The Insight

HyDE — How It Works

Step 1 — LLM Call #1: Generate Fake Answer EXTRA

Step 2 — Search with the Fake Answer

Step 3 — LLM Call #2: Generate Real Answer STANDARD RAG

HyDE — Walk-Through

1. User Query

2. LLM generates hypothetical answer (NOT from your docs — made up)

3. We embed this fake answer and search — it matches real docs about:

4. Real docs + original query → LLM → correct final answer

Query Rewriting — What & Why

The Pipeline

Examples

Query Rewriting vs HyDE

Query Rewriting

HyDE

When to Use Which?

Parent-Child Chunking — The Problem

Small Chunks (100–200 tokens)

Large Chunks (500–1000 tokens)

Parent-Child Chunking — The Solution

How It Works

At Search Time

Why It Works

Parent-Child Chunking — Example

What Happened

Hybrid Search — The Problem We Left

The Normalization Headache

What If We Didn’t Need Scores At All?

Reciprocal Rank Fusion (RRF)

Example — Merging BM25 and Dense results

When to Use Score Fusion (α)

When to Use RRF

Hierarchical Indexing — The Problem

Flat Search: 500,000 chunks

What Flat Search Returns

Hierarchical Indexing — The Solution

Three Levels of Search

The Library Analogy

When to Use

Hierarchical Indexing — Example

Level 1: Which document? (10,000 → 5)

Level 2: Which section? (5 docs × ~10 sections → 10)

Level 3: Which paragraph? (10 sections × ~5 paragraphs → 3)

Flat Search

Advanced RAG
& Intelligent Routing