Zucchetti LLM Course — Lecture 8

RAG Evaluation
& Observability

How do you know your system is actually working?

Kamyar Zeinalipour

Today’s Agenda

Hour 1 — Theory

S1 — The Evaluation Imperative

S2 — Retrieval Metrics

S3 — Generation Metrics & Hallucination

S4 — LLM-as-Judge

S5 — Observability & Tracing

☕ Break

🔬 Hour 2 — Hands-On Mega-Lab — Build, evaluate, and improve a full RAG pipeline

The Story So Far

Four levels of retrieval sophistication — you’ve built all of them (in theory)

L4 — Naive RAG — retrieve + generate, fixed pipeline

L5 — Advanced RAG — re-ranking, HyDE, routing, decomposition

L6 — Graph RAG — knowledge graphs, relationships, entity extraction

L7 — Agentic RAG — function calling, ReAct, self-correcting pipelines

You’ve built increasingly powerful systems. But how do you know they’re actually working? Today we learn to measure, trace, and prove it.

Section 1 — The Evaluation Imperative

The “It Looks Good” Trap

Manual Testing

You try 5–10 queries
Answers “look reasonable”
You ship to production
Users hit the 1,000 queries you didn’t test

Automated Evaluation

Run 200+ test queries
Compute metrics per query
Detect regressions automatically
Catch failures before users do

Manual testing gives you confidence. Automated evaluation gives you proof. You need both — but you can’t scale without automation.

Three Levels of Evaluation

Component

“Did we find the right chunks?”

Precision, Recall, MRR

Tests retrieval in isolation

End-to-End

“Is the answer correct and grounded?”

Faithfulness, Relevance

Tests retrieval + generation

System

“Are users satisfied?”

Thumbs up/down, task success

Tests the full user experience

Always start with component-level metrics. If retrieval is broken, fixing generation won’t help. Most RAG failures are retrieval failures.

Golden Datasets — Your Ground Truth

The foundation of every evaluation pipeline

What a Golden Example Contains

Field	Example	Purpose
`question`	“How many sick days am I entitled to?”	The user query
`ground_truth`	“Full-time employees receive 12 paid sick days per year.”	The correct answer
`expected_chunks`	[“sick_leave_policy.txt, chunk 3”]	Which documents should be retrieved

No golden dataset = no evaluation. This is not optional. You need human-written, verified answers to compare against.

Building a Golden Dataset — Practical

Stage	Size	Purpose
Development	15–50 examples	Quick iteration, catch obvious bugs
Staging	50–200 examples	Reliable metrics, regression testing
Production	200+ examples	Statistical significance, edge cases

Quality Checklist

Coverage: Questions span all document types and difficulty levels
Adversarial: Include queries designed to trigger failures (out-of-scope, ambiguous)
Human-validated: Answers reviewed by domain experts, not generated by LLM
Versioned: Track changes to the dataset alongside your pipeline changes

Section 2 — Retrieval Metrics

What Are We Measuring?

Two fundamental questions about your retrieval system

Precision

“Of what I found, how much is actually useful?”

Noise control — fewer irrelevant chunks

Recall

“Of everything useful, how much did I find?”

Coverage — nothing important missed

L4 Callback: Remember the “wrong chunks retrieved” failure mode? Precision@K is how we detect that.

Precision@K & Recall@K

$\text{Precision@K} = \frac{|\text{relevant} \cap \text{retrieved}_K|}{K}$

What fraction of retrieved chunks are relevant?

$\text{Recall@K} = \frac{|\text{relevant} \cap \text{retrieved}_K|}{|\text{relevant}|}$

What fraction of all relevant chunks did we find?

Example: K=5, 2 relevant chunks in corpus

Retrieved: [✅, ❌, ✅, ❌, ❌] → Precision@5 = 2/5 = 0.40 | Recall@5 = 2/2 = 1.00

Good recall (found everything), mediocre precision (60% noise).

Trade-off: Increase K → recall goes up, precision goes down. The sweet spot depends on your use case.

MRR — Mean Reciprocal Rank

“How high in the list is the first relevant result?”

$\text{MRR} = \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{\text{rank}_i}$

Example: 3 queries

Query	Retrieved (✅ = relevant)	First relevant at	Reciprocal Rank
Q1	[✅, ❌, ❌]	Position 1	1/1 = 1.00
Q2	[❌, ❌, ✅]	Position 3	1/3 = 0.33
Q3	[❌, ✅, ❌]	Position 2	1/2 = 0.50

MRR = (1.00 + 0.33 + 0.50) / 3 = 0.61

nDCG — Rewarding Top Results

Not all positions are equal — top-1 matters more than top-5

$\text{DCG@K} = \sum_{i=1}^{K} \frac{\text{rel}_i}{\log_2(i+1)}$

Intuition

A relevant chunk at position 1 gets a weight of 1/log₂(2) = 1.00

A relevant chunk at position 5 gets a weight of 1/log₂(6) = 0.39

Position matters. The best results should be at the top.

nDCG = DCG / ideal DCG. Normalized to [0, 1]. An nDCG of 1.0 means perfect ranking. Use nDCG when result order matters.

Hit Rate & Practical Guidance

Hit Rate — The Simplest Metric

$\text{Hit Rate@K} = \frac{|\text{queries with at least 1 relevant in top-K}|}{|\text{queries}|}$

Binary: did we find anything useful? Yes or no.

Metric	Best For	Target
Hit Rate@5	Quick health check	≥ 0.90
Precision@3	Context quality (less noise in prompt)	≥ 0.60
MRR	Top-1 result quality	≥ 0.70
nDCG@5	Full ranking quality	≥ 0.65

Start with Hit Rate and Precision@3. They’re easy to compute and catch most problems.

Diagnosing Retrieval Failures

When your metrics are bad, this flowchart tells you where to look

Symptom	Likely Cause	Fix (Lecture)
Low precision, OK recall	Too many irrelevant chunks	Re-ranking (L5)
OK precision, low recall	Missing relevant chunks	Better embeddings or HyDE (L5)
Both low	Embedding model too weak	Domain-specific model or hybrid search (L4)
Zero hits	Chunking split the answer	Parent-child chunking (L5)
High metrics, bad answers	Generation problem, not retrieval	Better RAG prompt (L4) or LLM-as-Judge (this lecture)

Always diagnose retrieval first. If your chunks are wrong, no amount of prompt engineering will save the answer.

Section 3 — Generation Metrics & Hallucination

Four Dimensions of Answer Quality

Faithfulness

Does the answer only use information from the retrieved context? No hallucination.

Answer Relevance

Does the answer actually address the user’s question? Not off-topic.

Completeness

Does the answer cover all aspects of the question? Nothing missing.

Factual Consistency

Are the stated facts correct? No distortion of source material.

Faithfulness is the most critical metric for production RAG. An unfaithful answer actively harms the user. An incomplete answer at least doesn’t lie.

Faithfulness — Is It Grounded?

“Does the answer ONLY use information from the retrieved context?”

Faithful PASS

Context: “Employees receive 12 sick days per year.”

Answer: “You get 12 sick days per year.”

Unfaithful FAIL

Context: “Employees receive 12 sick days per year.”

Answer: “You get 12 sick days, plus 5 personal days.”

L7 Callback: Self-RAG’s [IsSup] token asks exactly this question at runtime: “Is my answer supported by evidence?” Now we measure it offline with a judge.

Answer Relevance — Did It Answer?

“Does the answer address the actual question asked?”

Relevant PASS

Q: “How many sick days do I get?”

A: “Full-time employees receive 12 paid sick days per year.”

Irrelevant FAIL

Q: “How many sick days do I get?”

A: “Our leave policy is comprehensive and designed to support employee well-being…”

Right topic, wrong answer. The model retrieved the right document but gave a generic summary instead of the specific number.

Completeness & Factual Consistency

Completeness

Q: “What are the remote work rules?”

A: “You can work 2 days remote.”

Missing: eligibility criteria, approval process, equipment policy.

Partially correct, but dangerously incomplete.

Factual Consistency

Context: “Budget limit is €500.”

A: “The budget limit is €5,000.”

Off by 10×. The model distorted the source fact.

Completeness and factual consistency are harder to automate than faithfulness. They often require domain experts in the loop.

The RAGAS Framework EACL 2024

Automated evaluation for RAG — the standard framework

Metric	What It Measures	Needs
Faithfulness	Answer grounded in context?	Question + Context + Answer
Answer Relevancy	Answer addresses the question?	Question + Answer
Context Precision	Retrieved chunks actually relevant?	Question + Context + Ground Truth
Context Recall	All needed info retrieved?	Context + Ground Truth

Key insight: Context Precision & Recall measure retrieval quality. Faithfulness & Answer Relevancy measure generation quality. Together, they cover the full pipeline.

Es et al. (2024) — “RAGAs: Automated Evaluation of Retrieval Augmented Generation” — EACL 2024 (System Demonstrations)

Section 4 — LLM-as-Judge

From Runtime Grading to Offline Evaluation

Same technique, different purpose

L7: Runtime Grading

The retrieval grader runs inside the pipeline

Decides: “Is this chunk relevant?”

Affects the current query’s answer

Real-time, per-query, affects output

L8: Offline Evaluation

The judge runs after the pipeline

Scores: “Was this answer faithful?”

Produces a report, not an answer

Batch, systematic, measures quality

You already know this pattern from L7’s retrieval grader. Now we formalize it for systematic, offline quality measurement.

LLM-as-Judge — Prompt Design

System: You are a strict evaluation judge.

Given:
- QUESTION: {question}
- CONTEXT: {retrieved_context}
- ANSWER: {generated_answer}

Task: Is the ANSWER faithful to the CONTEXT?

Rules:
1. Every claim in the ANSWER must be supported by the CONTEXT
2. If ANY claim is not in the CONTEXT, score = 0
3. If all claims are supported, score = 1

Respond in JSON: {"score": 0 or 1, "reason": "..."}

L3 Callback: This is structured output — the JSON format you learned in Lecture 3. The judge uses the same output formatting technique to produce parseable results.

Worked Example: Catching a Hallucination

Context: “Full-time employees receive 12 paid sick days per year. Unused days do not carry over.”

Question: “How many sick days do I get? Do they carry over?”

Answer: “You receive 12 sick days per year. Unused days do not carry over. However, you can donate unused days to colleagues in need.”

Judge:

{"score": 0, "reason": "The claim about donating unused days to colleagues is NOT in the context. This is a hallucination."}

The first two claims are faithful. The third is hallucinated. A human reviewer might miss this — the judge catches it every time.

Biases & Limitations of LLM Judges

Bias	What Happens	Mitigation
Position bias	Prefers the first option in comparisons	Randomize order, run twice
Verbosity bias	Rates longer answers higher	Instruct: “Length is not quality”
Self-enhancement	GPT-4 rates GPT-4 answers higher	Use a different model family as judge
Leniency	Gives high scores too easily	Use strict rubrics with binary scores

The judge is also an LLM — it has its own failure modes. Always validate your judge against human annotators on a subset.

Zheng et al. (2023) — “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” — NeurIPS 2023 (Datasets & Benchmarks Track)

Section 5 — Observability & Tracing

What to Monitor in Production

L7 previewed the four pillars — now we go deep

Latency

Retrieval time, generation time, total. P50, P95, P99.

Cost

Tokens per query, cost per query, daily/monthly budget.

Quality Drift

Average faithfulness over time. nDCG trends. Regression alerts.

Error Rates

Tool failures, timeouts, empty retrievals, refusals.

L7 Callback: L7’s production considerations (slide 52) previewed these pillars — Observability, Cost, Latency, Error Handling. Now we formalize and instrument them for systematic measurement.

The Trace — From Query to Answer

Every query produces a complete trace you can inspect and debug

Query
+ timestamp

→

Retrieval
chunks, scores, ms

→

Generation
tokens, model, ms

→

Answer
+ quality score

What Each Node Logs

Node	Key Fields
Retrieval	chunk_ids, similarity_scores, latency_ms, k_requested, k_returned
Generation	model_name, prompt_tokens, completion_tokens, latency_ms, temperature
Answer	faithfulness_score, relevance_score, sources_cited

Attribution & Citation

Can every claim be traced to a source chunk?

Good Attribution

“Full-time employees receive 12 sick days per year [Source: sick_leave_policy.txt, section 2.1].”

Why It Matters

Trust: Users can verify the answer against the source
Debugging: When the answer is wrong, you know which chunk caused it
Compliance: Enterprise and clinical audits require traceability
Confidence: “I’m 92% confident based on 3 supporting chunks”

Min et al. (2023) — “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation” — EMNLP 2023

Tools: LangSmith, Phoenix, Langfuse

Tool	Type	Best For	Cost
LangSmith	SaaS	LangChain ecosystem, full tracing	Free tier + paid
Phoenix (Arize)	Open-source	Embedding drift, retrieval analysis	Free
Langfuse	Open-source	Self-hostable, full tracing, cost tracking	Free (self-host)
Custom logging	DIY	Maximum control, no vendor lock-in	Dev time

For Zucchetti: Start with custom structured logging (JSON logs per query). Graduate to Langfuse when you need a dashboard. Avoid vendor lock-in early.

Cost Monitoring & Alerting

What to Track

Metric	Alert When	Action
Avg tokens/query	> 2× baseline	Check if chunks got larger or prompt grew
Daily cost	> budget × 1.5	Check for query spike or infinite loops
nDCG@5 (7-day avg)	Drops > 10%	Data changed? Embedding model updated?
Faithfulness (7-day avg)	Drops > 5%	Prompt regression? Model version change?

Set alerts on both cost AND quality. A cost spike might mean your system is working harder. A quality drop means it’s working worse. You need to know both.

Key Takeaways

1. Golden datasets are non-negotiable. No ground truth = no evaluation.

2. Retrieval metrics (Precision, Recall, MRR) diagnose whether you’re finding the right chunks.

3. Faithfulness is the most critical generation metric — an unfaithful answer actively harms users.

4. LLM-as-Judge scales evaluation from 10 queries to 10,000 — same technique as L7’s retrieval grader, different purpose.

5. Observability = tracing + cost monitoring + quality alerting. Treat your RAG like any production service.

6. Always diagnose retrieval first. Most RAG failures are retrieval failures, not generation failures.

☕ Break — 10 minutes

When we return: Hands-On Mega-Lab — Build, evaluate, and improve a full RAG pipeline

Lab Preview — What You’ll Build

Everything from L4–L8 in one hands-on session

Build
L4

→

Evaluate
L8

→

Improve
L5–L7

→

Compare
L8

Build: Load, chunk, embed, store in ChromaDB, generate answers (L4)
Evaluate: Retrieval metrics + LLM-as-Judge on a golden dataset (L8)
Re-ranking: Cross-encoder two-stage retrieval (L5)
HyDE: Hypothetical Document Embeddings (L5)
Graph RAG: Entity graph + vector hybrid retrieval (L6)
CRAG: Self-correcting retrieval with chunk grading (L7)
Compare: Side-by-side metrics for all 5 strategies (L8)

What’s Next

Next → Lecture 9: Efficient Fine-Tuning (LoRA / QLoRA)

When prompting and RAG aren’t enough — teach the model new behavior.

The math of weight updates, low-rank adaptation, and domain-specific training.

Questions?

Let’s start the lab!

Kamyar Zeinalipour — Zucchetti LLM Course

RAG Evaluation& Observability