Lecture: 8, RAG Evaluation and Observability
Duration: 60 minutes
Format: Step-by-Step Live Demonstration Script & Interactive Notebook
This lab is structured specifically for live step-by-step instruction. Instead of running a monolithic script where code executes silently in the background, this lab is divided into 7 inspection blocks.
For each block, you can pause, show students intermediate outputs (raw text, cross-encoder rank shifts, LLM imagination in HyDE, entity nodes, CRAG rewrite triggers, and JSON traces), and explain the lecture concepts live.
| Step | Topic & Technique | Lecture | What You Show Live on Screen |
|---|---|---|---|
| Part 1 | Naive RAG & Ingestion | L4 | • Raw documents vs overlap chunking • ChromaDB L2 vector distance scores • RAG system prompt & source citations |
| Part 2 | Golden Data & Metrics | L8 | • Golden ground-truth Q&A entry • Step-by-step math for Precision@3, Recall@3, MRR • Raw JSON output returned by LLM-as-a-Judge |
| Part 3 | Re-ranking | L5 | • Top-9 Bi-Encoder candidates • Cross-Encoder score table ( ms-marco-MiniLM) • Rank shift table (which chunks moved UP/DOWN) |
| Part 4 | HyDE | L5 | • The “hypothetical answer” the LLM imagined • Search distance comparison (Query search vs HyDE search) • How HyDE bridges vocabulary mismatch |
| Part 5 | Knowledge Graph RAG | L6 | • Extracted entity nodes & currency/day values • Entity co-occurrence adjacency index • Hybrid score merging (Graph + Vector) |
| Part 6 | CRAG Self-Correction | L7 | • LLM chunk relevance grading (RELEVANT/IRRELEVANT)• Self-correction trigger on low quality • Query rewriter output (original vs rewritten query) |
| Part 7 | Tracing & Leaderboard | L8 | • Structured trace JSON log file • Strategy Leaderboard comparing all 5 methods • Production quality gate pass/fail report |
Open the notebook in Jupyter or Colab and run one cell at a time as you talk through each slide section:
Run the python script. It will pause between each part (Press Enter to continue...), allowing you to explain the outputs:
cd lab_07_evaluation
python3 lab_07_evaluation.py
| File | Description |
|---|---|
lab_07_evaluation.py |
Main step-by-step demonstration script |
lab_07_evaluation.ipynb |
Interactive 19-cell Jupyter notebook |
generate_lab07_notebook.py |
Generator script for the notebook |
data/policies/ |
5 HR policy documents (knowledge base) |
data/golden_dataset.json |
Human-verified ground truth Q&A pairs |