zucchetti-llm-course

Lab 03: Inference Benchmarking and Quantization

Lecture: 4, RAG Foundations and the Decision Matrix Duration: 35 minutes Difficulty: Beginner

Objectives

By the end of this lab you will:

• Benchmark LLM inference speed (tokens/sec, time-to-first-token)
• Understand how quantization affects quality vs. performance
• Compare local (Ollama) vs. cloud (DeepSeek) inference
• Build a decision framework for choosing the right model

Prerequisites

Requirement	How to Check
Python 3.10+	python --version
Dependencies	pip install openai rich python-dotenv tiktoken
Ollama + model	ollama list (should show at least one model)

Quick Start

# Pull different quantizations to compare (optional)
ollama pull deepseek-r1:8b          # Default
ollama pull llama3.2:3b             # Smaller model
ollama pull qwen2.5:7b              # Alternative

cd lab_03_benchmarking
python lab_03_benchmarking.py

Files

File	Description
lab_03_benchmarking.py	Main benchmark script with TODO section
lab_03_benchmarking.ipynb	Jupyter notebook version
solutions.py	Complete solutions
data/test_prompts.json	20 test prompts across 5 categories

Tasks

Task 1: Discover Models

The script auto-discovers all Ollama models and DeepSeek API. Observe the list.

Task 2: Watch the Benchmarks

The script sends 5 test prompts to each model and measures:

• Tokens/second — raw generation speed
• Time-to-first-token (TTFT) — how fast the model starts responding
• Total time — end-to-end latency

Task 3: Implement Quality Scoring

Find score_quality() and implement keyword-based scoring.

Task 4: Interpret the Decision Matrix

Use the results to answer: which model would you pick for each scenario?

• Low-latency chatbot
• Private medical Q&A system
• High-volume batch processing

Expected Output

A comparison table showing speed, quality, and cost across all models.

🆘 Troubleshooting

Issue	Solution
No models found	Pull models with ollama pull <model>
Streaming error	Some models don’t support streaming — script handles this
DeepSeek rate limit	Reduce n_prompts or add a delay between calls