zucchetti-llm-course

Lab 10: Production Deployment (Capstone)

Lecture: 10, Modern LLM Architectures and Multimodal AI Duration: 45 minutes Difficulty: Advanced

Objectives

Wrap the RAG pipeline into a production-ready FastAPI application.
Implement /health and /query endpoints.
Add basic observability (latency tracking, success/failure rates).
Run an asynchronous load test to measure throughput (Tokens/sec & Requests/sec).
Understand how to containerize an AI application with Docker.

Prerequisites

Requirement	How to Check
Python 3.10+	`python --version`
Dependencies	`pip install fastapi uvicorn openai chromadb aiohttp rich`
LLM access	Ollama or DeepSeek API key

Quick Start

1. Start the Server

Open a terminal and run the FastAPI server:

cd lab_10_production
python lab_10_server.py

You can view the interactive API documentation at http://localhost:8000/docs

2. Run the Load Test

Leave the server running. Open a second terminal and run the load tester:

cd lab_10_production
# Usage: python lab_10_loadtest.py <concurrency> <total_requests>
python lab_10_loadtest.py 5 20

Files

File	Description
`lab_10_server.py`	FastAPI application serving the RAG API
`lab_10_loadtest.py`	Asynchronous load testing script
`lab_10_production.ipynb`	Jupyter notebook walkthrough
`solutions.py`	Async implementation solution
`Dockerfile`	Docker configuration for containerization
`data/test_queries.json`	30 sample queries for the load tester

Tasks

Task 1: Test the API via Swagger UI

Start the server and go to http://localhost:8000/docs in your browser. Try executing a /query request through the UI:

{
  "query": "What is the remote work policy?",
  "max_results": 3
}

Task 2: Observe the Load Test

Run the load test with concurrency=1 and total_requests=5. Note the average latency. Now run it with concurrency=10 and total_requests=20. What happens to the latency?

Task 3: The Async Challenge

If you look at lab_10_server.py, the LLM API call is synchronous. This means if 10 requests come in at once, they queue up, blocking the server. Look at solutions.py to see how to implement AsyncOpenAI so the server can handle multiple requests concurrently.

Task 4: Docker Containerization (Optional)

Look at the Dockerfile. It shows how you would package this entire application (including the pre-computed ChromaDB vector store) into a single image that can be deployed to AWS, GCP, or Azure.

Expected Output (Load Test)

Starting load test: 20 requests with concurrency 5
Completed: 20/20

Load Test Results
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric             ┃    Value ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time         │   12.45s │
│ Concurrency        │        5 │
│ Successful Requests│       20 │
│ Failed Requests    │        0 │
│ Avg Latency        │    2.85s │
│ Max Latency        │    4.10s │
│ Throughput (Req/s) │     1.60 │
└────────────────────┴──────────┘

🆘 Troubleshooting

Issue	Solution
Server fails to start	Check if port 8000 is already in use (`lsof -i :8000`).
“Collection not found”	Run Lab 04 to build the vector DB first, or wait for the server to auto-generate it.
High failure rate in load test	If using DeepSeek API, you might be hitting rate limits. Reduce concurrency.