zucchetti-llm-course

Lab 09: Vision-Language Models

Lecture: 10, Modern LLM Architectures and Multimodal AI Duration: 30 minutes Difficulty: Intermediate

Objectives

Encode local images (prescriptions, lab reports) as base64 strings.
Pass images to a Vision-Language Model (VLM) via the API.
Prompt the VLM to act as an extractor and return structured JSON.
Understand the benefits of VLMs over traditional OCR pipelines.

Prerequisites

Requirement	How to Check
Python 3.10+	`python --version`
Dependencies	`pip install openai rich python-dotenv`
Local VLM	`ollama pull llava:7b` (recommended for privacy)
Cloud VLM	GPT-4o or DeepSeek (if vision supported) in `.env`

Quick Start

cd lab_09_vlm
python lab_09_vlm.py

Files

File	Description
`lab_09_vlm.py`	Main script with TODO section
`lab_09_vlm.ipynb`	Jupyter notebook version
`solutions.py`	Vision API formatting solution
`data/documents/`	Sample images (prescription, lab report)

Tasks

Task 1: View the Images

Open the images in data/documents/ using your computer’s image viewer. Notice the layout, the handwriting, and the tabular data. Traditional OCR struggles with this context.

Task 2: Implement the Vision API Call

Find the analyze_image() function. You need to construct the messages array in the format expected by vision models. Unlike text-only models where content is a string, vision models expect content to be a list of dictionaries, containing both text and the base64 image URL.

Hint:

"content": [
    {"type": "text", "text": prompt},
    {"type": "image_url", "image_url": {"url": f"data:{mime_type};base64,{base64_image}"}}
]

Task 3: Test Extraction

Run the script. The script prompts the VLM to extract the data and format it exclusively as JSON. Watch how it handles both the prescription (handwriting) and the lab report (tabular data).

Expected Output

Processing: medical_prescription.png
{
  "patient_name": "Mario Rossi",
  "date": "2024-03-25",
  "medication": "Amoxicillin 500mg",
  "dosage": "1 tablet",
  "instructions": "every 8 hours for 7 days. Take with food."
}

🆘 Troubleshooting

Issue	Solution
`API call not implemented`	You must complete the TODO in Task 2.
Model returns markdown block	Sometimes the model wraps JSON in ` ```json `. The script tries to clean this up, but if parsing fails, it will print the raw output.
Ollama timeouts	Vision models can be slow to initialize on CPU. Give it time, or switch to a cloud API.