Zucchetti LLM Course — Lecture 2

Tokenization, Embeddings
& Similarity

From text to tokens to vectors — and how to measure meaning


Kamyar Zeinalipour

Today’s Agenda

0:00 Section 1: Tokenization & BPE, WordPiece, SentencePiece
0:25 Section 2: Embeddings & Similarity
0:50 Section 3: Multilingual & Domain Alignment
1:05 β˜• Break (10 min)
1:15 πŸ”¬ Hands-on Lab: Embeddings Exploration
1:50 Key Takeaways & Q&A
Total: 2 hours · Theory first, then hands-on lab

Section 1 of 3

Tokenization &
Its Consequences

The invisible bottleneck most practitioners ignore

What is Tokenization?

LLMs don't see words or characters β€” they see tokens


Input text:

"Hello, how are you?"


Token IDs:

[9906, 11, 1268, 527, 499, 30]


BPE (Byte-Pair Encoding)

Used by GPT, LLaMA β€” starts from characters, merges frequent pairs

WordPiece / SentencePiece

Used by BERT, T5 β€” similar idea, slightly different merge strategy

BPE Algorithm β€” Step by Step

BPE starts from characters and merges the most frequent pair at each step.

Vocabulary: {l, o, w, e, r, s, t, n}

Step 0: [l, o, w, e, r]     (start from characters)

Step 1: [lo, w, e, r]    (merge most frequent pair: l+o)

Step 2: [low, e, r]     (merge lo+w)

Step 3: [low, er]      (merge e+r)

Step 4: [lower]        (merge low+er β†’ done!)

Vocabulary sizes

  • GPT-2: 50,257 tokens
  • GPT-4: ~100,000 tokens
  • LLaMA 3: 128,256 tokens

The tradeoff

  • Larger vocab β†’ fewer tokens per text β†’ faster
  • Larger vocab β†’ bigger embedding matrix β†’ more memory
  • Sweet spot: 32K–128K tokens

πŸ“„ Sennrich et al., "Neural Machine Translation of Rare Words with Subword Units", ACL 2016

WordPiece & SentencePiece

BPE is not the only tokenizer. Two other algorithms dominate the landscape.

WordPiece (Google, 2012)

Used by BERT, DistilBERT, Electra

  • Like BPE, but merges based on likelihood, not frequency
  • Picks the pair that maximizes the language model probability
  • Uses ## prefix for subword continuations
"playing" → ["play", "##ing"]

SentencePiece (Google, 2018)

Used by T5, LLaMA 2, Gemma, Mistral 7B

  • Treats input as raw text — no pre-tokenization needed
  • Language-agnostic: works on any script (Chinese, Arabic, Italian…)
  • Uses █ for spaces: █Hello
"New York" → ["▁New", "▁York"]
BPEWordPieceSentencePiece
Merge criterionMost frequent pairMax likelihoodBPE or Unigram (configurable)
Pre-tokenizationRequired (whitespace)Required (whitespace)Not required (raw text)
Used byGPT, LLaMA 3 (tiktoken)BERT, ElectraT5, LLaMA 2, Gemma
All three produce subword tokens. The key difference: SentencePiece is language-agnostic — it handles Italian, Chinese, and Arabic without special rules.

πŸ“„ Schuster & Nakajima, "Japanese and Korean Voice Search", ICASSP 2012  |  Kudo & Richardson, "SentencePiece", EMNLP 2018

Why Tokenization Matters

Problem 1: Non-English = More Tokens

TextLanguageTokens (GPT-4)
"heart failure"English2 tokens
"insufficienza cardiaca"Italian5 tokens
"心不全"Japanese4 tokens

This means for non-English languages:

  • Higher cost β€” more tokens = more API charges
  • Less context β€” fewer words fit in the context window
  • Lower quality β€” less non-English data in training

Why Tokenization Matters

Problem 2: Numbers are Unreliable

Tokenizing: "Revenue grew 15.7% this quarter"


β†’ ["Revenue", " grew", " 15", ".", "7", "%", " this", " quarter"]


The number 15.7 is split into 3 separate tokens. The model has no concept of "15.7" as a single number.


Never trust an LLM for arithmetic. Always use code for calculations and let the LLM decide what to calculate.

Try It Yourself!

See how different models tokenize the same text:


πŸ”—

platform.openai.com/tokenizer


Try pasting these and compare the token count:

"The quarterly revenue report shows a 15.7% increase"
"Il rapporto trimestrale mostra un aumento del 15,7%"

Every token costs money and takes up context window space. Understanding tokenization helps you optimize cost and design better prompts.

Section 2 of 3

Embeddings &
Similarity

Translating meaning into geometry

What is an Embedding?

Think of it like GPS coordinates for meaning

πŸ“ GPS

Rome β†’ [41.9, 12.5]

Naples β†’ [40.8, 14.3]

Close coords = close cities

🧠 Embeddings

"machine learning" β†’ [0.23, -0.45, ...]

"deep learning" β†’ [0.21, -0.42, ...]

Close vectors = similar meaning

Instead of 2 dimensions, embeddings use hundreds or thousands of dimensions (depending on the model) to capture nuances of meaning.

Similar meaning β†’ similar vectors:

"machine learning" β†’ [0.23, -0.45, 0.12, 0.89, ..., -0.31]

"deep learning"    β†’ [0.21, -0.42, 0.15, 0.87, ..., -0.28]

↑ nearly identical numbers! ↑


Unrelated meaning β†’ different vectors:

"coffee machine"   β†’ [-0.67, 0.33, 0.55, -0.12, ..., 0.71]

↑ completely different numbers ↑

Texts with similar meanings have vectors that point in similar directions. This is the foundation of semantic search.

Word2Vec β€” Where Embeddings Started

The 2013 paper that started the embedding revolution.

king βˆ’ man + woman β‰ˆ queen

Vector arithmetic captures semantic relationships!

Skip-gram Objective

max Σt Σj∈window log P(wt+j | wt)

Given a word, predict its neighbors. Words that appear in similar contexts get similar vectors.

Key insight

  • "You shall know a word by the company it keeps" β€” Firth, 1957
  • "dog" and "cat" appear near similar words ("pet", "food", "vet")
  • So their vectors end up close together
  • This idea powers ALL modern embeddings

πŸ“„ Mikolov et al., "Efficient Estimation of Word Representations in Vector Space", Google 2013  |  Pennington et al., "GloVe: Global Vectors", Stanford EMNLP 2014

Visualizing Embedding Clusters

High-dimensional vectors projected to 2D. Related terms naturally cluster together.

Distance Metrics — How to Measure Similarity

We have vectors. Now how do we compare them? There are 3 main options:

Cosine Similarity ⭐
sim(A,B) = (A·B) / (||A|| × ||B||)

Measures the angle between vectors. Ignores vector length — only cares about direction. Default for most embedding models.

Dot Product
sim(A,B) = A·B = Σi ai · bi

Sensitive to vector magnitude. Longer vectors score higher. Use when length = importance.

Euclidean Distance
dist(A,B) = ||A − B||&sub2; = √(Σi (ai − bi)²)

L2 distance in vector space. Smaller = more similar (opposite direction from cosine).

Most embedding models normalize vectors to unit length. When normalized: cosine = dot product. We use cosine similarity as our default — next slide: how to interpret the scores.

Cosine Similarity — What the Scores Mean

We chose cosine. But what does a score of 0.85 actually mean?

cos(θ) = (A · B) / (||A|| × ||B||)

0.90+

Near-duplicate

"car" ↔ "automobile"

0.70–0.90

Same topic

"machine learning" ↔ "deep learning"

0.40–0.70

Loosely related

"machine learning" ↔ "stock market"

< 0.40

Unrelated

"car" ↔ "pizza"

In RAG systems, you'll typically set a threshold of ~0.70 to filter relevant results. You'll practice this in Lecture 3.

Embedding Models Landscape

Model Dimensions Languages Best for
all-MiniLM-L6-v2 384 English Fast, lightweight, English-only tasks
multilingual-MiniLM 384 50+ languages Multilingual, runs locally βœ…
BGE-M3 1024 100+ languages State-of-the-art multilingual
text-embedding-3-large 3072 Multilingual OpenAI API (highest quality)
For this course, we use paraphrase-multilingual-MiniLM-L12-v2 β€” it runs locally, supports 50+ languages, and is fast on CPU.

Why Dimensions Matter

Lower dimensions (e.g. 384)

  • Faster to compute
  • Less storage needed
  • Good for most tasks
  • May lose fine-grained distinctions

Higher dimensions (e.g. 3072)

  • Captures subtle differences
  • Better for specialized domains
  • Slower and more expensive
  • Needs more training data to fill

Think of it like resolution: 384-dim = "720p", 3072-dim = "4K" β€” more detail, but more cost.

Section 3 of 3

Multilingual &
Domain Alignment

When your data doesn't speak English

Cross-Lingual Alignment

Multilingual models map different languages into a shared vector space


"heart failure" (English)

β‰ˆ same vector as β‰ˆ

"insufficienza cardiaca" (Italian)


βœ… Multilingual Models

  • One model handles many languages
  • Cross-lingual search works
  • Good for mixed-language data

⚠️ Translation Pipeline

  • Translate to English first
  • Use English-only model
  • Adds latency + translation errors

Domain-Specific Challenges

The Problem

General models are trained on Wikipedia, news, books β€” not on your domain's jargon.


General Model

"ERP" and "SAP" β†’ low similarity πŸ˜•

(doesn't know they're related enterprise systems)

Domain-Adapted Model

"ERP" and "SAP" β†’ high similarity βœ…

(trained on enterprise data, knows the relationship)

For enterprise applications, you'll often need to fine-tune embedding models on your domain data. We'll cover this in Lecture 9.

Practical Guidance

Which model should you use?


ScenarioRecommended ModelWhy
English-only, fast prototyping all-MiniLM-L6-v2 Fast, lightweight, good enough
Multilingual content multilingual-MiniLM 50+ languages, runs locally
Production, high accuracy BGE-M3 or OpenAI State-of-the-art quality
Specialized domain Fine-tuned model Trained on your data (Lecture 9)

Why This Matters for Your Work


  • Embeddings power search β€” find relevant documents by meaning, not just keywords. This is the core of Lectures 3–8.
  • Tokenization drives cost β€” understanding how your text gets tokenized helps you optimize API spend and design better prompts.
  • Model choice is a daily decision β€” multilingual vs. English-only, small vs. large, local vs. cloud. You'll make this call on every project.
  • "LLMs predict, they don't know" β€” this mental model prevents you from trusting LLMs with tasks they can't handle (math, fresh facts, guaranteed correctness).

Everything from Lecture 2 onward builds on these fundamentals.

Prompting, RAG, Agents, Fine-Tuning β€” they all depend on embeddings and understanding how LLMs work.

πŸ”¬ Hands-on Lab

Embeddings Exploration

Let’s see everything we learned — step by step in Colab

Step 1 Load 20 terms across 4 categories
Step 2 Tokenization: EN vs IT token cost
Step 3 3 architectures in action: encoder, decoder, encoder-decoder
Steps 4–5 Generate embeddings + cosine similarity heatmap
Step 6 2D cluster map — see the “meaning space”
Steps 7–8 Most similar pairs + semantic search
Step 9 Word2Vec analogy: king − man + woman ≈ queen

Open Google Colab → run all cells step by step

lab_01_embeddings.ipynb

Step 10 (Model Comparison) is a bonus — try it if time permits!

Lecture 2 — Key Takeaways


  • Transformers process all tokens in parallel using self-attention β€” this is what made modern AI possible
  • LLMs are next-token predictors β€” they don't "understand", they predict
  • Tokenization is the invisible bottleneck β€” non-English text costs more tokens
  • Embeddings map text to vectors β€” similar meanings = similar directions
  • Cosine similarity measures semantic closeness in vector space
  • Use multilingual models for non-English data, not English-only models

Next lecture → Lecture 3: Prompt Engineering

How to communicate effectively with LLMs

Questions?


Repository: github.com/KamyarZeinalipour/zucchetti-llm-course


Lecture 2 of 10 · Zucchetti LLM Course