Linear Algebra for Machine Learning — Part 2: The Dot Product and the Idea of Similarity
Part 2 of a series that builds the math behind AI, one small step at a time.
Quick recap: In Part 1, we learned a vector is an arrow — it has a length (how much) and a direction (which way). We also learned a unit vector is that same arrow, shrunk to length 1, keeping only the direction.
Why Do We Need This?
Say you type “sneakers for running” into a search bar. The search engine has millions of stored items. How does it decide which ones are close to what you typed?
Both your search and every stored item get turned into vectors (we saw why in Part 1). But having two arrows isn’t useful by itself. You need one more thing: a way to check if two arrows agree with each other.
That check is called the dot product. It’s the single tool behind search engines, face unlock, recommendation feeds, and even how AI reads sentences. Without it, a computer could store vectors all day and never compare them to anything.
Build the Picture First
Picture two arrows starting at the same point.
If they point almost the same way, they “agree” a lot.
If they point straight across from each other — a perfect right angle — they don’t agree or disagree. They’re just unrelated.
If they point in opposite directions, they disagree completely.
The dot product is just a number that captures this agreement:
- Big positive number → arrows point almost the same way.
- Zero → arrows are unrelated (perpendicular).
- Negative number → arrows point opposite ways.
The Shadow Trick
Here’s the easiest way to see a dot product, no formula needed yet.
Take one arrow, call it a, and lay it flat on the ground. Now shine a light straight down onto the second arrow, b, so it casts a shadow onto a.
That shadow length — how far b reaches along a — is basically what the dot product measures (then scaled by the length of a). If b points almost the same way as a, its shadow is long. If b points sideways to a, its shadow shrinks to nothing. If b points backward, the “shadow” flips to the other side, which is where the negative number comes from.
Writing It With Numbers
For two vectors like a = (4, 3) and b = (2, 1), the dot product is worked out like this:
dot product = (4 × 2) + (3 × 1) = 8 + 3 = 11
Simple rule: multiply matching numbers, then add everything up.
There’s also a second way to write the same thing, using length and angle:
dot product = (length of a) × (length of b) × cos(angle between them)
Both formulas give the exact same answer. The first is easier to compute. The second is easier to understand — it’s telling you the dot product depends on three things: how long each arrow is, and how far apart their angle is.
From Dot Product to Cosine Similarity
There’s one problem with the plain dot product: it’s affected by length, not just direction.
A short product review and a long product review can mean almost the same thing, but if the long one has a “bigger” vector, its dot product score gets inflated just because it’s longer — not because it means more.
The fix from Part 1 comes back here: shrink both vectors to length 1 first (unit vectors), then take the dot product. This removes length completely and leaves only direction. This new version has a name: cosine similarity.
cosine similarity = dot product ÷ (length of a × length of b)
The result always lands between -1 and 1:
- 1 → pointing exactly the same way (perfect match)
- 0 → unrelated
- -1 → pointing exactly opposite
This one number is what search engines, recommendation systems, and AI chatbots use, over and over, billions of times a day, to decide “how close is this to that.”
Face Embeddings: Seeing It in Action
A face recognition system doesn’t compare pixels directly — pixels shift too easily with lighting and angle. Instead, it turns each face into a vector (a “face embedding”) — a long list of numbers describing learned traits about that face.
Photos of the same person end up as vectors that are close together, pointing almost the same way. Photos of different people end up pointing in very different directions.
Unlocking your phone with your face is really just this: turn the new photo into a vector, check its cosine similarity against your stored vector, and if the number is high enough, unlock.
Quick Checks (Ask Yourself)
Why does the dot product measure similarity? Because it grows when two arrows point the same way, and shrinks (even going negative) as they point apart.
Get Sujal Neupane’s stories in your inbox
Join Medium for free to get updates from this writer.
Why isn’t a perpendicular angle the smallest possible value? It feels like “very different,” but perpendicular actually means unrelated, not opposite. The true smallest value happens when the arrows point in completely opposite directions.
Why do we ignore length in cosine similarity? Because length usually reflects something unrelated to meaning — like review length, or how confident a model felt — not how similar two things actually are.
Can a dot product be negative? Yes. That happens when the angle between two arrows is more than 90°, meaning they lean toward opposite directions.
If two vectors are identical, what’s their cosine similarity? Exactly 1 — a perfect match, since they point in the exact same direction.
What if two vectors are exact opposites, like (2,3) and (-2,-3)? Cosine similarity is exactly -1 — total disagreement.
Two photos of the same cat, one bright and one dark. Would their embeddings be close? Usually yes — a well-trained model learns to focus on the shape and features of the cat, not the brightness, so both vectors should point in a similar direction.
Mistakes People Often Make
Mistake: thinking 0 is the lowest possible score. Zero just means “unrelated.” Negative numbers go lower still, meaning “opposite.” This trips people up because in everyday life, zero often feels like the bottom.
Mistake: thinking a bigger dot product always means more similar. Not true if the vectors have different lengths — a long vector can produce a big dot product just by being long, not because it’s more similar. This is exactly why cosine similarity exists: to remove that trap.
Mistake: thinking perpendicular means “opposite.” Perpendicular means “no relationship,” not “opposite.” Full opposite only happens at 180°.
Mistake: assuming cosine similarity and dot product are the same thing. Dot product cares about length and direction. Cosine similarity only cares about direction. They agree only when both vectors already have length 1.
Questions Worth Sitting With
- If two things are described by long lists of numbers, why is comparing their direction often more useful than comparing them number-by-number?
- What real information gets thrown away when we switch from dot product to cosine similarity?
- Could two very different images end up with a high cosine similarity by accident? What would that mean about the model?
- Why might a search engine prefer cosine similarity over raw distance between two vectors?
Interview Questions
Basic Q: What does the dot product measure? A: How much two vectors point in the same direction — how much they “agree.”
Basic Q: What’s the range of cosine similarity? A: From -1 (completely opposite) to 1 (exactly the same direction), with 0 meaning unrelated.
Intermediate Q: Why do search and recommendation systems use cosine similarity instead of plain dot product? A: Because plain dot product is affected by vector length, which often has nothing to do with real similarity. Cosine similarity removes that bias. Wrong answer people give: “Cosine similarity is just faster to compute.” (Not true — it actually takes more steps than a plain dot product.)
Intermediate Q: If two vectors have a dot product of 0, are they unrelated or opposite? A: Unrelated. A dot product of 0 means the vectors are perpendicular, not opposite.
Advanced Q: Two face embeddings have high cosine similarity but the photos are clearly different people. What could be going wrong? A: The model may not have learned good features — maybe it’s relying on something shared between the photos (like lighting or background) instead of true facial structure. This points to a training or data problem, not a math problem.
Research-level Q: In large embedding models, why do some designs force every vector to sit exactly on a unit circle (length 1) during training, instead of letting cosine similarity handle normalization afterward? A: Forcing unit length during training keeps the model from learning a “shortcut” — like just growing vector length to boost raw dot product scores — so the model is forced to improve the actual direction, which is what carries the meaning.
How This Connects to Everything Else
- Geometry — the dot product is just the shadow of one arrow on another.
- Search engines — a search is turned into a vector and compared to millions of stored vectors using cosine similarity.
- Recommendation systems — your taste vector is compared against every product’s vector.
- Face recognition — matching a live photo to a stored photo is a cosine similarity check.
- Transformers and LLMs — “attention,” the core trick behind ChatGPT-style models, is built directly on dot products between word vectors. Every word checks its dot product against every other word to decide what to pay attention to.
- Neural networks — every single layer in a neural network computes a dot product between its input and its learned weights, before anything else happens.
A Little History
The dot product comes from the same 1800s physics work that gave us vectors — mainly through Josiah Willard Gibbs and Oliver Heaviside, who simplified Hamilton’s early vector ideas into the clean, practical form we use today. They needed a way to compute work in physics — how much a force actually contributes to moving an object in a certain direction. That’s the exact same shadow idea shown above. Decades later, this became the backbone of how AI compares anything to anything else.
Where You’ll See This in Real Life
- Face unlock — comparing your face vector to the one stored on your phone.
- Search engines — matching your search vector to page vectors.
- Spotify / Netflix recommendations — matching your taste vector to content vectors.
- Spam filters — comparing an email’s vector to known spam vectors.
- Self-driving cars — checking how aligned two motion vectors are, to predict if paths will cross.
- Medical AI — comparing a new patient’s data vector to vectors from known past cases.
- LLMs like ChatGPT — attention scores between words are dot products, calculated constantly as the model reads and writes.
Why This Matters Later
Almost every “smart” feature in AI — search, recommendations, chatbots, face recognition — is quietly running dot products and cosine similarity underneath. If this idea feels shaky, attention in Transformers (a future part) will feel like a magic trick instead of something you can reason about. Once this clicks, you’ll start noticing it everywhere: any time an AI system says “this is similar to that,” a dot product did the work.
Three Things to Remember
- The dot product measures how much two arrows agree — big positive means “same way,” zero means “unrelated,” negative means “opposite.”
- Cosine similarity is the dot product with length removed, so only direction is compared.
- Face recognition, search, and recommendations all boil down to the same move: turn things into vectors, then check their cosine similarity.
In one line: The dot product is a handshake between two arrows — the more they agree, the stronger the handshake.
In math: dot product = (a1 × b1) + (a2 × b2) + … and cosine similarity = dot product ÷ (length of a × length of b).
In real life: Every face unlock, search result, and recommended video starts with a cosine similarity check between vectors.
Next: Part 3 — Matrices as Transformation Machines, where we stop comparing vectors and start moving them: stretching, rotating, and flipping them using matrices.
