Vector Databases Explained

Ashutosh Tripathi

22 min read

·

Sep 28, 2026

--

Listen

Share

Vector databases have quickly become one of the most important building blocks in modern AI applications. If you have worked with RAG, semantic search, recommendation systems, or AI assistants, you have probably come across terms such as embeddings, vector search, cosine similarity, ANN, HNSW, metadata filtering, and vector stores.

The individual concepts are not particularly difficult. What makes them confusing is that they are usually explained separately, without showing how they fit together.

This article tries to connect the dots.

We will start with the basic idea behind a vector database, then gradually move into similarity search, indexing, HNSW, filtering, and finally ChromaDB. The goal is not just to learn the API, but to understand what is happening underneath it and why these design choices matter.

Press enter or click to view image in full size

1. Why do we need vector databases?

Let’s start with a simple example. Imagine we have a knowledge base containing these documents:

Document 1:
How to reset your corporate password
Document 2:
Steps for changing your enterprise account credentials
Document 3:
How to configure Kafka consumer partitions
Document 4:
Troubleshooting login and credential issues

Now a user asks: How do I change my password?

A traditional database can easily answer questions such as:

SELECT *
FROM documents
WHERE text LIKE '%password%';

But keyword matching has an obvious limitation. The user could ask: I forgot my corporate credentials. How can I get access again?

The relevant document might not contain the exact words from the question. What we really want is to search by meaning, not just by matching words. That is the problem vector search is designed to solve.

2. The basic idea behind embeddings

An embedding model converts an object into a vector of numbers. For example:

"How do I change my password?"
                |
                v
         Embedding model
                |
                v
[0.12, -0.45, 0.81, 0.07, ...]

A document is converted in the same way:

"Steps for changing your enterprise account credentials"
                |
                v
         Embedding model
                |
                v
[0.15, -0.42, 0.79, 0.05, ...]

The two vectors are close to each other in the embedding space. Another document such as:

"How to configure Kafka consumer partitions"

will produce a vector that is much farther away. So instead of comparing text directly, we can compare their numerical representations. That is the fundamental idea behind semantic search.

A useful mental model

Think of an embedding as assigning every object a location in a mathematical space. Similar things are expected to be located near each other.

Embedding = representation
Vector database = storage + indexing + retrieval of those representations

This distinction is important. The vector database does not magically understand language. The embedding model determines how meaning is represented. The vector database makes it practical to search that representation efficiently.

3. What exactly is a vector?

A vector is simply an ordered collection of numbers. For example: [0.12, -0.45, 0.81, 0.07]. This vector has a dimension of 4. Modern embedding models often produce vectors with hundreds or thousands of dimensions.

For example: 384 dimensions, 768 dimensions, 1024 dimensions,1536 dimensions, 3072 dimensions

The exact dimension depends on the embedding model. That dimension has a direct impact on storage requirements.

Suppose we store: 10 million vectors in 1536 dimensions in float32. A float32 requires 4 bytes. So the raw vector storage is approximately: 10,000,000 × 1536 × 4 ≈ 61.4 GB

And that is only the vector data. We still need space for:

• indexes
• metadata
• document text
• IDs
• database overhead
• replicas
• caches
• temporary memory

This simple calculation becomes surprisingly useful when designing a production system.

4. What does a vector database actually store?

A common misconception is that a vector database stores nothing more than a large collection of vectors. In practice, a useful record usually looks more like this:

+------------------------------------------------+
| ID                                             |
| document-123                                   |
+------------------------------------------------+
| Embedding                                      |
| [0.13, -0.71, 0.25, ...]                       |
+------------------------------------------------+
| Document                                       |
| "Kafka consumer is experiencing high lag..."   |
+------------------------------------------------+
| Metadata                                       |
| team = platform                                |
| environment = production                       |
| year = 2026                                    |
+------------------------------------------------+

There are four things worth remembering:

ID: A unique identifier for the record.

Embedding: The numerical representation used for similarity search.

Document: The original content or text associated with the vector.

Metadata: Structured information such as tenant, category, date, source, permissions, or document type.

A simplified mental model is:

                Vector Database
                      |
       +--------------+--------------+
       |              |              |
      ID          Embedding       Metadata
                      |
                   Document

5. Vector databases versus traditional databases

This is where it helps to stop thinking of vector databases as a replacement for databases such as PostgreSQL or MySQL. They solve a different class of problem. A traditional relational database is excellent at questions like:

SELECT *
FROM orders
WHERE customer_id = 123
AND amount > 1000;

The database is evaluating explicit predicates. A vector database is optimized for questions like:

Which stored objects are closest to this query vector?

That is a fundamentally different operation.

Press enter or click to view image in full size

The important point is that modern systems often use both.For example:

                      Application
                         |
             +-----------+-----------+
             |                       |
             v                       v
       PostgreSQL               Vector DB
             |                       |
       structured data         embeddings
       transactions            semantic search
       relationships           metadata filtering

A customer-support application might keep: customer, ticket, payment, transaction, status and timestamp in PostgreSQL, while keeping: support conversation embeddings, knowledge-base embeddings and product-description embeddings in a vector store. The application can then combine the two.

6. Similarity search

This is the core operation of a vector database. Suppose we have: Query vector = Q

and millions of stored vectors:

V1
V2
V3
...
Vn

We want to find the vectors closest to Q.

This is called nearest-neighbor search.

For example:

                    Query
                      ●
                    /   \
                   /     \
                  ●       ●
                 D1       D2
                          |
                          ● D3

If the application asks for the top 5 results, the vector database returns the five most similar candidates according to the selected distance metric.

7. How is similarity actually calculated?

There isn’t one universal definition of similarity. The most common measures you’ll encounter are:

• cosine similarity
• Euclidean distance (L2)
• inner product / dot product

Cosine similarity:

Cosine similarity measures the angle between vectors.Conceptually:

       A
       /
      /  θ
     /
    /
   B

If two vectors point in nearly the same direction, their cosine similarity is high. For normalized vectors:

same direction  → close to 1
unrelated       → around 0
opposite        → close to -1

The important intuition is:

Cosine similarity cares about the direction of the vectors rather than simply their absolute magnitude.

Euclidean distance:

Euclidean distance measures the geometric distance between two points. Smaller distance means the vectors are closer.

A ●----------------● B
        distance

Inner product

The dot product is: A · B = Σ(ai × bi)

Its behavior depends on whether the vectors are normalized. This is why choosing a distance metric isn’t just a database configuration detail. You should understand what the embedding model and your retrieval task expect.

8. Exact search works… until it doesn’t

The simplest way to search a vector database is brute force. For every query:

compare query against V1
compare query against V2
compare query against V3
...
compare query against Vn

sort results
return top K

This is exact nearest-neighbor search. It gives us the true nearest neighbors. The problem is scale. If we have one billion vectors, every query potentially requires a comparison against one billion vectors. A rough way to think about the cost is: O(N × D) where: N = number of vectors and D = vector dimension

For small collections, this may be perfectly acceptable. At large scale, it becomes expensive. That’s why vector databases use Approximate Nearest Neighbor (ANN) indexes.

9. What is ANN?

ANN stands for: Approximate Nearest Neighbor. The goal is not to search every vector. Instead, the index tries to quickly identify the region of the vector space containing the most promising candidates. This introduces a trade-off:

Exact Search
    |
    +-- Maximum recall
    +-- More computation
    +-- Higher latency at scale

ANN Search
    |
    +-- Much faster
    +-- Scales better
    +-- May sacrifice some recall

This trade-off is one of the most important ideas in vector search. A production engineer shouldn’t ask only: How fast is the search? The better questions are: How much recall are we getting? What p95/p99 latency are we getting? How much memory does the index consume?

10. Common vector indexing approaches

There are several families of ANN techniques. The ones you will commonly encounter are:

Flat / brute force
HNSW
IVF
Product Quantization
Disk-oriented ANN techniques

They solve the same broad problem in different ways. For many application developers, HNSW is the most important one to understand first.

11. HNSW explained

HNSW stands for: Hierarchical Navigable Small World. That sounds intimidating, but the basic idea is surprisingly intuitive. Instead of treating vectors as a flat list, HNSW builds a graph connecting nearby vectors. Imagine:

      A -------- B
      |          |
      |          |
      C -------- D
       \        /
        \      /
           E

Each vector is a node. Edges connect nodes that are considered useful neighbors. The key idea is that HNSW uses multiple layers. Conceptually:

Layer 2
        A
        |
        D

Layer 1
    A ---- B ---- D
           |
           C

Layer 0
 A ---- B ---- C ---- D ---- E ---- F ---- G ---- H

The upper layers contain fewer nodes. They act like highways. The bottom layer contains more detail.

12. Why does the HNSW hierarchy help?

Suppose we’re searching for a vector near: Q .Instead of examining every node in the graph, we can first move quickly through the upper layer. Something like:

Start
  |
  v
A -------- D
           |
           v
          Q-region

Then we descend into lower layers and perform a more detailed search. The mental model is:

High layer
    ↓
quick navigation

Lower layer
    ↓
refinement

Bottom layer
    ↓
detailed nearest-neighbor search

The HNSW paper describes this as a hierarchical graph structure for efficient approximate nearest-neighbor search. The important thing to remember is not the paper’s implementation details. Remember:

HNSW uses a graph and hierarchy to navigate toward promising regions of the vector space instead of scanning every vector.

13. HNSW insertion

When a new vector is inserted, HNSW has to decide where it belongs in the graph and which other vectors it should connect to. Conceptually:

New vector
    |
    v
Find nearby existing nodes
    |
    v
Create graph connections
    |
    v
Add node to appropriate layer(s)

So the index isn’t free. There is work involved in maintaining it. This creates another important trade-off:

More effort during indexing
            |
            v
Potentially better search quality

In other words, vector databases are constantly balancing:

write performance
search performance
memory
recall
latency

14. Important HNSW parameters

The names vary slightly between implementations, but three concepts are particularly important: M / max neighbors, efConstruction and efSearch

M / max neighbors

This controls how many connections a node can maintain. More connections generally mean:

denser graph
higher memory usage
more construction work
potentially better recall

Fewer connections mean the opposite.

efConstruction

This controls how much effort is spent while building the graph.

Higher values generally mean:

more work during indexing
better graph construction
potentially better search quality

Lower values:

faster indexing
less construction work
potentially lower recall

efSearch

This is a query-time parameter. It controls how aggressively the search explores the graph.

Higher:

more candidates explored
potentially higher recall
higher latency / CPU

Lower:

fewer candidates explored
lower latency
potentially lower recall

This is one of the most useful HNSW tuning concepts to remember. You can think of efSearch as:

How hard should I look before I decide I have found enough good candidates?

15. Recall versus latency

Suppose we know the true top-10 nearest neighbors. An ANN search returns 10 results, but only 8 are among those true top 10. Then a simple recall calculation would be: 8 / 10 = 80%

You could increase the search effort and perhaps improve that recall. But then the query becomes more expensive. Conceptually:

                         Recall
                           ^
                           |
                      ●
                   ●
                ●
             ●
          ●
       ●
------+------------------------------> Search cost

There is usually no single magical configuration. The right setting depends on the application’s requirements. For a search application, you might care a lot about recall. For an extremely latency-sensitive recommendation service, you may accept a small amount of recall loss to hit a strict p99 target.

16. Metadata filtering

Pure similarity search is often not enough.Imagine an enterprise knowledge base containing documents for: Visa,Amazon,Google,Microsoft. A user asks:

How do I troubleshoot Kafka consumer lag?

Finding the semantically closest documents isn’t sufficient. We also need to consider:

tenant = Visa
environment = production
department = platform
document_type = runbook

So the query becomes:

semantic similarity
+
structured filtering

This combination is extremely important in real applications.

17. Why metadata is so useful

Consider a vector record:

{
  "id": "doc-1001",
  "metadata": {
    "tenant": "visa",
    "country": "IN",
    "team": "platform",
    "type": "runbook",
    "year": 2026
  }
}

Now we can ask for: documents semantically related to: "Kafka consumer troubleshooting"

where:tenant = visa
AND
team = platform
AND
year >= 2025

This gives us much more control than semantic similarity alone. It also has an important architectural consequence:

Metadata isn’t just decoration. It becomes part of the retrieval design.

18. ChromaDB

Let’s make all of this concrete with ChromaDB. At the application level, the hierarchy is roughly:

Client
   |
Tenant
   |
Database
   |
Collection
   |
Records

A collection contains records consisting of concepts such as:

ID
Embedding
Document
Metadata

A simplified picture looks like:

Chroma
                   |
              Collection
                   |
       +-----------+-----------+
       |           |           |
      IDs     Embeddings   Metadata
                   |
                Documents

Chroma is useful for learning because its API makes the major vector-database concepts relatively easy to see.

19. Creating a Chroma collection

A simple example:

import chromadb
client = chromadb.PersistentClient(
    path="./chroma_data"
)
collection = client.get_or_create_collection(
    name="engineering_docs"
)

The important concept here is the collection.You can think of it as a logical container for related records.

For example:

engineering_docs
support_tickets
product_catalog
customer_knowledge

could be separate collections. Exactly how you divide collections should be driven by things such as:

• lifecycle
• tenancy
• security
• embedding model
• query patterns
• operational requirements

rather than simply creating a new collection for every category.

20. Adding documents

A simple Chroma example might look like:

collection.add(
    ids=[
        "doc-1",
        "doc-2"
    ],
    documents=[
        "Kafka consumer lag troubleshooting guide",
        "PostgreSQL query optimization guide"
    ],
    metadatas=[
        {
            "team": "platform",
            "type": "kafka",
            "year": 2026
        },
        {
            "team": "database",
            "type": "postgres",
            "year": 2026
        }
    ]
)

At a conceptual level, this triggers:

Document
   |
   v
Embedding function
   |
   v
Vector
   |
   +--------------------+
   |                    |
   v                    v
Vector index       Metadata storage

The exact internal implementation depends on the Chroma deployment and configuration, but this is the mental model worth keeping.

21. The role of the embedding function

This part is easy to overlook. Suppose we add: “Kafka consumer lag”. Chroma needs an embedding before it can perform vector similarity search. Conceptually:

Text
 |
 v
Embedding model
 |
 v
[0.12, -0.45, 0.81, ...]
 |
 v
Vector index

The same idea applies when a user searches.

User query
 |
 v
Embedding model
 |
 v
Query vector
 |
 v
Vector search

This leads to an important rule:

The query and indexed documents need compatible embeddings.

You generally cannot casually change the embedding model underneath an existing vector collection and expect retrieval to remain equivalent.

22. Embedding model versioning

Consider this migration:

Version 1
embedding-model-v1
1536 dimensions
     ↓
Version 2
embedding-model-v2
1536 dimensions

Even though both models produce 1536-dimensional vectors, their vector spaces may be completely different. So you should think about embedding configuration as part of your data contract.

A production system may keep track of:

embedding_model
embedding_dimension
distance_metric
index_version

For example:

{
  "embedding_model": "model-v2",
  "dimension": 1536,
  "metric": "cosine"
}

This becomes particularly important during migrations and re-indexing.

23. Querying Chroma

Now suppose we want to find documents related to: “How can I troubleshoot Kafka consumer lag?” A typical query is:

results = collection.query(
    query_texts=[
        "How can I troubleshoot Kafka consumer lag?"
    ],
    n_results=5
)

The conceptual flow is:

User query
    |
    v
Embedding function
    |
    v
Query vector
    |
    v
ANN index
    |
    v
Nearest candidates
    |
    v
Documents + metadata

You don’t have to manually calculate the vector or perform the nearest-neighbor math when using Chroma’s higher-level text-query API.

24. query() versus get()

These two operations are worth memorizing.

query() Means: Find things similar to this query.

For example:

collection.query(
    query_texts=["Kafka troubleshooting"],
    n_results=5
)

This is a similarity-search operation.

get() Means: Give me these records.

For example:

collection.get(
    ids=["doc-1", "doc-2"]
)

This is useful when you already know which records you want. So the simple memory trick is:

query() → similarity
get()   → retrieval

25. Metadata filtering in Chroma

Suppose we only want Kafka documents. We can filter metadata:

results = collection.query(
    query_texts=["Kafka consumer troubleshooting"],
    n_results=5,
    where={
        "type": "kafka"
    }
)

The conceptual query becomes:

semantic similarity
WHERE
type = kafka

This is one reason vector databases are more useful than treating embeddings as nothing more than arrays stored in a file.

26. Comparison filters

Chroma supports metadata comparisons such as:

$eq
$ne
$gt
$gte
$lt
$lte

For example:

where={
    "year": {
        "$gte": 2025
    }
}

This means:

year >= 2025

27. $in and $nin

Suppose we want several acceptable types:

where={
    "type": {
        "$in": [
            "kafka",
            "redis",
            "postgres"
        ]
    }
}

This is equivalent to:

type IN (kafka, redis, postgres)

There is also $nin for the opposite case. This becomes particularly useful when the metadata represents categories, tenants, environments, or document types.

28. $and and $or

You can also combine filters. For example:

where={
    "$and": [
        {
            "team": "platform"
        },
        {
            "year": {
                "$gte": 2025
            }
        }
    ]
}

This means:

team = platform
AND
year >= 2025

Or:

where={
    "$or": [
        {
            "type": "kafka"
        },
        {
            "type": "redis"
        }
    ]
}

which means:

type = kafka
OR
type = redis

The syntax is less important than the underlying idea:

Semantic retrieval can be constrained with structured predicates.

29. where versus where_document

Another small distinction that is easy to forget:

where is for metadata.

where={
    "team": "platform"
}

while: where_document is for filtering the document content itself.

For example:

where_document={
    "$contains": "Kafka"
}

The mental model is:

where
   ↓
metadata

where_document
   ↓
document text

Chroma also supports additional document-content operators such as regex-based filtering.

30. Combining everything

A realistic query can combine semantic search with structured and document-level constraints.

For example:

results = collection.query(
    query_texts=[
        "Kafka consumer troubleshooting"
    ],
    n_results=10,
    where={
        "$and": [
            {
                "team": "platform"
            },
            {
                "year": {
                    "$gte": 2025
                }
            }
        ]
    },
    where_document={
        "$contains": "Kafka"
    }
)

Conceptually:

                    Query
                      |
                Query embedding
                      |
        +-------------+-------------+
        |             |             |
    metadata      document       vector
     filter         filter       search
        |             |             |
        +-------------+-------------+
                      |
                      v
                 candidates
                      |
                      v
                    Top-K

This is much closer to a production retrieval pipeline than a simple:

question → vector → nearest vector

31. Chroma and HNSW

Chroma’s single-node architecture uses HNSW for approximate nearest-neighbor indexing. The collection configuration can specify the HNSW space and relevant parameters.

For example:

collection = client.create_collection(
    name="engineering_docs",
    configuration={
        "hnsw": {
            "space": "cosine",
            "ef_construction": 200
        }
    }
)

Current Chroma documentation supports HNSW settings such as:

space
ef_construction
ef_search
max_neighbors
num_threads
batch_size
sync_threshold
resize_factor

The exact options and defaults are version-dependent, so the current Chroma documentation should be checked when configuring production deployments.

32. The space setting

The HNSW configuration can use spaces such as:

l2
cosine
ip

Conceptually:

space = how vector distance is measured

For example:

"space": "cosine"

means the index uses cosine-based distance. Don’t memorize one metric as universally correct. The choice depends on:

embedding model
vector normalization
retrieval problem
ranking behavior

33. HNSW in Chroma: the parameters that matter

The three parameters worth remembering are:

max_neighbors
ef_construction
ef_search

Think of them this way:

max_neighbors
    |
    +-- graph connectivity
    +-- memory
    +-- index structure
ef_construction
    |
    +-- how hard we work while building the graph
ef_search
    |
    +-- how hard we work while searching the graph

So:

     Index construction
             |
      ef_construction
             |
             v
        graph quality
             |
             v
          ef_search
             |
             v
       query-time recall

This mental model is much more useful than memorizing configuration defaults.

34. HNSW and RAM

This is where vector databases become an infrastructure problem rather than simply an application-library problem. For single-node Chroma, the HNSW index needs to be resident in system memory for query/update operations. That means vector capacity planning matters.

For example: 50 million vectors × 1536 dimensions × 4 bytes already gives us roughly: 307GB of raw float32 vectors.

And then we still need to account for index overhead, metadata, documents, operating-system memory, and other runtime needs. So the question:

How many vectors can this machine store?

is not enough.

You need to ask:

How many vectors can this machine search efficiently while maintaining the required latency and recall?

Those are very different questions.

35. A note about Chroma at larger scale

It is worth being precise here because Chroma’s architecture has evolved.The current Chroma documentation distinguishes between:

Single-node Chroma
        ↓
HNSW

and:

Distributed Chroma / Chroma Cloud
        ↓
SPANN-based architecture

SPANN is designed around a different scaling model and aims to reduce the memory requirements associated with very large-scale vector search.

So the statement:

“Chroma uses HNSW”

needs qualification.

A better statement is:

Single-node Chroma uses HNSW; distributed/cloud Chroma uses a different ANN architecture based on SPANN.

This distinction matters when discussing architecture rather than simply writing a local proof of concept.

36. Why pure vector search isn’t enough

Suppose somebody searches: ERR_KAFKA_1047 . A vector model may understand that the query is related to Kafka errors. But when dealing with exact identifiers, error codes, ticket numbers, SKU values, usernames, or configuration keys, lexical search can be much more useful.

This is why production systems often combine multiple retrieval methods.

For example:

                User query
                    |
          +---------+---------+
          |                   |
          v                   v
    Keyword search       Vector search
          |                   |
          +---------+---------+
                    |
                    v
              Candidates
                    |
                    v
                Reranker
                    |
                    v
                 Top K

This is generally referred to as hybrid search.

The idea isn’t that one approach replaces the other.

It is that they complement each other.

37. Reranking

A vector database may retrieve, say, 50 candidates. Those candidates don’t necessarily need to be the final answer. A more expensive model can take those candidates and rank them again.

For example:

Vector search
     |
     v
50 candidates
     |
     v
Reranker
     |
     v
10 strongest candidates

This gives us a useful architecture:

ANN
 |
 +-- cheap
 +-- fast
 +-- high recall candidate generation

Reranker
 |
 +-- more expensive
 +-- more precise
 +-- final ranking

This separation is extremely useful in RAG systems.

38. Chunking matters just as much as the vector database

One of the easiest mistakes in RAG is to spend a lot of time choosing a vector database while ignoring chunking. Suppose you have a 500-page manual. You could embed the entire manual: 500-page document in 1 vectror
Now someone asks:

What is the recommended Kafka retention period?

The vector represents the entire manual. That may not be sufficiently precise. Instead:

500-page document
        |
        v
      chunks
        |
 +------+------+------+
 |      |      |      |
 v      v      v      v
C1     C2     C3     C4
 |      |      |      |
 v      v      v      v
V1     V2     V3     V4

Now the system can retrieve the specific section that is relevant. This leads to an important conclusion: Retrieval quality is not a property of the vector database alone. It depends on:

document parsing
+
chunking strategy
+
embedding model
+
distance metric
+
index configuration
+
metadata
+
query formulation
+
filtering
+
reranking

39. The complete RAG flow

Once all the pieces are put together, a typical RAG system looks like this.

Ingestion

Documents
    |
    v
Parsing
    |
    v
Chunking
    |
    v
Embedding model
    |
    v
Vectors
    |
    v
Vector database

The vector database contains:

ID
Embedding
Document
Metadata
Index

Query

User question
     |
     v
Query embedding
     |
     v
Metadata filtering
     |
     v
ANN search
     |
     v
Top-N candidates
     |
     v
Optional reranking
     |
     v
Top-K chunks
     |
     v
Prompt augmentation
     |
     v
LLM
     |
     v
Answer

This is the mental model I keep coming back to when working with RAG systems.

Similarity Search

Similarity search is the process of finding items in a dataset that are most similar to a given query item. At the core of similarity search is a distance or similarity metric that quantifies how alike two data points are. The choice of metric depends on the nature of the data and the application.

Common Distance and Similarity Metrics

1. L2 Distance (Euclidean Distance)

The L2 distance between two vectors a and b is the square root of the sum of the squared differences between corresponding elements:

L2(a,b)= (a1​−b1​)2+(a2​−b2​)2​

For higher dimensions we proceed analogously. For instance, to calculate the L2 distance between vectors q=[1,2,3] and r=[4,5,6]:

L2(q,r)= (q1−r1)2+(q2−r2)2+(q3−r3)2
L2(q,r)= (1−4)2+(2−5)2+(3−6)2L2(q,r)= (1−4)2+(2−5)2+(3−6)2​
L2(q,r)= ≈5.20L2(q,r)= ≈5.20

Use Case: A common use case for L2 distance is determining the closest point to a given location in two-dimensional (2D) or three-dimensional (3D) space. This approach is frequently used in computer vision tasks, where spatial proximity between features or objects plays a critical role.

2. Dot Product (Inner Product) Similarity and Distance

Definition:
The dot product of two vectors a: (a1,a2) and b:(b1,b2)is:

a⋅b= a1​b1​+a2​b2​​

Example 1:
Let’s see how we can calculate the dot product between vectors a=[4,8] and vector b=[11.5,5]:​
a⋅b= a1b1+a2b2
a⋅b= 4×11.5+8×5
a⋅b=86

Example 2:
For higher dimensions we proceed analogously. For instance, to calculate the dot product between vectors q=[1,2,3]and r=[4,5,6]:​
q⋅r= q1r1+q2r2+q3r3
q⋅r= 1×4+2×5+3×6
q⋅r= 32q⋅r= 32

Note: Dot product is a similarity metric: the greater the dot product, the more similar the vectors are. In order to convert dot product to a distance, one calculates the negative of the dot product: the greater the negative dot product, the greater the distance between two vectors.

Properties:

• The dot product of two vectors can be positive, negative, or zero, depending on the angle between them.
• Larger dot product values typically indicate a higher degree of similarity between the vectors, especially when they are pointing in similar directions.
• If a distance-like metric is needed, the negative of the dot product can be used. In this case, larger (more negative) values correspond to greater dissimilarity or distance.
• The dot product is sensitive to both the magnitude and direction of the vectors, meaning that changes in either will affect the result.
• It is frequently used in machine learning models, such as in neural networks for computing activations or in matrix factorization for recommendation systems.

Use Case: When the length of the vector (representing something such as relevance, confidence, or quantity) is meaningful. For instance, in recommendation systems, the direction of vectors typically indicates two products are about the same topic, but larger magnitudes might indicate that a product is more popular. In this case using dot product similarity makes sense, because one would want to recommend the products that are not only about the same topic, but also popular.

3. Cosine Similarity and Distance

Cosine similarity measures the cosine of the angle between two vectors:

Press enter or click to view image in full size

Properties:

• Cosine similarity focuses on the orientation of vectors rather than their magnitude, measuring the cosine of the angle between them.
• It is particularly well-suited for high-dimensional, sparse data, such as text embeddings or term-frequency vectors in natural language processing.
• This metric is invariant to vector length, meaning that scaling a vector up or down does not affect the similarity score.

Use Case:

• A common use case for cosine similarity is measuring document similarity in natural language processing (NLP), where it helps identify texts with similar content regardless of their length.

Choosing the Right Metric:

Press enter or click to view image in full size

Practical Considerations

Normalization: Normalize vectors if you expect to only need to calculate cosine similarity. For many natural language processing tasks and text embedding models, this is the default option.

High-Dimensional Data:

• L2 distance can suffer from the ‘curse of dimensionality.’ Consider using a different metric or using a dimensionality reduction technique if the data exhibits high dimensionality.
• Cosine distance often performs better in high dimensions, and is the default choice in many natural language processing and text-based tasks.
• Dot product can be computed efficiently using matrix operations, and can be used to calculate cosine similarity if the vectors are normalized.

Choosing the right distance metric is crucial for effective similarity search. L2 distance works well for continuous, lower-dimensional data where magnitude matters. Cosine distance excels with high-dimensional, sparse data where direction is more important than magnitude. Dot product offers computational efficiency and is useful when both magnitude and direction contribute to similarity. Understanding these trade-offs helps in selecting the most appropriate metric for your specific use case.

FAISS

FAISS (Facebook AI Similarity Search) is a library from Meta for fast vector similarity search. It is designed to search embeddings efficiently, and can run on a single machine using CPU or GPU. Unlike a vector database, FAISS itself does not provide a built-in database/server layer; you interact with it programmatically.

What problem does FAISS solve?

Suppose you have embeddings:

Document 1 → [0.2, 0.8, 0.1, ...]
Document 2 → [0.9, 0.1, 0.4, ...]
Document 3 → [0.3, 0.7, 0.2, ...]
..

A user’s query is also converted into an embedding:

Query → [0.25, 0.75, 0.15, ...]

FAISS helps you find the k most similar vectors to that query efficiently.

This is fundamental to applications such as semantic search and RAG.

FAISS Index: An index determines how FAISS searches through vectors. Different indexes trade off: Speed ↔ Memory ↔ Accuracy

1. Flat Index: The simplest approach.For a query vector, compare it against every vector in the collection.

Query
  │
  ├── compare with Vector 1
  ├── compare with Vector 2
  ├── compare with Vector 3
  ├── ...
  └── compare with Vector N

It can use Euclidean distance or dot product, then returns the k nearest vectors. Advantage: Very accurate because it performs brute-force comparison. Disadvantage: Slow for large datasets because every vector must be examined.

2. IVF — Inverted File Index: IVF tries to avoid searching the entire dataset. The vectors are first clustered around centroids, typically using techniques such as k-means. Conceptually:

All vectors
                  │
        ┌─────────┼─────────┐
        ↓         ↓         ↓
      Cell 1    Cell 2    Cell 3
       ●●●       ●●●●      ●●
        ...

When a query arrives:

Query
  ↓
Find nearest centroid/cells
  ↓
Search mainly inside those cells
  ↓
Return nearest vectors

So instead of searching millions of vectors, you search a smaller subset.

Advantage: Faster than brute force.

Trade-off: Some genuinely close vectors might exist in another cell and therefore be missed, reducing recall/accuracy somewhat.

3. LSH — Locality-Sensitive Hashing:LSH uses hash functions designed so that similar vectors are likely to land in the same bucket.

Vector A ──┐
Vector B ──┼──→ Bucket 1
Vector C ──┘

Vector X ──┐
Vector Y ──┴──→ Bucket 2

For a query, FAISS looks in the relevant buckets rather than comparing against everything.

Advantage: Fast and memory-efficient.

4. HNSW — Hierarchical Navigable Small World: HNSW uses a graph organized into multiple layers.

Imagine:

Layer 2:       A -------- X
                \
                 \
Layer 1:    A ---- D ---- X ---- Z

Layer 0:    A-B-C-D-E-F-G-X-Y-Z

The top layers are sparse and act like express routes. As the search moves downward, the graph becomes denser and provides more precise local connections.

Search roughly works like:

Start at upper layer
       ↓
Move toward promising region
       ↓
Go to lower layer
       ↓
Refine the search
       ↓
Find nearest vectors

The transcript describes HNSW as providing a strong combination of speed and accuracy, particularly for large datasets.

FAISS vs ChromaDB

Press enter or click to view image in full size

When should you use FAISS?

Use FAISS when:

• You want high-performance vector search
• You want full control over the indexing approach
• Your workload fits a single machine
• You are comfortable managing the surrounding storage/application layer yourself

For quick prototyping or metadata-heavy AI applications, the transcript favors ChromaDB; for distributed, production-oriented vector database requirements, it favors Milvus

Further reading

• Chroma documentation — Collections, querying, filtering, and configuration
• Malkov & Yashunin — Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs
• Documentation for the embedding model you choose
• pgvector documentation if you are evaluating PostgreSQL as your vector store