Vector Databases Explained: How Embeddings, Similarity Search, and AI Retrieval Really Work

It reads your question, searches through hundreds of pages of documents, finds the relevant information in seconds, and responds as if it had always known the answer.

But there's a problem.

Large language models don't continuously learn from the internet. Their knowledge is limited by the data they were trained on. Once training finishes, the model's parameters remain fixed until it's retrained or updated.

So where did the answer come from?

It wasn't hidden somewhere inside the billions of parameters of the model.

It wasn't downloaded from Google during your conversation.

And it certainly wasn't memorized during training.

Yet the model still retrieved information it had never seen before.

Modern AI systems solve this puzzle every single day. Customer support assistants answer questions about newly uploaded company manuals. Coding assistants understand repositories created this morning. Legal copilots search contracts written hours ago. Medical assistants retrieve the latest clinical guidelines without retraining the underlying model.

The language model is only one piece of the system.

Behind the scenes, another component is quietly doing most of the work. It stores millions—even billions—of pieces of information, not as paragraphs or rows in a table, but as mathematical representations that can be searched by meaning rather than by exact words.

Ask about "electric vehicles," and it can still find documents that only mention "battery-powered cars."

Search for "heart attack," and it understands that "myocardial infarction" may describe the same concept.

Traditional databases were never designed to solve problems like these.

To understand how modern AI retrieves knowledge, recommends similar content, powers semantic search, and enables Retrieval-Augmented Generation (RAG), we first need to understand the technology that makes it all possible.

The answer isn't a larger language model.

It's an entirely different kind of database.

And once you understand how it works, you'll start seeing it behind almost every modern AI application you use today.

Modern AI systems contain an entire retrieval layer that most users never see. We'll uncover what the "???" represents throughout this article.

The Search Problem Nobody Talks About6 min read

Imagine opening Google and searching for:

best car for long road trips

Within milliseconds, Google understands what you're asking. It doesn't simply look for web pages containing the exact words best, car, long, road, and trips. It considers synonyms, context, spelling mistakes, related concepts, and hundreds of other signals before deciding which pages are most relevant.

Now imagine performing the same search in a traditional database.

Suppose you have a table containing thousands of automobile articles.

SELECT *
FROM articles
WHERE title LIKE '%car%';

1 rows returned

Then

LIKE '%automobile%'

0 row returned

Neither article contains the exact phrase "best car for long road trips."

A traditional database doesn't understand that:

• car and automobile often describe the same thing.
• road trip and long-distance travel are closely related.
• SUV is a type of car.
• electric vehicle is another category of automobile.

To the database, these are simply different strings of text.

It has no concept of meaning.

That's because relational databases were never designed to understand language. Their job is to store structured information efficiently and retrieve records that match explicit conditions. If you ask for rows where the title contains the word car, the database will faithfully return rows containing car—and ignore everything else.

For decades, this approach worked perfectly.

Banking systems, inventory management, payroll software, airline reservations, and countless other applications rely on relational databases because structured data has well-defined relationships. Customer IDs match customer records. Product IDs match inventory. Order numbers reference invoices.

Natural language is completely different.

Humans rarely use the exact same words to express the same idea.

Consider these three queries:

• best car for families
• best automobile for families
• best family vehicle

This difference seems small until your application contains millions of documents.

Now imagine you're building an AI assistant for a hospital. A doctor searches for "heart attack treatment," but every medical document uses the clinical term "myocardial infarction." A traditional keyword search may completely miss the most relevant documents—not because they don't exist, but because the wording is different.

The same problem appears everywhere.

A customer asks for “cancel my subscription” the documentation says “terminate recurring billing”. An engineer searches for “login bug” the issue tracker says ”authentication failure”. The ideas are identical. The words are not.

Traditional search systems don't understand this distinction because they don't search ideas. They search text.

Modern search engines became much smarter over time.

Instead of simply matching words, algorithms like BM25 rank documents based on how important each term is within a document and across an entire collection.

This dramatically improved search quality.

But it still depends on matching words.

If two documents describe the same idea using completely different vocabulary, lexical search can still fail.

This isn't a flaw in SQL.

It isn't a flaw in Elasticsearch.

And it certainly isn't a flaw in BM25.

They're solving the problem they were designed to solve: finding documents that contain matching terms.

Modern AI, however, asks a completely different question.

Instead of asking:

Which documents contain these words?

And for decades, they've answered that question remarkably well. Banking systems, search engines, e-commerce websites, and enterprise applications still rely on these technologies because matching structured data and keywords is exactly what they were built to do.

Modern AI, however, asks an entirely different question.

Which documents express this meaning?

That subtle shift changes everything.

Instead of comparing characters in a string, AI systems need to compare ideas. Instead of searching for exact words, they need to recognize that car, automobile, and vehicle often describe the same concept—even when none of those words appear together.

Traditional databases were never designed to understand relationships like these.

Not because they're poorly designed.

Because they were solving a different problem.

The challenge is that computers don't naturally understand meaning. They only understand numbers.

So if we want a machine to search ideas instead of words, we first need to answer a much deeper question:

How do you convert language into mathematics?

That question led to one of the most important breakthroughs in modern AI: Embeddings.

The Birth of Embeddings: Turning Meaning into Coordinates7 min read

Imagine opening Google Maps and searching for Paris. Within milliseconds, the application knows exactly where Paris is. It doesn't need a photograph of the city or a written description of its history. Instead, Paris is represented by a set of coordinates that uniquely identify its location on Earth. Those coordinates don't describe what Paris looks like—they simply tell us where it exists on a map.

Now imagine replacing cities with ideas.

Suppose someone says the word car. Without consciously thinking about it, your brain immediately connects it to automobile, vehicle, SUV, truck, and dozens of related concepts. If someone instead says banana, your mind instinctively places that idea somewhere completely different. Somehow, your brain behaves as though every concept occupies a position inside an invisible mental map where similar ideas naturally live close together while unrelated ideas drift farther apart.

No one ever handed you this map.

You built it throughout your life by reading, listening, observing, and experiencing the world. Every conversation, every book, every article, and every memory subtly adjusted where ideas belonged inside your understanding of language.

Modern language models learn something remarkably similar.

Instead of memorizing dictionary definitions or storing lists of synonyms, they gradually organize language into a mathematical landscape. During training, the model repeatedly encounters billions of words appearing together in different contexts. Words and phrases that frequently appear in similar situations slowly migrate toward one another, while concepts that rarely share context naturally separate. Over time, the model constructs a representation of language where meaning emerges from position rather than from explicit rules.

This is one of the most profound shifts in modern artificial intelligence.

Machines don't learn that car means automobile because someone programmed that relationship. They learn it because, across billions of examples, both words appear in remarkably similar contexts. News articles, technical documents, conversations, research papers, and books all reinforce the same statistical relationships until the model begins organizing language into neighborhoods of meaning.

The result is not a dictionary.

Instead of two coordinates, modern embedding models often describe every word, sentence, paragraph, or document using hundreds or even thousands of numerical coordinates. You don't need to visualize all of these dimensions individually. The important idea is much simpler: every additional coordinate gives the model another way to distinguish meaning. Just as adding altitude creates a richer representation of the physical world than a flat map alone, additional dimensions allow language models to capture increasingly subtle relationships between ideas.

These coordinates are called embeddings.

An embedding is simply the location of an idea inside this mathematical landscape.

Notice what this definition deliberately avoids. An embedding is not the meaning itself. It is not a definition, a summary, or a collection of keywords. It is an address. Just as geographic coordinates tell us where Paris exists on Earth, an embedding tells us where a concept exists inside a space of meaning. Two ideas that occupy nearby locations are usually related, while ideas that appear far apart tend to describe completely different concepts.

But where do these coordinates come from?

Nobody manually assigns them.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

embedding = model.encode(
    "Artificial intelligence is changing search."
)

print(len(embedding))

384

This sentence has now become 384 coordinates describing where its meaning exists in semantic space.

This is why modern embedding models can recognize relationships they were never explicitly taught. They understand that doctor is related to hospital, programmer is related to software, and planet is related to solar system, not because someone encoded those facts by hand, but because the geometry of language naturally emerged during training.

Once every sentence, document, image, or piece of code can be represented as coordinates, something extraordinary becomes possible.

Searching for information is no longer a problem of comparing words.

It becomes a problem of comparing locations.

And once information becomes location, the next question almost asks itself:

If every document has coordinates, how do we determine which ones are closest to our query?

That question leads us directly to the mathematics of similarity search.

Similarity Search: Finding Meaning Through Distance9 min read

Imagine you've just converted every document in your database into coordinates. Your research papers, customer support articles, source code, emails, and documentation all now exist as points inside a massive mathematical landscape.

A user asks:

"How do I reduce GPU memory usage during inference?"

That question is also converted into coordinates using the same embedding model. Instead of searching through millions of documents word by word, the system places the query onto the same map.

Now the problem becomes surprisingly simple.

The database doesn't ask:

"Which documents contain these words?"

Instead, it asks:

"Which documents are closest to this point?"

Everything that follows in a modern vector database is built around answering that single question as efficiently as possible.

Humans solve this kind of problem instinctively. If someone asks which café is closest to your house, you don't compare every café in the city. You estimate which locations are nearest based on distance.

Vector databases do exactly the same thing.

The only difference is that instead of measuring physical distance between cities, they measure mathematical distance between ideas.

But this immediately raises another question.

How should we measure that distance?

Surprisingly, there isn't just one answer.

Different applications use different definitions of "close," and each one changes how similarity is calculated.

Measuring Similarity

If two cities are only one kilometer apart, we naturally consider them close together.

Ideas work similarly, but in high-dimensional mathematical space we need formulas to measure closeness.

The three most common approaches are Euclidean Distance, Cosine Similarity, and Dot Product. While they often produce similar results, they answer subtly different questions and are suited for different types of embedding models.

Euclidean Distance: The Straight-Line Answer

Euclidean distance is the most intuitive because it matches how we think about the physical world.

Imagine standing in a park.

One friend is standing five meters away.

Another is standing twenty meters away.

Without any calculations, you know which person is closer.

Euclidean distance asks exactly the same question inside an embedding space:

"What is the shortest straight-line distance between these two points?"

If the distance is small, the ideas are considered similar.

If the distance is large, they are considered different.

This approach is easy to understand and works well for many applications, but modern language models often care less about absolute distance and more about whether two ideas point in the same semantic direction.

Cosine Similarity: Comparing Direction Instead of Distance

Now imagine two arrows.

One is short.

One is long.

Both point in exactly the same direction.

Are they describing the same idea?

In many language models, the answer is yes.

Cosine similarity ignores how long the arrows are and focuses almost entirely on where they point. If two embeddings have nearly the same direction, cosine similarity considers them highly related, even if their magnitudes differ significantly.

This turns out to be an incredibly useful property for semantic search because meaning is often captured more by direction than by absolute length.

As a result, cosine similarity has become one of the most widely used metrics in modern embedding systems.

Dot Product: Similar Direction, Strong Confidence

Dot product is closely related to cosine similarity but also considers the magnitude of the embeddings.

Imagine two arrows pointing in the same direction.

If both arrows are long, the dot product becomes larger.

If one arrow is much shorter, the score decreases.

This additional information can be valuable because some embedding models intentionally encode confidence or importance through vector magnitude.

For this reason, several modern retrieval systems use dot product instead of cosine similarity, particularly when working with embedding models trained for Maximum Inner Product Search (MIPS).

Which One Is Best?

This is one of the most common questions asked by developers.

The answer is:

It depends on the embedding model.

Some models are trained assuming cosine similarity.

Others optimize for dot product.

Some normalize embeddings, making Euclidean distance and cosine similarity produce nearly identical rankings.

The important lesson isn't to memorize formulas.

It's to understand that similarity is not a universal concept.

Every embedding model defines "close" slightly differently.

Code Example: Calculating Similarity with NumPy

Once embeddings are represented as arrays of numbers, calculating similarity becomes surprisingly straightforward.

import numpy as np

car = np.array([0.42, 0.18, -0.91])
automobile = np.array([0.39, 0.16, -0.89])

cosine = np.dot(car, automobile) / (
    np.linalg.norm(car) * np.linalg.norm(automobile)
)

print(cosine)

0.998

A score close to 1 indicates that the two embeddings are pointing in nearly the same direction, suggesting they represent highly similar meanings.

Code Example: FAISS

Finding the closest document becomes even more efficient using libraries like FAISS.

import faiss
import numpy as np

index = faiss.IndexFlatIP(384)

index.add(document_embeddings)

scores, indices = index.search(
    query_embedding,
    k=5
)

In just a few lines, FAISS searches thousands—or even millions—of embeddings and returns the nearest neighbors to the query.

As elegant as this approach is, one practical question remains.

Finding the closest point among ten documents is easy.

Finding the closest point among ten million documents is something entirely different.

Modern AI systems can't afford to compare every query against every embedding one by one. At production scale, that would be far too slow for real-time search, recommendations, or Retrieval-Augmented Generation (RAG).

Clearly, we need something more than just embeddings and similarity metrics.

We need a system that can store, organize, filter, and search billions of embeddings efficiently while returning results in milliseconds.

That system is called a vector database.

And understanding what it actually does is the next piece of the puzzle.

So What Is a Vector Database?9 min read

The previous sections answered three fundamental questions.

First, we discovered why traditional keyword search struggles to understand meaning. Then we learned how language models convert words into coordinates through embeddings. Finally, we explored how similarity metrics allow computers to compare those coordinates mathematically.

But one obvious question still remains.

If every document becomes an embedding, where do all of those embeddings actually live?

Modern AI systems aren't comparing vectors stored inside a spreadsheet or loading millions of embeddings into memory every time a user asks a question. They need a system that can permanently store these representations, retrieve them in milliseconds, update them as new information arrives, and filter them using metadata such as author, date, language, or document type.

That system is called a vector database.

Unlike traditional databases, which primarily organize rows and columns, a vector database is optimized around one central task: storing embeddings and retrieving the most semantically similar ones as quickly as possible. Everything else—indexes, metadata, storage engines, replication, and filtering—exists to make that retrieval efficient at scale.

Every modern Retrieval-Augmented Generation (RAG) system follows this general pipeline. The vector database sits between embeddings and the language model, retrieving relevant context before the LLM generates a response.

Once you see the entire pipeline, the role of a vector database becomes much easier to understand.

Imagine uploading a 500-page PDF into an AI assistant.

The language model doesn't memorize that document.

Instead, the document is broken into smaller chunks—typically paragraphs or sections that can fit comfortably inside an embedding model's context window. Each chunk is then converted into an embedding, transforming human language into coordinates that represent meaning.

These embeddings, along with useful metadata such as the source document, page number, title, author, or publication date, are stored inside the vector database.

Later, when someone asks a question, the exact same embedding model converts the query into coordinates. Instead of searching the original text directly, the database searches for embeddings located nearest to the query in semantic space. The matching document chunks are returned to the application, which then passes them to the language model as context before generating the final response.

The LLM never searched the PDF itself.

The vector database did.

Storing More Than Just Embeddings

One common misconception is that vector databases only store embeddings.

In practice, they store much more.

Every embedding is usually accompanied by metadata that describes the original content. This metadata doesn't influence semantic meaning directly, but it allows applications to filter results before or after similarity search.

For example, an enterprise knowledge base might store information like:

When someone searches for "parental leave policy," the database can first limit the search to HR documents written in English before performing similarity search on the remaining embeddings.

Without metadata, semantic search would often return relevant—but unusable—results.

CRUD Still Exists

Although vector databases introduce new ways of searching, they haven't abandoned traditional database operations.

Applications still need to create, update, delete, and retrieve records as information changes over time.

Imagine a company updating its employee handbook.

The old document shouldn't remain searchable forever.

Instead, the application removes the outdated embeddings, generates new ones from the revised document, and inserts them back into the database. The process feels familiar to anyone who has worked with relational databases—only the stored data happens to be embeddings rather than rows of structured values.

This means vector databases support many of the same operations developers already expect:

• Create new embeddings when new documents arrive.
• Read embeddings during similarity search.
• Update metadata or replace outdated embeddings.
• Delete documents that should no longer appear in search results.

Semantic search may be new, but database fundamentals still apply.

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

index = pc.Index("knowledge-base")

index.upsert([
    {
        "id": "doc-101",
        "values": embedding,
        "metadata": {
            "title": "Employee Handbook",
            "department": "HR"
        }
    }
])

Filtering Before Searching

One of the biggest advantages of vector databases is that semantic similarity doesn't have to search everything.

Suppose you're building a multilingual AI assistant.

A user asks a question in French.

Even if the database contains millions of English documents, there's no reason to search them.

Instead, the application can first filter documents where:

language = "French"

Only then does similarity search begin.

This combination of metadata filtering and semantic retrieval makes vector databases both faster and more accurate than searching every embedding indiscriminately.

Searching at Scale

Everything we've discussed so far works perfectly for hundreds of documents.

Real production systems are very different.

GitHub Copilot indexes billions of lines of code.

Enterprise search systems index decades of documentation.

Scientific databases contain millions of research papers.

Medical retrieval systems store enormous collections of clinical literature.

Searching through every embedding individually would quickly become impossible.

The challenge is no longer understanding meaning.

The challenge is finding nearby embeddings quickly enough that users never notice the search happening.

Code Example – pgvector

One of the most interesting aspects of vector search is that you don't always need a dedicated vector database.

Extensions like pgvector allow PostgreSQL to store embeddings directly alongside traditional relational data.

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding VECTOR(768)
);

This makes it possible to combine SQL queries with semantic search, giving developers the best of both worlds for many applications.

Everything now fits together.

Documents become chunks.

Chunks become embeddings.

Embeddings are stored inside a vector database.

Queries become embeddings.

Similarity search retrieves the nearest neighbors.

The language model receives those retrieved results as context before generating a final answer.

At this point, we understand each individual component.

But modern AI systems don't use these components in isolation.

They work together as part of a much larger retrieval pipeline.

When you upload a PDF to ChatGPT, GitHub Copilot searches your codebase, or an enterprise AI assistant answers questions about internal documentation, dozens of operations happen behind the scenes before the language model generates its first token.

The vector database is only one piece of that system.

So before we open the database itself and explore its internal architecture, let's step back and look at the complete picture.

What actually happens when you ask ChatGPT a question about your company's PDF?

What actually happens when you ask ChatGPT a question about your company's PDF?9 min read

Imagine your company has accumulated ten years of documentation.

Employee handbooks, engineering specifications, API documentation, meeting notes, research papers, design documents, contracts, incident reports, and thousands of internal knowledge base articles are scattered across different systems. Somewhere inside those documents lies the answer to your question—but no language model has memorized any of it.

Now imagine asking an AI assistant:

"What changed in our API authentication system after version 4?"

The response arrives in just a few seconds.

How?

Did the language model search every PDF?

Did it memorize every document?

Or did it somehow connect to your company's file system?

The answer is none of those.

Behind the scenes, an entire retrieval pipeline begins working long before the language model generates its first word. Every component has a specific responsibility, and understanding this pipeline explains not only how Retrieval-Augmented Generation (RAG) works, but also why vector databases have become one of the most important pieces of modern AI infrastructure.

The first thing to notice is that the language model appears almost at the end of the pipeline—not at the beginning.

That surprises many people.

The language model isn't responsible for finding information. Its primary responsibility is generating natural language. Finding the right information belongs to an entirely different retrieval system.

Let's follow the journey one step at a time.

Step 1 — The User Asks a Question

Everything begins with a natural language query.

Suppose a software engineer asks:

"How do I deploy our Kubernetes cluster after the latest security update?"

To a human, this looks like an ordinary sentence.

To the retrieval system, it's simply the starting point of a search operation.

Before anything can be retrieved, the question itself must be transformed into an embedding using exactly the same embedding model that was used to process the company's documents.

Only then do the query and the documents exist inside the same semantic space.

Step 2 — Searching the Vector Database

Once the question becomes coordinates, the vector database takes over.

Instead of scanning documents line by line, it searches through millions of stored embeddings looking for nearby points in semantic space. The retrieval system doesn't care whether the documents contain identical words. It only cares whether they express similar ideas.

If the engineer searches for "API authentication," the database may retrieve documents discussing OAuth tokens, JWT validation, or identity providers, even if the original wording is completely different.

This is the moment where semantic search replaces keyword search.

Step 3 — Retrieving Context

Notice something important.

The vector database does not return an answer.

It returns document chunks.

Those chunks might be paragraphs from an employee handbook, sections of API documentation, pieces of source code, or excerpts from technical specifications. Individually they are just fragments of information, but together they provide the evidence the language model needs to answer accurately.

This distinction is critical.

Vector databases retrieve.

Language models generate.

Confusing these two responsibilities is one of the most common misunderstandings in modern AI systems.

Step 4 — Building the Prompt

The retrieved document chunks are now combined with the user's original question.

Instead of asking the language model only:

"How do I deploy Kubernetes?"

the application sends something much richer.

It includes the user's question, the retrieved document excerpts, system instructions, formatting rules, and any additional context required for the task.

This entire package becomes the final prompt.

The language model now has the information it needs to answer accurately without relying solely on its training data.

Step 5 — The Language Model Generates the Answer

Only now does the language model begin generating text.

By this stage, the difficult work has already been completed.

The documents have been embedded.

The vector database has retrieved the relevant context.

The prompt has been assembled.

The language model simply reasons over the supplied information and produces a response grounded in the retrieved documents.

This is why modern AI assistants can answer questions about information that never existed during their original training.

The knowledge wasn't memorized.

It was retrieved.

Step 6 — Where the Vector Database Fits

By now, the role of the vector database should feel much clearer.

It doesn't replace SQL.

It doesn't replace the language model.

Instead, it becomes the bridge connecting stored knowledge with intelligent generation.

Without the vector database, the language model has no efficient way to search millions of documents.

Without the language model, retrieved documents remain unorganized fragments.

Together, they form a complete retrieval system.

Building This in Production

The architecture you've just seen isn't limited to research labs.

Thousands of production systems use a remarkably similar stack.

A typical deployment might include:

Notice that every layer can be replaced independently. You can swap the embedding model without changing the API, replace the vector database without modifying the frontend, or migrate from a hosted language model to a local model while preserving the rest of the architecture. This modularity is one of the reasons RAG systems have become so popular in production.

query = embed(user_question)

results = vector_db.search(
    query,
    top_k=5
)

prompt = build_prompt(
    user_question,
    results
)

answer = llm.generate(prompt)

Even production RAG systems follow this same logical sequence, although each step is usually implemented with much greater sophistication.

Production Tools

If you're building a RAG application yourself, these tools are widely adopted and well supported:

The pipeline is now complete.

We know how documents become embeddings, how embeddings are stored, how relevant information is retrieved, and how a language model transforms that retrieved context into an answer.

One question remains.

When the vector database receives a query, what actually happens inside it?

How are millions of embeddings stored, indexed, filtered, replicated, and searched fast enough to respond in milliseconds?

To answer that, we need to open the database itself and examine the engineering hidden beneath the API. That is the focus of the next chapter: Inside a Vector Database.

Inside a Vector Database: What Happens After You Press Enter?8 min read

The previous chapter followed the journey of your question through an entire Retrieval-Augmented Generation (RAG) pipeline. By the time the query reached the vector database, it had already been transformed into an embedding, and the database's job was clear: find the most relevant pieces of information as quickly as possible.

From the outside, this process looks almost effortless. A question is submitted, a few milliseconds pass, and the application receives the document chunks it needs.

But inside the database, something far more sophisticated is happening.

Imagine pausing time at the exact moment you press Enter.

Your query has just become an embedding.

It is now approaching a database containing millions—or perhaps billions—of other embeddings.

What happens next?

To answer that question, we need to open the database itself.

The first thing to understand is that a vector database is much more than a collection of embeddings.

If you were to open its storage engine, you wouldn't find millions of vectors arranged neatly in a single file waiting to be searched. Instead, you would discover multiple specialized components, each responsible for solving a different problem.

Some components decide where embeddings should be stored.

Others manage metadata.

Others organize indexes that make retrieval fast.

Others replicate data across machines to prevent failures.

Still others compress embeddings to reduce memory usage.

All of these systems cooperate every time a query arrives.

The similarity search we explored earlier is only one small part of the overall process.

Storage: Where Do Embeddings Actually Live?

After an embedding is generated, it has to be stored somewhere permanently.

During development, a few thousand vectors can comfortably fit inside memory.

Production systems are very different.

A customer support platform may index millions of conversations.

A code search engine may contain billions of source code embeddings.

Scientific search engines routinely process tens of millions of research papers.

The database therefore has to balance two competing goals.

It must retrieve vectors quickly while also storing enormous amounts of information efficiently.

For this reason, modern vector databases typically divide storage into multiple layers. Frequently accessed indexes remain in memory for speed, while larger datasets are stored on disk and loaded only when necessary.

This approach is similar to how operating systems rely on RAM for active data while using SSDs or hard drives for long-term storage. The difference is that vector databases must optimize this process specifically for similarity search rather than general file access.

Metadata: Meaning Needs Context

Embeddings capture meaning.

They do not capture context.

Suppose two engineering documents discuss authentication systems.

One belongs to the mobile application.

The other belongs to the backend infrastructure.

Semantically, both documents may be very similar.

Without additional information, the database cannot distinguish between them.

This is why vector databases store metadata alongside every embedding.

Metadata describes properties of the original document rather than its meaning.

It may include the document title, author, department, language, publication date, access permissions, or any other information the application needs during retrieval.

When a user searches only within engineering documentation published during the last six months, the database first uses metadata to reduce the search space before semantic retrieval even begins.

The result is both faster and more accurate.

Filtering Before Searching

Metadata becomes even more valuable when the database contains millions of embeddings.

Imagine searching every document in an enterprise knowledge base when you only have permission to access one department.

Not only would that waste computational resources, it could also expose confidential information.

Instead, the retrieval engine first filters documents according to metadata rules.

Only the remaining embeddings participate in similarity search.

This dramatically reduces the amount of work the database must perform while ensuring the retrieved information satisfies application requirements.

Most production AI systems combine metadata filtering and semantic retrieval for exactly this reason.

Indexes: Organizing the Search Space

Imagine walking into a library containing twenty million books with no shelves, no catalog, and no organization.

Finding a single book would require examining every book one by one.

A vector database faces the same challenge.

Embeddings cannot simply be stored in one enormous collection.

They must be organized so nearby vectors can be found quickly.

This organization is called an index.

An index is not the data itself.

It is a specialized structure that tells the database where it should search.

Just as a book index points readers toward relevant pages instead of forcing them to read the entire book, a vector index dramatically reduces the amount of work required during retrieval.

In the next chapter we will explore exactly how indexes such as HNSW, IVF, and DiskANN accomplish this.

For now, the important idea is much simpler.

The index is what transforms an enormous collection of embeddings into something searchable.

Compression: Fitting Billions of Embeddings Into Memory

A modern embedding may contain hundreds or thousands of numerical coordinates.

Now multiply that by one billion documents.

The amount of memory required quickly becomes enormous.

Rather than storing every embedding in its original form, many vector databases compress vectors while preserving as much semantic information as possible.

Compression reduces storage costs, improves cache efficiency, and allows larger datasets to fit inside memory.

The challenge is balancing storage savings against retrieval accuracy.

Compress too aggressively, and nearby embeddings may no longer remain nearby.

Compress too little, and infrastructure costs increase dramatically.

Finding the right balance is one of the key engineering decisions behind large-scale vector search systems.

Replication: Preparing for Failure

Imagine an AI assistant serving millions of users every day.

What happens if one server suddenly crashes?

Without replication, every embedding stored on that machine becomes temporarily unavailable.

Modern vector databases avoid this problem by maintaining multiple copies of important data across different machines or availability zones.

If one server fails, another immediately continues serving requests.

Replication is not primarily about speed.

It is about reliability.

Users should never notice that hardware has failed.

Large production deployments therefore replicate indexes, metadata, and stored embeddings to eliminate single points of failure.

Putting Everything Together

By now, the internal architecture should feel much less mysterious.

A vector database is not simply storing embeddings.

It is simultaneously managing persistent storage, metadata, filtering rules, indexes, compression strategies, replication, and retrieval.

Every query passes through these components before the application receives its results.

Each layer exists because modern AI systems must balance speed, scalability, reliability, and cost.

At this point, we understand what a vector database stores and how its internal components cooperate.

But we have deliberately avoided one crucial question.

How does the index itself find nearby embeddings without comparing the query against every vector in the database?

That challenge lies at the heart of modern vector search.

The answer is a family of algorithms collectively known as Approximate Nearest Neighbor (ANN) algorithms—techniques that allow databases to search billions of vectors in milliseconds while sacrificing only a tiny amount of accuracy.

The next chapter explores those algorithms in depth, beginning with the simplest possible approach: comparing every vector one by one, and why that quickly becomes impractical at scale.

How Can a Database Search One Billion Vectors in Milliseconds?10 min read

By now, we've built almost the entire retrieval pipeline.

A user's question becomes an embedding. The vector database stores millions of document embeddings. Similarity search determines which documents are closest to the query, and the language model generates an answer using those retrieved results.

One critical question still remains.

Suppose your vector database contains one billion embeddings.

How does it find the nearest neighbors in just a few milliseconds?

The obvious solution seems straightforward.

Compare the query with every embedding.

Measure the similarity.

Sort the results.

Return the closest vectors.

Mathematically, that works perfectly.

Practically, it becomes a disaster.

If every query required comparing itself against one billion vectors, even extremely fast hardware would struggle to deliver interactive response times. Every additional document would increase the amount of work the database must perform.

For a tiny dataset, this isn't a problem.

For production-scale AI systems, it quickly becomes impossible.

Modern vector databases solve this challenge using a family of techniques collectively known as Approximate Nearest Neighbor (ANN) algorithms.

Notice the word approximate.

These algorithms deliberately avoid checking every vector. Instead, they sacrifice a tiny amount of mathematical perfection in exchange for enormous improvements in speed.

Finding the exact nearest neighbor is sometimes less valuable than finding an almost identical neighbor one thousand times faster.

That tradeoff is what makes modern semantic search practical.

The Simplest Algorithm: Linear Search

Let's begin with the simplest possible solution.

Imagine you have only ten document embeddings.

When a query arrives, you calculate its similarity score against the first embedding, then the second, then the third, continuing until every document has been compared.

Finally, you sort the results and return the closest matches.

This approach is called linear search, sometimes referred to as brute-force search.

Conceptually, it's beautiful because it guarantees the mathematically correct answer. No approximations are involved, and no document is skipped.

For small datasets, linear search is often the best choice because it introduces almost no additional complexity.

The problem appears when the dataset begins to grow.

Searching one thousand embeddings is still manageable.

Searching one million is expensive.

Searching one billion for every user request becomes impractical.

The algorithm scales linearly with the number of stored vectors. Double the dataset, and you roughly double the work required for every search.

Eventually, the database spends more time comparing vectors than serving users.

KD Trees: Organizing Space

If checking every vector is inefficient, perhaps we can organize vectors into a structure that allows large portions of the search space to be ignored.

That is exactly the idea behind KD Trees.

Rather than storing points in one enormous list, a KD Tree repeatedly divides space into smaller regions.

Imagine organizing houses by country, then city, then street.

If someone asks for an address in Paris, you don't inspect houses in Tokyo.

You immediately eliminate enormous portions of the search space.

KD Trees apply a similar strategy to vectors.

At low dimensions, this works remarkably well.

The problem is that modern embeddings don't live in two or three dimensions.

They often contain hundreds or even thousands of coordinates.

As dimensionality increases, the partitions become less effective. Eventually, the tree behaves almost like linear search again.

This phenomenon is known as the curse of dimensionality, and it fundamentally limits many classical search structures.

LSH: Finding Similarity Through Hashing

If partitioning space becomes difficult in high dimensions, perhaps we should stop trying to organize every point precisely.

Locality Sensitive Hashing (LSH) takes a completely different approach.

Instead of searching every vector, it hashes similar vectors into the same buckets with high probability.

Think of it as assigning neighborhoods rather than exact addresses.

When a query arrives, the algorithm searches only within the most promising neighborhoods.

The search becomes dramatically faster because most of the database is ignored.

Of course, there's a tradeoff.

Occasionally, the true nearest neighbor may end up in a different bucket and never be examined.

That's why LSH is approximate.

But for many applications, the speed improvement is well worth the tiny loss in accuracy.

IVF: Searching Only the Right Neighborhood

One idea naturally follows from LSH.

Instead of randomly grouping vectors, what if we intentionally clustered them?

This is the foundation of the Inverted File Index (IVF).

During indexing, the database groups similar vectors together into clusters.

When a query arrives, it first identifies which clusters are closest to the query and searches only inside those clusters.

Most of the database is ignored entirely.

The larger the dataset becomes, the greater the benefit.

Instead of examining one billion vectors, the search may inspect only a few thousand candidates before computing exact similarity scores.

HNSW: Navigating a Graph Instead of Scanning Data

Today's production vector databases frequently rely on Hierarchical Navigable Small World (HNSW) graphs.

Rather than organizing vectors into trees or clusters, HNSW connects nearby vectors into a network.

Imagine standing in an unfamiliar city.

Instead of checking every street, you ask locals for directions.

Each person points you toward someone even closer to your destination.

Eventually, you arrive without ever exploring the entire city.

HNSW behaves similarly.

The algorithm starts from an entry point and repeatedly moves toward neighbors that appear increasingly similar to the query.

Because each step gets closer to the target, the search converges remarkably quickly.

This is one of the reasons HNSW has become the default index for many modern vector databases.

Product Quantization (PQ): Compressing Vectors

Searching billions of embeddings is only part of the challenge.

Storing them efficiently is equally important.

A single embedding may contain hundreds or thousands of floating-point numbers.

Multiply that by billions of documents and memory requirements become enormous.

Product Quantization (PQ) addresses this problem by compressing vectors into compact codes while preserving enough information for similarity search.

The compressed representation dramatically reduces memory usage and allows much larger datasets to fit into RAM or cache.

The database performs most searches on these compressed vectors before refining the final results using the original embeddings when necessary.

Compression introduces a small amount of approximation, but the infrastructure savings are substantial.

DiskANN: When RAM Is No Longer Enough

Even compressed vectors eventually outgrow available memory.

Large AI systems may store tens or hundreds of billions of embeddings.

Keeping everything in RAM becomes prohibitively expensive.

DiskANN extends approximate nearest neighbor search to SSD storage.

Rather than requiring the entire index to remain in memory, it intelligently loads only the portions needed during search.

The result is a system capable of searching datasets far larger than available RAM while maintaining remarkably low latency.

This approach has become increasingly important as enterprise knowledge bases, recommendation systems, and AI search engines continue growing.

Choosing the Right ANN Algorithm

Each algorithm exists because no single solution works best in every situation.

The remarkable thing about modern vector databases is that they rarely rely on just one of these ideas.

A production system might combine HNSW for navigation, PQ for compression, metadata filtering for narrowing the search space, and SSD-backed storage inspired by DiskANN to handle datasets that exceed available memory.

Each technique solves a different engineering problem, and together they make billion-scale semantic search practical.

Understanding these algorithms explains why vector search feels almost instantaneous—even when the database contains more information than a human could read in several lifetimes.

The next question is no longer how vector search works, but which implementation is right for your own applications. In the next chapter, we'll compare the most popular vector databases—from pgvector and FAISS to Pinecone, Milvus, Qdrant, Weaviate, Redis, and Chroma—to understand where each one excels and where its tradeoffs begin.

Choosing the Right Vector Database7 min read

Before we compare anything...

One of the most common questions developers ask after learning how vector databases work is surprisingly simple:

"Which vector database should I use?"

It sounds like there should be a single correct answer.

There isn't.

Choosing a vector database is a bit like choosing a programming language or a cloud provider. Every system is optimized for different trade-offs. Some prioritize developer experience. Others focus on billion-scale datasets. Some integrate seamlessly into existing PostgreSQL deployments, while others are purpose-built for semantic search from the ground up.

A database that performs exceptionally well for a personal RAG project may become prohibitively expensive at enterprise scale. Likewise, a system capable of indexing billions of vectors might be unnecessarily complex for a startup storing only a few thousand documents.

The goal, then, isn't to find the "best" vector database.

The goal is to find the one that best matches your workload.

Before diving into individual products, it's helpful to understand the criteria that actually matter when evaluating a vector database.

Image

What Actually Matters?

Benchmark charts often compare databases using queries per second or indexing speed. Those metrics are useful, but they rarely answer the questions most engineers care about.

Can the database handle metadata filtering efficiently?

Does it support hybrid search?

Can it scale across multiple machines?

Is it open source?

How difficult is it to operate in production?

How expensive does it become when your dataset grows from one million vectors to one billion?

These practical considerations often matter far more than small performance differences.

For the remainder of this section, we'll compare the most widely used vector databases using criteria that reflect real production deployments rather than isolated benchmark numbers.

Pinecone

Pinecone is one of the most widely adopted managed vector databases and is often the first platform developers encounter when building Retrieval-Augmented Generation systems. Its biggest strength is simplicity. Developers can focus on building AI applications without worrying about index management, replication, scaling, or infrastructure.

The platform handles operational complexity automatically while providing excellent metadata filtering, hybrid search, and low-latency retrieval. This makes it particularly attractive for startups and enterprise teams that prefer managed services over operating distributed infrastructure.

The trade-off is cost. Managed convenience comes with recurring operational expenses, especially as datasets and query volumes increase.

For teams that prioritize developer productivity over infrastructure management, Pinecone remains one of the strongest choices available.

Qdrant

Qdrant has become one of the fastest-growing open-source vector databases, largely because it combines excellent performance with a developer-friendly architecture. Unlike many traditional databases that later added vector support, Qdrant was designed specifically for semantic search.

Its metadata filtering is among the strongest in the industry, making it particularly well suited for Retrieval-Augmented Generation, recommendation systems, and enterprise document search.

Developers can self-host Qdrant for complete control or use its managed cloud offering when operational simplicity becomes more important.

For many teams, Qdrant offers one of the best balances between performance, flexibility, and operational cost.

Milvus

Milvus is designed for scale.

When organizations need to index hundreds of millions—or even billions—of embeddings, Milvus becomes an attractive option because its architecture was built specifically for distributed deployments.

Its indexing capabilities, scalability, and support for multiple ANN algorithms make it particularly appealing for large AI platforms and research organizations.

The downside is operational complexity. Running Milvus effectively often requires more infrastructure expertise than lightweight alternatives.

For enterprise AI infrastructure, however, that investment can be worthwhile.

Weaviate

Weaviate takes a slightly different approach by offering a broader ecosystem around semantic search. Beyond vector storage, it includes modules for knowledge graphs, multimodal retrieval, and integrations with popular embedding providers.

Developers building sophisticated semantic applications often appreciate its rich API design and extensibility.

The learning curve is slightly steeper than lightweight databases, but the flexibility is substantial.

pgvector

Not every application needs a dedicated vector database.

If your project already relies heavily on PostgreSQL, adding pgvector can dramatically simplify your architecture.

Instead of operating two separate databases—one relational and one vector—you can store embeddings alongside traditional relational data.

For small to medium-sized applications, this can be an elegant solution.

Eventually, however, very large vector workloads begin to expose PostgreSQL's limitations compared to specialized vector search engines.

Image

Redis

Redis is traditionally known as an in-memory key-value store, but recent versions have introduced vector search capabilities.

Its greatest strength remains latency.

Applications requiring extremely fast retrieval—particularly when vectors are already closely tied to cached application state—can benefit from Redis.

For large document retrieval systems, however, dedicated vector databases generally provide richer indexing and search capabilities.

FAISS

FAISS is not a database.

It's an indexing library.

This distinction is important.

FAISS gives developers direct access to state-of-the-art similarity search algorithms, making it extremely popular in research and custom infrastructure.

What it doesn't provide is everything surrounding those algorithms.

There is no built-in API, authentication, replication, distributed management, or metadata filtering.

If you're building your own search engine from scratch, FAISS is an outstanding foundation.

If you're deploying a production enterprise RAG system, you'll usually want additional infrastructure around it.

Chroma

Chroma focuses on simplicity.

Getting started takes only a few minutes, making it ideal for tutorials, prototypes, notebooks, and local AI experiments.

Many developers begin their vector search journey with Chroma before eventually migrating to larger production systems.

Its limitations become apparent primarily when scaling beyond modest workloads.

Which One Should You Choose?

There is no universally best vector database.

Instead, the decision depends on the scale of your application, operational expertise, infrastructure preferences, and budget.

The recommendations below summarize common production scenarios.