Vector Database Explained: How It Works, When You Need One and What to Choose

What Is a Vector Database?

A vector database is a database designed to store embeddings and retrieve the ones most similar to a query vector. An embedding is a fixed-length list of numbers produced by a model, so that items with similar meaning end up close together in a high-dimensional space. Instead of asking "does this row contain the word X?", you ask "which stored items are nearest in meaning to this one?"

Three ideas define the category:

• Vectors as the primary data. Each record holds an ID, an embedding and usually metadata such as source, date or customer.
• Similarity as the query. You search by closeness, not equality.
• Indexes built for speed. Comparing a query against every vector gets slow as data grows, so these systems use specialised indexes.

If you are new to the models that produce embeddings, our guide on what a large language model is gives the background.

How a Vector Database Works, Step by Step

Step 1: Turn data into embeddings

An embedding model converts each document chunk, image or product into a vector. The length depends on the model; common sizes run from a few hundred to a couple of thousand numbers. You must use the same model for stored data and for queries, otherwise the numbers are not comparable.

Long documents are usually split into chunks first. Chunk size is a design choice, and it affects what the search can find.

Step 2: Measure similarity

Closeness is calculated with a distance or similarity metric. Cosine similarity compares direction, Euclidean (L2) distance compares straight-line gap, and inner product compares alignment and size. pgvector, for example, supports L2, inner product, cosine and L1 distance, among others. Use the metric your embedding model was trained for; a mismatch can quietly hurt result quality.

Step 3: Search exactly or approximately

Exact search compares the query to every vector and returns the true nearest ones. It gives perfect recall but cost grows with data size. Approximate nearest neighbour (ANN) search uses an index to look at only a fraction of the vectors. pgvector's documentation puts the trade-off plainly: by default it performs exact search with perfect recall, while approximate indexes "trade some recall for speed."

Here is a tiny brute-force example you can run. The four-number vectors are invented toy values, not real model output:

python

import numpy as np

‍

docs = {

    "refund policy":      [0.9, 0.1, 0.0, 0.1],

    "how to return item": [0.8, 0.2, 0.1, 0.0],

    "gpu pricing":        [0.0, 0.9, 0.3, 0.1],

    "office wifi":        [0.1, 0.0, 0.9, 0.4],

}

names = list(docs)

M = np.array([docs[n] for n in names], dtype=float)

M /= np.linalg.norm(M, axis=1, keepdims=True)      # normalise

q = np.array([0.85, 0.15, 0.05, 0.05]); q /= np.linalg.norm(q)

scores = M @ q                                      # cosine similarity

for i in np.argsort(-scores)[:3]:

    print(f"{names[i]:<20} {scores[i]:.3f}")

Running it ranks "refund policy" (0.995) and "how to return item" (0.994) far above "gpu pricing" (0.188). That is the whole idea in miniature. A real system does the same thing across millions of vectors, which is why the index matters.

Step 4: Filter and combine with keywords

Real queries add conditions: only this tenant, only documents after a date. This is metadata filtering. Many systems also combine vector search with keyword search, often called hybrid search, because exact terms such as product codes or names still matter.

Filtering has traps. pgvector's documentation notes that with approximate indexes, "filtering is applied after the index is scanned," so a restrictive filter can return fewer rows than you asked for. Test filtered queries, not just unfiltered ones.

Indexes in Plain Words: HNSW, IVF and Flat

HNSW comes from a paper by Yu. A. Malkov and D. A. Yashunin. Its abstract describes a fully graph-based approach built as a multi-layer set of proximity graphs, where starting the search at the upper layer allows "a logarithmic complexity scaling." That is the authors' claim about how cost grows with dataset size; real performance depends on your data, settings and hardware.

pgvector's documentation compares the two indexes it offers. HNSW "has better query performance than IVFFlat (in terms of speed-recall tradeoff)" but "has slower build times and uses more memory." IVFFlat builds faster and uses less memory, and should be created after the table has data. Both are approximations, so results can differ from an exact search.

Why LLM Applications Need Vector Databases

A language model only knows what was in its training data plus what you place in its prompt. Retrieval-augmented generation fixes the gap by fetching relevant material at question time. The original RAG paper by Lewis and colleagues (NeurIPS 2020) combined a pre-trained model with a "dense vector index of Wikipedia" as non-parametric memory, which is the same pattern used today.

In a modern RAG pipeline, the flow is:

1. Chunk and embed your documents, then store them in a vector database.
2. Embed the user's question with the same model.
3. Retrieve the nearest chunks, applying filters such as access rights.
4. Place those chunks in the prompt so the model answers from them.

Vector search is not the only retrieval choice. Relationship-heavy questions can suit graphs, which we compare in GraphRAG vs vector RAG, and agents that decide when and how to retrieve are covered in agentic RAG. If you are weighing retrieval against training, read RAG vs fine-tuning. Vector stores also appear as long-term recall in AI agent memory, and what you retrieve is only part of what the model sees, a topic handled under context engineering.

Beyond RAG, the same technology drives product recommendations, duplicate detection, image search and support-ticket matching.

Vector Database vs Traditional Database

They are complements. Your orders, customers and payments still belong in a transactional database. Our overview of SQL vs NoSQL explains how those models differ. The boundary is also blurring: pgvector adds vector search to PostgreSQL, and its documentation lists ACID compliance, point-in-time recovery and joins among the benefits of staying inside Postgres.

Do You Need a Dedicated Vector Database?

Not always. Match the tool to the size and risk of the project.

• A library such as FAISS. FAISS is not a database; its package page describes it as "a library for efficient similarity search and clustering of dense vectors," developed primarily at Meta's Fundamental AI Research group, with optional GPU support and the ability to search sets too large for RAM. It suits experiments and embedded use where you manage storage yourself.
• Vector search inside your existing database. If your data already lives in PostgreSQL and your scale is moderate, pgvector keeps vectors next to your business data and lets you join them. It is often the lowest-friction start.
• A dedicated vector database. Systems such as Milvus, Qdrant, Weaviate and Pinecone are built around vector workloads, and typically add scaling, replication and richer filtering. Consider one when vector volume, query load or operational needs outgrow a general database. Compare current features and pricing yourself; they change quickly.

A sensible path is to prototype small, measure retrieval quality, and move up only when a real limit appears.

A Worked Scenario: A Support Assistant for a Lender

This is a hypothetical example to show the approach; the company and numbers are invented.

A mid-sized Indian lender wants an assistant that answers staff questions from loan policy PDFs, circulars and FAQs.

Build. The team splits documents into chunks, attaches metadata (document type, effective date, product line), embeds them and loads them into PostgreSQL with pgvector, because their customer data already lives there.

Problem. Early answers cite an outdated circular. Staff ask about "foreclosure charges" and the system retrieves a similar-sounding but superseded page.

Fixes. The team adds an effective-date filter so only current documents are searched, adds keyword matching so exact product names rank well, and tests filtered queries to confirm they still return enough results. They also keep a set of real staff questions with known correct source pages.

Evaluation. Each change is scored against that set, using the practices in how to evaluate an LLM application. Only if query volume or index size later strains Postgres would they consider a dedicated system.

Common Mistakes

• Skipping evaluation. Retrieval quality decides answer quality. Measure it with real questions.
• Mixing embedding models. Store and query with the same model, and re-embed if you change it.
• Poor chunking. Chunks that are too large blur meaning; too small lose context.
• Ignoring metadata and permissions. Filter by access rights before results reach the model.
• Trusting approximate results blindly. Check recall against exact search on a sample.
• Choosing the product before the problem. A dedicated system adds cost and operations you may not need yet.
• Treating it as a replacement for SQL. Keep transactional data where transactions belong.

Skills to Learn Next

To work with vector databases you need Python and SQL, a feel for embeddings and similarity metrics, retrieval evaluation, basic data engineering and the habit of testing with real queries. Learners who want structured, mentored practice can look at the LLM, SLM and agentic RAG development programme offered with IIT Jammu; check its page for current eligibility, syllabus and fees before deciding.

TL;DR: A vector database stores embeddings, which are lists of numbers that capture the meaning of text, images or audio, and quickly finds the ones closest to a query. It powers semantic search, recommendations and retrieval-augmented generation (RAG). Most systems use approximate nearest-neighbour indexes such as HNSW, trading a little accuracy for a lot of speed. Small projects can start with pgvector or FAISS; dedicated systems earn their place at larger scale.