Generative AI Architecture · Part 5 of 8

Vector Databases

The indexing step of a RAG pipeline needs somewhere to store embeddings and search them quickly. That somewhere can be a dedicated vector database, purpose-built for the job, or an extension added to a database the team already runs. The choice affects cost, latency, and operational overhead.

What a vector index does

Comparing a query vector against every stored vector one at a time, an exact nearest-neighbor search, returns the true nearest neighbors, but its cost grows linearly with the number of vectors: doubling the collection roughly doubles the search time. A vector index can instead use an approximate method, which builds a structure that finds vectors very likely, though not guaranteed, to be the true nearest neighbors, in far less time. The HNSW (Hierarchical Navigable Small World) method, for example, is reported to scale roughly logarithmically with collection size (Malkov and Yashunin).

Exact or approximate search

Exact search returns perfect recall and suits small to medium collections, where its cost is acceptable. As a collection grows, an approximate index trades some recall for much faster queries. Some systems default to exact search and let you add an approximate index later.

A separate vector database adds a system to integrate and a second copy of the data to keep in sync with the primary data store, and it usually adds cost. Vector support inside a database the team already runs avoids the second copy. The choice depends on the features you need and what the team already runs.

Common options