Generative AI Architecture · Part 4 of 8

Retrieval-Augmented Generation

Expanding the knowledge of models beyond their training.

A foundation model's knowledge is frozen at training time; it has no way to know about information that was not included in the training such as recent events, a company's internal documentation, yesterday's support tickets, or a database row that changed an hour ago. Retrieval-augmented generation (RAG), closes that gap by finding relevant information not included in the training and handing it to the model as part of the prompt, so the model answers from an augmented knowledge base. Semantic search supplies the retrieval half; a production pipeline wraps several more steps around it.

The full pipeline

The pipeline splits into two phases running on different schedules.

Ingestion pipeline: runs ahead of time

flowchart TD
  A[Ingestion] --> B[Parsing]
  B --> C[Chunking]
  C --> D[Embedding]
  D --> E[Indexing]
  

• Ingestion pulls source content in: PDFs, wiki pages, support tickets, database exports.
• Parsing converts that raw content into clean text, stripping layout noise, extracting tables, handling scanned pages.
• Chunking splits long documents into smaller pieces.
• Embedding converts each chunk into a vector.
• Indexing stores those vectors in a structure built for fast nearest-neighbor search.

Query-time pipeline: runs on every query

flowchart TD
  Q[Query] --> RET[Retrieval]
  IDX[Indexing] --> RET
  RET --> RANK[Reranking]
  RANK --> PROMPT[Prompt construction]
  PROMPT --> GEN[Generation]
  GEN --> VAL[Validation]
  

• Retrieval takes an incoming query, embeds it the same way, and finds the closest-matching chunks.
• Reranking re-scores the first pass of candidates, pushing the best matches to the top.
• Prompt construction assembles the final prompt from the system instructions, the user's question, and the retrieved chunks.
• Generation is the model producing an answer from that assembled prompt.
• Validation checks the answer before it reaches the user: does it cite the retrieved sources, does it avoid claiming something the sources don't support, does it match the expected format.

Chunking: size, overlap, and metadata

Chunk size is the length each piece of a document is split into before embedding. Too large, and a chunk mixes several unrelated topics into one vector. Too small, and a chunk loses important surrounding context, for example a sentence that only makes sense next to the paragraph before it.

Chunk overlap repeats a small amount of text at the boundary between adjacent chunks, so context that spans a boundary is partly preserved in both. Metadata attached to each chunk (source document, section title, date, author, access permissions) is stored beside the vector rather than embedded, and it makes filtering possible later: for example, retrieving only chunks from documents a specific user is allowed to see, or only ones updated in the last month.

Hybrid search and reranking

Because keyword search and semantic search have different strengths, many production systems run both and combine the results, an approach called hybrid search: keyword search catches exact terms semantic search can miss (product codes, error messages, names), and semantic search catches paraphrases keyword search can't see. Reranking then reorders that combined candidate list, since the retrieval pass that produced it is tuned for speed over precision.

Possible failure modes

• Plausible-but-wrong retrieval. A chunk can be topically similar to the query and still be the wrong answer: an old pricing page instead of the current one, a similar-sounding but different product's documentation. Semantic similarity measures relatedness; correctness is a separate question.
• Chunks losing surrounding context. A chunk that reads as a self-contained fact in isolation can depend on a caveat, exception, or condition stated a paragraph earlier that didn't make it into the same chunk.
• Stale indexes. If the ingestion pipeline doesn't run again when source content changes, retrieval can returnin outdated content.

What doesn't belong in a vector database

It's tempting, once a RAG pipeline exists, to route every kind of data lookup through it. Structured, relational data, user accounts, orders, inventory counts, anything with exact fields and precise relationships, belongs in a relational database and is queried with SQL. “What's this customer's current subscription tier?” has one correct answer sitting in a row; a vector search would return the closest-sounding match. Databases: Relational and NoSQL covers how to choose the right store for structured data. RAG handles unstructured, meaning-driven lookups (a policy document, a past conversation), and many applications run both side by side.