How RAG Really Works

How RAG Really Works

Think of RAG as a smart librarian plus a clever storyteller working together. The librarian quickly finds the right books (retrieval), and the storyteller reads from those books to answer your question perfectly (generation). No guessing — just facts from your own documents.

The whole process has three big stages:

1. Indexing (the preparation stage — done once or when you add new documents)
2. Retrieval (finding the right information when someone asks a question)
3. Generation (turning that information into a natural answer)

First Stage: Indexing (also called Ingestion)

This happens before anyone asks anything. It's like preparing your library in advance.

You start with your documents — PDFs, Word files, web pages, notes, company policies, anything you want the AI to know about.

Step by step:

• Load the documents and clean them up a bit (remove weird formatting, headers, footers).
• Split them into smaller pieces called "chunks". Why? Because LLMs have limits on how much text they can read at once, and smaller pieces help find exactly the relevant parts.
• Take each chunk and turn it into a vector — a list of numbers that captures the meaning of the text. This is done by an embedding model (like a mini-AI trained to understand similarity between sentences).
• Store all those vectors (plus the original text chunks) in a special database called a vector database. It is designed to quickly find similar vectors when you search.

Popular embedding models right now include things like BGE, E5, or OpenAI's text-embedding models.

Popular vector databases include Pinecone, Weaviate, Qdrant, Chroma, or even pgvector if you're using PostgreSQL.

Once this is done, your "knowledge base" is ready and waiting.

Second Stage: Retrieval (this happens every time someone asks a question)

Now a user types: "What is our vacation policy?"

Here's what happens super fast:

• The system takes the user's question and sends it to the same embedding model used earlier. It turns the question into a vector too.
• That query vector goes to the vector database.
• The database does a similarity search — it looks for the chunks whose vectors are closest (most similar in meaning) to your question vector. Usually it returns the top 3 to 10 matches (called top-k).
• Sometimes there's an extra step called re-ranking to make sure the best ones are really the best, but in basic naive RAG we skip that for speed.

You now have a handful of relevant text chunks pulled from your documents.

Third Stage: Generation (the final answer part)

The system builds a prompt for the large language model (LLM) like GPT, Claude, Llama, or Gemini.

A simple prompt looks something like this:

"You are a helpful assistant. Use only the following context to answer the question. If you don't know, say so.

Context:

[chunk 1 text]

[chunk 2 text]

[chunk 3 text]

Question: What is our vacation policy?

Answer:"

The LLM reads the question plus the retrieved context and writes a clear, natural answer. Because it has the exact relevant info right there, it is much less likely to make things up.

The full flow in one line:

User question → Embed question → Search vector DB → Get relevant chunks → Stuff them into prompt → Send to LLM → Get grounded answer → Show to user

Here's a simple text version of the diagram many people use:

User Query

↓

Embed Query (turns text into numbers/vector)

↓

Vector Database Search (find similar vectors)

↓

Retrieve Top Chunks (relevant text pieces)

↓

Build Prompt: Question + Chunks

↓

Send to LLM

↓

Generated Answer (with facts from your docs)