What Happens Before Your AI Answers? The Answer Is RAG
Everyone’s talking about AI. Agentic AI, LLMs, foundation models. But ask most people how an AI actually knows a company’s refund policy and I get silence. It wasn’t trained on your documents. So where does that answer come from? That’s RAG. And it’s not just another LLM trick.
Let me narrate a story. I’m managing a large company with thousands of documents scattered everywhere employee policies, product manuals, contracts, meeting notes from the last three years, everything is everywhere. Now a new employee joins my team. They’re confident, they’re fluent, they answer every question without hesitation but they’re wrong. Not because they’re incompetent because they never actually read my documents. They’re answering from general experience. Best guesses dressed up as facts. That new employee is your AI without RAG.
The fix is obvious when I think about it. Before they start answering, I give them the relevant documents and ask them to read first, then respond. Suddenly they’re accurate not because they got smarter, but because now they’re speaking from my actual documents, not assumptions. That is the entire soul of RAG. Retrieve first, then answer. Technically, RAG works by converting documents into embeddings dense vector representations of meaning and retrieving the most relevant ones at query time using similarity search.
And here’s something most people skip over entirely. Why not just train the AI on my company documents from the start? I get why people ask this. It seems logical but retraining a large language model (LLM) every time my refund policy updates or a new contract gets signed would cost thousands of dollars, take weeks, and still be outdated by the time it’s done. RAG fixes all of that by keeping my knowledge completely updatable, and always current.
Now let’s open the hood:
My documents don’t sit in a vector database as text. They live as meaning encoded in numbers. The AI can’t search them by matching keywords because two people can ask the exact same question in completely different words and a keyword search would miss half of them.
So every document gets broken into smaller chunks. Some systems use simple token limits or fixed sizes. Others use Natural Language Processing to split at natural idea boundaries recognizing where one thought ends and another begins so a paragraph about my refund policy becomes one chunk and a section about shipping timelines becomes another.
Each chunk then goes through an embedding model. It reads the chunk and converts it into a long list of numbers called a vector a mathematical representation of what that chunk actually means. Here’s where it gets interesting: two chunks saying the same thing in completely different words end up with very similar numbers. The model is capturing meaning, not language. All these vectors get stored in a vector database, organised entirely by meaning. When a question arrives it gets converted into a vector too, and the system searches for whichever stored vectors sit closest those are the most semantically relevant chunks and only those get handed to the AI.
What happens when a question arrives:
One of my customers types: what is the return window for international orders?
The system doesn’t search for those words. It converts the question into an embedding in the same vector space as my documents, then asks the vector database: which chunks are closest in meaning to this? It finds the top few, pulls them out instantly, places them right next to the customer’s question and hands everything together to the AI.
The AI reads the question. Reads the retrieved context. Answers from my actual documents.
Without RAG:
Get Mallik Siddarth’s stories in your inbox
Join Medium for free to get updates from this writer.
Customer: “What is the return window for international orders?”
AI answers: “Most companies typically offer a 30-day return window for international orders, though policies vary by retailer.”
Confident, fluent. Completely pulled from general internet knowledge, not my policy.
With RAG:
> Customer asks: “What is the return window for international orders?”
> AI answers: “International orders are eligible for return within 14 days of delivery. The item must be unused and in original packaging. Return shipping costs are the customer’s responsibility unless the item arrived damaged.”
Specific, accurate. Straight from my actual refund policy document.
This is why RAG matters beyond just accuracy. Without grounding, my AI hallucinates not randomly, but confidently. It fills gaps with plausible-sounding answers that are simply not true. When I hand the model my actual document it answers from evidence, not invention.
RAG isn’t just another way to use an LLM. It’s a different philosophy entirely, I keep my knowledge outside the model, retrieve it when needed, and ground every answer in something real.
Internally: each chunk passes through an embedding model like OpenAI embeddings or Sentence Transformers, converting it into a dense vector capturing semantic meaning, not keywords. Those vectors get stored in a database like Pinecone, Weaviate, or FAISS organised by meaning so similar ideas cluster together regardless of wording. At query time the question gets embedded using the same model, a similarity search runs — cosine similarity or dot product and the top-k closest chunks get injected into the AI’s prompt as context.
Chunking → embeddings → vector database → similarity search → top-k retrieval → context injection.
All that math. All those vectors. All that similarity search just to do what I would do as a good manager on day one. Hand the employee the right documents before they open their mouth.
— No RAG, no swag, only lag in the bag!
