RAG explained for beginners: build your first retrieval-augmented generation system
You type a question into an AI chatbot. It gives you a confident, well-structured answer. Impressive. Then you ask it about a clause buried in your company’s nondisclosure agreement, or an update your legal team posted last Tuesday, and the whole illusion collapses. The model either confesses ignorance or, worse, invents something that sounds exactly right but is not.
That failure has a name. It is a knowledge boundary problem. And retrieval-augmented generation, almost always abbreviated to RAG, is the architecture the industry settled on to fix it.
This article breaks RAG down from scratch. No jargon walls, no hand-waving. By the end, you will have built a working legal document Q&A assistant in Python. It reads a PDF contract, converts its contents into searchable meaning, and answers your questions using only what the document actually says.
What we will cover
- Why large language models have a knowledge problem
- The anatomy of a RAG pipeline
- Core concepts: chunking, embeddings, vector stores, and retrieval
- Building a legal Q&A assistant with Python, OpenAI, and FAISS
- The full lifecycle of a single question
- What goes wrong in real RAG systems
- Advanced techniques engineers use in production
The knowledge problem with large language models
A large language model is trained on a fixed snapshot of text. Books, articles, code repositories, public web pages. All frozen at a point in time. When training ends, the model’s knowledge stops updating. Ask it about yesterday’s news and it has nothing. Ask it about your private merger agreement and it genuinely cannot know.
This creates three practical failure modes.
The cutoff wall. GPT-4, Claude, Gemini: every major model has a training cutoff date. Events, policy changes, new regulations, updated pricing, anything that happened after that date simply does not exist inside the model’s weights.
The private data gap. Your employment contracts, your legal memos, your internal research. None of that was in the training data. The model has never seen it.
Hallucination. When a model does not know an answer, it does not gracefully stop. It predicts the next statistically plausible token. The result is text that reads like confident expertise but may be completely fabricated. In legal and medical contexts, that is genuinely dangerous.
Some people try stuffing their documents directly into the prompt. Works fine for a two-page memo. Falls apart entirely for a 400-page contracts archive. Every model has a context window, a hard ceiling on how much text it can process in one call. Even models with large windows become less reliable when overwhelmed with irrelevant material. The model loses the thread.
Fine-tuning is the other obvious fix. But fine-tuning is expensive, slow, and static. If your document changes, you retrain. It is the wrong tool for dynamic, frequently updated knowledge.
RAG sidesteps all of this. It does not teach the model new facts. It gives the model a reference document at the exact moment it needs to answer a question.
What RAG actually is
RAG stands for retrieval-augmented generation. The three words describe the three steps.
Retrieval: find the relevant passages from your data source. Augmented: add those passages to the user’s question. Generation: ask the LLM to write an answer using only what was retrieved.
The model’s job is no longer to recall facts from training memory. Its job is reading comprehension. You hand it a relevant excerpt and ask it to synthesize an answer. That is a task LLMs do extraordinarily well.
Think of it this way. A closed-book exam tests memory. An open-book exam tests understanding. RAG converts every query into an open-book exam, where the book is your specific data.
The anatomy of a RAG pipeline
Before writing a single line of code, you need to understand what is actually happening inside a RAG system. There are five stages.
1. Document ingestion
Everything starts with raw source material. PDFs, Word files, web pages, database records, Markdown notes. All of it qualifies. In the AI world, we call all of these “documents” regardless of format. The first task is loading that content into memory in a form your code can work with.
2. Chunking
You cannot feed a full 200-page legal contract into a vector database as a single object. Retrieval would be meaningless. You need to break the document into smaller, self-contained pieces called chunks. A chunk might be a paragraph, a section, or a fixed number of characters.
Chunk size matters more than most beginners expect. Too large, and your retrieved results carry a lot of irrelevant text that confuses the model. Too small, and individual chunks lose the context needed to answer questions that span multiple sentences. There is no universal correct answer. It depends on your document structure. A good starting point for prose-heavy documents is somewhere between 400 and 700 characters, with a small overlap between adjacent chunks so that no idea gets accidentally split apart.
LangChain’s text splitter documentation covers the different splitting strategies in detail if you want to go deeper on this topic.
3. Embedding
This is where the real magic lives, and it confuses beginners more than anything else. Let us slow down.
Computers cannot compare text the way humans do. They need numbers. An embedding model converts a piece of text into a list of numbers called a vector. That vector does not just encode the words. It encodes the meaning of the words in relation to everything the embedding model learned during its own training.
Here is the key property: semantically similar text produces numerically similar vectors. The sentence “What is the termination notice period?” and the clause “Either party may terminate this agreement with thirty days written notice” will produce vectors that are close together in mathematical space, even though they share almost no words.
This is not keyword matching. It is meaning matching. That distinction is what makes RAG dramatically more useful than a traditional search engine.
OpenAI’s text-embedding-3-small model, which we will use in our project, maps text into a 1536-dimensional space. You do not need to understand the math. You just need to know that closer vectors mean more similar meaning.
You can read more about how embeddings work in OpenAI’s embedding guide.
4. Vector storage
Once every chunk has been converted into a vector, you need somewhere to store them. A regular relational database stores rows and supports exact-match queries. That is useless here. You need a database that can find the numerically closest vectors to a given query vector, fast.
FAISS (Facebook AI Similarity Search) is an open-source library built for exactly this. It was developed by Meta’s AI research team and is one of the most widely used vector index libraries in the world. FAISS runs entirely on your local machine, requires no external service, and handles millions of vectors efficiently. For our project, it is the perfect fit.
Other production options include Pinecone, Weaviate, and Qdrant, all managed vector databases with additional features like filtering, namespacing, and real-time updates.
5. Retrieval and generation
When a user asks a question, the system embeds the question using the same embedding model used during indexing. It then queries the vector store for the top-N most similar chunks. Those chunks get inserted into a prompt template alongside the original question, and the whole package is sent to the LLM.
The prompt typically includes an instruction like: “Answer the user’s question using only the context provided below. If the answer is not in the context, say so.”
That instruction is load-bearing. Without it, the model might blend retrieved context with its training memory and hallucinate anyway.
Building a legal Q&A assistant
Enough theory. We are going to build a tool that lets you ask questions about any PDF legal document: a contract, a lease agreement, terms of service, whatever you have. The assistant will read the document, index it, and answer questions based purely on what is written inside.
What we are using:
- OpenAI Python SDK: for embeddings and the chat model
- LangChain: to manage the pipeline components
- FAISS: for the local vector index
pypdf: to extract text from PDF filespython-dotenv: for clean API key management
Project setup
Create a fresh project directory and set up an isolated environment:
mkdir legal-rag-assistant
cd legal-rag-assistant
python3 -m venv venv
source venv/bin/activate # macOS and Linux
# venv\Scripts\activate # Windows Command Prompt
# .\venv\Scripts\Activate.ps1 # Windows PowerShellInstall the dependencies:
pip install openai langchain langchain-openai langchain-community faiss-cpu pypdf python-dotenvYou will need an OpenAI API key. Get one from platform.openai.com. Create a .env file in your project root:
OPENAI_API_KEY=your_actual_key_herePlace any PDF contract or legal document in the project folder. For this tutorial, name it legal_document.pdf. A sample NDA, a lease agreement, or any multi-page document with dense prose works well.
Step 1: Load the PDF
Create a file called legal_rag.py and start with imports and document loading:
import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnablePassthroughload_dotenv()print("Loading document...")
loader = PyPDFLoader("legal_document.pdf")
pages = loader.load()print(f"Loaded {len(pages)} pages.")
print(pages[0].page_content[:300])
PyPDFLoader extracts raw text from each page and wraps it in a LangChain Document object. Each object carries the text content and metadata like the page number. Running this block lets you confirm the extraction worked before going further.
Step 2: Chunk the document
print("Splitting into chunks...")
splitter = RecursiveCharacterTextSplitter(
chunk_size=600,
chunk_overlap=60
)
chunks = splitter.split_documents(pages)print(f"Created {len(chunks)} chunks.")
print(chunks[0].page_content)RecursiveCharacterTextSplitter tries to split on natural boundaries, paragraphs first, then sentences, then characters, before falling back to a hard character limit. The chunk_overlap=60 parameter repeats the last 60 characters of one chunk at the start of the next. This overlap matters because legal text often has sentences that start a clause on one chunk boundary and finish it on the next. Without overlap, those sentences get separated and retrieval becomes less accurate.
Step 3: Create embeddings and build the FAISS index
print("Embedding chunks and building vector index...")
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = FAISS.from_documents(chunks, embedding_model)print("Vector index ready.")This is the most computationally significant step. For each chunk, the code sends the text to OpenAI’s embedding API, receives a 1536-dimensional vector back, and stores that vector in the FAISS index alongside a pointer to the original text. For a 20-page contract, this might involve 50 to 100 API calls, each taking a fraction of a second.
Get Mealer Mike’s stories in your inbox
Join Medium for free to get updates from this writer.
FAISS stores the index in memory by default. For persistent storage across sessions, you can save and reload it:
# Save the index
vector_store.save_local("faiss_legal_index")# Load it back later
vector_store = FAISS.load_local(
"faiss_legal_index",
embedding_model,
allow_dangerous_deserialization=True
)Step 4: Build the retriever and prompt
retriever = vector_store.as_retriever(search_kwargs={"k": 4})prompt_template = PromptTemplate.from_template("""
You are a precise legal document assistant. Use ONLY the context below to answer the question.
If the answer cannot be found in the context, respond with: "This information is not present in the document."
Do not add information from outside the provided context.Context:
{context}Question:
{question}Answer:
""")
The k=4 parameter tells FAISS to return the four most semantically similar chunks for any given query. Four is a reasonable default for legal documents. It gives enough context to capture multi-clause answers without overloading the prompt with noise.
The prompt template is deliberate. That phrase, “Use ONLY the context below,” is what keeps the model grounded. Without that constraint, the model might blend the retrieved clause with vague training-set knowledge about legal norms and produce a plausible but wrong answer.
Step 5: Assemble the chain and run the chatbot
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt_template
| llm
)print("\n--- Legal Document Assistant ---")
print("Ask questions about your document. Type 'exit' to quit.\n")while True:
question = input("Your question: ")
if question.lower() in ["exit", "quit"]:
print("Session ended.")
break response = rag_chain.invoke(question) if isinstance(response.content, list):
answer = response.content[0]["text"]
else:
answer = response.content print(f"\nAnswer: {answer}\n")
Setting temperature=0 tells the model to be as deterministic as possible. No creativity, no paraphrasing beyond what is needed. For a legal assistant, that is exactly right. You want the model reading and summarizing what is on the page, not improvising.
The RunnablePassthrough() from LangChain's LCEL interface passes the user's raw question straight through to the {question} slot in the prompt template. The question simultaneously gets sent to the retriever via the context pipe, which fetches the relevant chunks, formats them, and inserts them into {context}. Both streams merge into the assembled prompt, which gets sent to GPT-4o mini for generation.
What happens when you ask a question
Walk through the full lifecycle of a single question: “What happens if either party wants to end the agreement early?”
Step 1: Embedding the query. The question goes to text-embedding-3-small. It returns a 1536-number vector representing the semantic meaning of that question.
Step 2: FAISS similarity search. The system scans the index and calculates the distance between the query vector and every stored chunk vector. The four closest chunks are returned. These will likely include the termination clause, the notice period section, and possibly any penalty or survival clause.
Step 3: Prompt construction. The raw text of those four chunks gets slotted into {context}. The original question goes into {question}. The resulting prompt might be 800 words long, mostly contract text plus the question and instructions.
Step 4: LLM generation. GPT-4o mini reads the assembled prompt, identifies the relevant sentences in the context, and writes a clear summary. It will not cite statutes it learned during training. It will not invent a 90-day notice period if the document says 30. It reads and reports.
Step 5: Output. The answer prints to the terminal. Something like: “According to the document, either party may terminate this agreement by providing thirty days written notice to the other party. Termination does not affect any obligations accrued prior to the termination date.”
That answer came entirely from the document. Not from GPT’s training data. Not from a guess. From the specific clause on the specific page.
Pipeline flow diagram
User question
|
v
[Embedding model] ──────────────────────┐
| |
v v
Query vector [FAISS vector index]
|
Top-4 similar chunks
|
v
[Prompt template assembly]
context + question merged
|
v
[GPT-4o mini LLM]
|
v
Final answerWhat goes wrong in real RAG systems
A basic RAG prototype is not hard to build. A RAG system that works reliably in production is a different challenge entirely.
Poor chunking decisions
The most common mistake. Chunks that are too large drag irrelevant text into the context window. Chunks that are too small lose the sentence-level continuity needed to answer anything non-trivial. Legal documents are especially tricky because they use defined terms that only make sense in reference to earlier definitions. A clause referencing “the Effective Date” means nothing without the chunk that defined it.
Semantic chunking, which splits on sentence boundaries rather than character counts, often produces better results for formal prose. LangChain’s experimental semantic chunker is worth exploring once your basic pipeline is working.
Retrieval missing the mark
Semantic search is powerful, but it is not perfect. A user who asks about “cancellation” might not retrieve the clause labeled “Termination” because those two words, despite meaning the same thing in context, sometimes sit at different positions in embedding space. The fix is usually a combination of semantic search and keyword search, a technique called hybrid retrieval. Weaviate’s documentation on hybrid search explains the mechanics well.
Hallucination despite context
RAG reduces hallucination dramatically, but does not eliminate it. A poorly worded prompt, or a model that is too creative, can still blend context with training-set guesses. Setting temperature=0 helps. Strong, specific instructions in the prompt template help more. Some teams add a post-generation check that asks a second model call to verify: "Is this answer fully supported by the provided context?"
Stale vector indexes
This one is quiet and dangerous. You build your index on version one of the contract. The contract gets amended. The vector index still holds version one’s vectors. The model now answers questions about the amended agreement using outdated clauses.
Build a refresh mechanism from the start. Whether that is a scheduled job that re-embeds on file change detection or a more sophisticated pipeline, stale indexes are a production hazard. Tools like LlamaIndex’s ingestion pipeline handle incremental document updates reasonably well.
Latency stacking
Three network calls happen in sequence: embed the query, query the vector store, call the LLM. Each takes time. If the embedding step takes 80ms, FAISS lookup takes 10ms, and GPT-4o mini takes 1.2 seconds, the user waits over a second and a half before seeing anything. Caching common queries and pre-warming the embedding step can cut perceived latency significantly.
Advanced techniques worth knowing
Once your baseline system is working, these are the directions production teams typically move.
Reranking
FAISS returns the top-K most similar chunks, but similar is not the same as relevant. A cross-encoder reranker, a second model that reads both the query and each candidate chunk together, can dramatically improve the quality of what gets passed to the LLM. Cohere’s rerank API is one of the easiest ways to add this step without building from scratch.
Agentic retrieval
In a standard RAG setup, the system always retrieves. Every query triggers an embedding call and a FAISS lookup, even if the question is “thanks” or “hello”. Agentic RAG introduces a decision layer. An LLM agent first decides whether retrieval is even necessary for a given question. If the question is conversational or general, it answers directly. If it is specific and factual, it triggers the retrieval pipeline. LangGraph is the framework most teams use to wire up these conditional flows.
Graph RAG
Standard chunking treats text as isolated segments. Graph RAG extracts entities, including people, organizations, defined terms, dates, and obligations, and maps the relationships between them as a knowledge graph. This makes it possible to answer questions that require traversing relationships rather than matching text. “Which parties are bound by the confidentiality clause, and which of their affiliates does it extend to?” is a relational question that flat chunk retrieval handles poorly. Microsoft’s GraphRAG implementation is open-source and worth examining.
Multi-modal RAG
Legal documents often include tables, signature blocks, and diagrams that standard text extraction mangles or drops entirely. Multi-modal RAG systems process the document as a visual artifact, treating pages as images, using vision models to understand charts and tables, and then indexing that understanding alongside the extracted text. This is a fast-moving area, with GPT-4o’s vision capability being one practical entry point.
Metadata filtering
Sometimes you do not want to search the entire document corpus. A user asking about their specific lease agreement should not get results from someone else’s contract. Metadata filtering lets you attach structured properties to each chunk, such as document ID, date range, jurisdiction, or contract type, and filter the vector search to only consider relevant chunks before running similarity matching. FAISS supports this through its metadata-aware wrappers in LangChain.
Giving it a web interface
Once your terminal chatbot is working, the natural next step is a browser-based UI. Streamlit is the fastest path for Python developers. A few dozen lines of code turn a command-line script into a browser-based tool with a file uploader, a chat window, and session state. The Streamlit documentation has a chat elements guide that fits directly onto this kind of application.
Gradio is another option, with a built-in ChatInterface component that wraps LLM chat apps with almost no configuration. For teams that need something more production-grade, FastAPI as a backend with any modern frontend framework gives full control over the architecture.
The bigger picture
RAG matters because it solves a structural problem with language models, not a surface-level one. The model’s reasoning capability is not the weak point. The knowledge access is. By separating what the model can reason about from what the model was trained on, RAG makes it possible to build AI systems that are both highly capable and reliably grounded in specific, controlled data.
Every enterprise legal tool, every internal knowledge base chatbot, every document review assistant being built right now uses some version of this architecture. The specifics vary: the vector database, the embedding model, the chunking strategy, the retrieval method. But the core idea does not change.
You have now built one. Your legal Q&A assistant can read any PDF you hand it and answer questions with citations drawn strictly from the document’s content. That is a genuine, deployable capability. Point it at a different document type: a technical specification, a research paper, an HR policy archive. It works just as well, because the pipeline does not care what kind of text it is indexing.
The interesting work starts now: experimenting with chunk sizes, adding reranking, wiring up a web interface, connecting multiple documents as a corpus. Each of those directions builds directly on what you have already written.
