RAG System Design Interview: The Questions Behind the Diagram

Anyone can draw “documents → embeddings → vector DB → LLM.” Here’s what interviewers ask next.

Arvind Kumar

6 min read

·

Oct 23, 2026

--

Listen

Share

Retrieval-augmented generation (RAG) means the model looks things up before it answers, instead of relying only on what it learned in training. It’s behind most “chat with your docs” features, and it has become a standard system design interview question.

The basic diagram takes about two minutes to draw:

documents → chunk → embed → vector database
                                   ↓
user question → embed → search → top-k chunks → LLM → answer

Every candidate draws this. The interview is decided by the follow-up questions. Here are the ones that come up most, what each one is really testing, and what a strong answer includes.

Full story for non-members | Instagram-@arvind.codefarm | E-Books on Java/Microservices/Springboot | Whatsapp Group

Press enter or click to view image in full size

Running example: an internal assistant that answers employee questions using 200,000 company documents: policies, wikis, runbooks, and PDFs.

Q1: “How do you chunk the documents?”