RAG exists to solve one problem: language models don’t know your private, fresh, or proprietary data. It fixes that by retrieving relevant context at query time and handing it to the model to ground its answer. Let’s build one from scratch.
The two halves of every RAG system
Indexing (offline): load, clean, chunk, embed, store. Querying (online): embed the query, retrieve, rerank, assemble the prompt, generate, cite. Get those two pipelines right and everything else is tuning.
Indexing: turning documents into searchable meaning
Load and clean your sources, strip boilerplate, and OCR where needed. Chunk content into passages of roughly 256 to 1024 tokens with 10 to 20 percent overlap. Embed each chunk into a vector. Then store the vectors plus metadata (source, tenant, timestamp) in a vector database.
Querying: finding the right context fast
Embed the query the same way you embedded chunks. Retrieve the top candidates using hybrid search: dense vectors for meaning plus sparse BM25 for exact terms and IDs. Rerank with a cross-encoder for much higher precision. Assemble the prompt from system instructions, retrieved context, and the question, then generate and cite, instructing the model to answer only from the context.
The decisions that actually matter
Chunk size and overlap is the biggest lever on retrieval quality. Hybrid weighting balances semantic versus keyword matches. k and reranking trade precision against latency and cost. Vector DB choice is managed versus self-hosted, like pgvector, Qdrant, or Weaviate. And freshness, how you handle updated or deleted documents, keeps answers correct.
Killing hallucinations
Improve retrieval quality, instruct the model to say ‘I don’t know’ when context is weak, require citations, and add a groundedness check that verifies claims against retrieved text before returning.
How you’d be graded in an interview
A strong answer separates retrieval metrics (recall@k, MRR, nDCG) from generation metrics (faithfulness, answer relevance, correctness), and proposes an eval set for both. Mentioning multi-tenancy and access control at retrieval time turns a good answer into a great one.
Want to test yourself on this? Refer 3 friends to unlock our Interview Question Bank: 40 Gen AI system design questions with answer focus. Your share link is at the bottom of this email.