Every model has a knowledge cutoff and no idea what's in your internal wiki, your support tickets, or last week's sales data. Retrieval-augmented generation (RAG) fixes that by fetching relevant documents at query time and feeding them to the model as context, instead of hoping the answer was baked into training.
The pipeline, in four steps
- Chunk your documents into passages small enough to be useful and large enough to keep context.
- Embed each chunk into a vector and store it in a vector database.
- At query time, embed the user's question and retrieve the nearest chunks.
- Pass those chunks to the model alongside the question, and generate an answer grounded in them.
That's the whole idea. The engineering difficulty lives in the details: chunking strategy, hybrid search (keyword plus vector), re-ranking, and knowing when retrieved context is actually relevant versus noise that drags the answer off course.
When RAG is the wrong tool
If your knowledge base is small enough to fit in a long-context prompt, you may not need retrieval at all — just paste it in. And if the task is reasoning over structured data rather than unstructured documents, a SQL or API call beats a vector search every time. RAG earns its complexity when the corpus is large, changes often, and the answers genuinely depend on specific passages rather than general knowledge.
The best RAG systems we've shipped spend more engineering time on retrieval quality than on the model itself.
If you're evaluating whether RAG fits a problem you have, that's exactly the kind of question our AI Opportunity Assessment is built to answer.