RAG in depth
How retrieval-augmented generation is actually built: chunking, hybrid search, reranking, citation duty, and the measurement without which nobody knows if it works.
The idea
Instead of asking a model from memory, you first search your own documents for relevant passages and attach them to the question. The model then answers on the basis of those passages and can cite them.
The gain is not better language but verifiability.
The flow
- 01
Chunk
Cut documents into passages that make sense on their own.
- 02
Embed
Convert each passage into a vector and store it.
- 03
Search
Retrieve the most similar passages for the question, usually twenty to fifty.
- 04
Rerank
A more accurate model puts the candidates into a better order.
- 05
Answer
Give the best five to ten passages plus the question to the model, instructed to answer only from them and to cite.
Chunking
| Strategy | When |
|---|---|
| Fixed length with overlap | Simple, a good starting point |
| At headings | Structured documents, manuals, statutes |
| At meaning shifts | Prose with no structure |
| Load the parent chunk | Search small chunks, supply the larger context |
The last is often the best in practice: search with small precise chunks and hand the model the surrounding larger passage.
Metadata is not decoration
- Store source, version date, validity period and permission per chunk.
- Filter by permission during the search, not after. Otherwise too few results come back.
- Exclude or clearly mark superseded versions. A system quoting an expired policy is worse than none.
Separate measurement
A system with Recall@10 = 0.62 and faithfulness of 0.95 has no model problem
but a search problem. Any work on the prompt is wasted there.
The evaluation set
It is the one investment that pays off in every RAG project.
- 100 to 300 real questions from operations, not invented ones.
- For each question the identifier of the passage that answers it.
- Deliberately included questions whose answer is not in the corpus. The system must say it does not have the answer.
- Questions with several correct sources, to test completeness.
- The set is versioned and rerun on every change to the system.
Enforcing citation
The instruction "cite the source" alone is not enough. It becomes reliable only with a check after generation:
- 01
Pass numbered passages
Every passage gets an identifier visible in the prompt.
- 02
Demand structured output
Answer and the list of identifiers used, separately, as JSON.
- 03
Verify
Hold every statement in the answer against the cited passages, automatically or with a second model.
- 04
Refuse when unsupported
Better no answer than an unsupported one. The refusal rate is tracked as a metric.
Cost per request
| Item | Tokens | Share |
|---|---|---|
| System prompt | 400 | 4 % |
| Eight passages of 600 tokens | 4,800 | 52 % |
| Question and history | 300 | 3 % |
| Answer | 400 | 4 % |
| Total without caching | 5,900 |
The system prompt and frequently supplied passages can be reused via prompt caching, which in practice saves 30 to 60 percent of the input cost. See Speeding up inference.
An evaluation set for retrieval
FREE ACCOUNT
Evaluation set and scoring for RAG
A schema for the evaluation set and a script that measures recall, faithfulness and the refusal rate separately.
Access code2 items
Related courses and sources
Hugging Face agents course
Tool calls, planning, and the safeguards without which an agent is not viable in operation. Hands-on, with runnable code.
For anyone building an agent who needs to know which safeguards belong with it.
LangChain documentation
Building blocks for retrieval, tool calls and agents. Useful as a catalogue of the patterns, even if you end up building without the framework.
Useful as a catalogue of patterns, even if you end up building without the framework.
OWASP Top 10 for LLM applications
The list of weaknesses that actually occur in systems built on language models, from prompt injection to insecure tool integration.
For anyone building. The list replaces no review, but it is the best starting point for one.
Retrieval-Augmented Generation
The paper that joined retrieval and generation. The origin of the pattern that makes your own documents usable with citations.
For anyone making their own documents usable; the origin of the pattern.
Short courses on AI tooling
Units of around an hour on prompting, retrieval over your own documents, agents and evaluation. Free, and close to what is actually being deployed.
For practitioners with a specific question; a unit takes about an hour.