Building RAG Systems That Actually Work

Retrieval-Augmented Generation (RAG) is now the default pattern for grounding language models in organisational knowledge. But most implementations fail quietly — they pass demos and fall apart in production. Here's what actually matters.

Chunking is 80% of the work

The most common mistake is treating chunking as an afterthought. If your chunks are too large, the retriever returns bloated context and the LLM loses the needle in the haystack. If they're too small, you lose coherence.

What works:

• Semantic chunking over fixed-size windows
• Preserve paragraph boundaries — don't split mid-sentence
• Overlap 10–15% between chunks to preserve context at boundaries
• For structured data (tables, forms), extract and index separately

Retrieval quality > model quality

Swapping GPT-3.5 for GPT-4 on a bad retriever won't fix your answers. Invest in retrieval first:

• Hybrid search (dense vector + BM25 keyword) consistently beats either alone
• Re-ranking with a cross-encoder after initial retrieval narrows the field
• Always evaluate retrieval quality with a golden set before evaluating answer quality

Grounding and citation

A RAG system without citations is a hallucination risk. Force the model to cite:

Answer only from the provided context. If the context does not contain enough 
information to answer, say so. Cite the source document for each claim.

With rehype-pretty-code, code blocks are highlighted automatically. The investment in citation structure pays back in user trust and debuggability.

The honest answer on when to use RAG

RAG is the right tool when the corpus changes frequently and fine-tuning is too expensive to run every update cycle. For static knowledge bases that change quarterly, fine-tuning is often cheaper and gives better accuracy. Evaluate both before committing.