Retrieval-Augmented Generation has become the default pattern for grounding LLM responses in domain-specific knowledge. But there is a significant gap between the "hello world" RAG demo and a production system that handles real-world documents, user queries, and accuracy requirements.
Over the past year I have deployed RAG pipelines for several enterprise clients — a legal-tech platform processing 500k+ case documents, a healthcare knowledge base serving clinicians, and an internal IT support bot for a 10,000-person organisation. Here is what I wish I had known at the start.
Chunking Is Everything
The single biggest lever for RAG quality is your chunking strategy. Naive fixed-size chunking (512 tokens with 50-token overlap) works surprisingly well for homogeneous text, but real-world documents are heterogeneous: they contain tables, lists, headers, code blocks, and metadata that naive splitting destroys.
What worked: A hybrid approach combining semantic chunking (splitting on topic boundaries using embedding similarity) with structural awareness (respecting markdown headers, table boundaries, and list groupings). This improved our retrieval precision by 34% over fixed-size chunks.
Embedding Model Selection Matters More Than You Think
Do not default to OpenAI's text-embedding-ada-002 without benchmarking alternatives. For domain-specific corpora, fine-tuned or specialised embedding models often dramatically outperform general-purpose ones.
Evaluation Is Non-Negotiable
You cannot improve what you do not measure. We built an evaluation harness that runs nightly against a golden dataset of 200 question-answer pairs, tracking retrieval precision, answer accuracy, and hallucination rate. This caught two regressions that would have shipped to production.
Conclusion
RAG is not a solved problem — it is an engineering discipline. The teams that treat it with the same rigour as traditional software engineering (testing, monitoring, iterating) will build systems that actually work.
