Retrieval-Augmented Generation: A Practical Architecture for Enterprise RAG
RAG is the most production-ready pattern for LLMs in the enterprise. Here's a battle-tested architecture with the gotchas we've learned.
What is retrieval-augmented generation?
Retrieval-Augmented Generation (RAG) is an architecture that enhances large language models with external knowledge. Instead of relying solely on the model's training data, a RAG system retrieves relevant documents from a knowledge base and includes them in the prompt context, producing answers that are grounded in your organisation's actual data.
For enterprises, RAG solves a critical problem: LLMs hallucinate. They produce confident but incorrect answers when they lack specific knowledge. By grounding responses in retrieved documents, RAG reduces hallucinations, enables citation of sources and makes AI systems trustworthy enough for production use.
The core components
1. Document ingestion and chunking
Your source documents — PDFs, wikis, databases — must be processed into chunks small enough for effective retrieval but large enough to preserve context. Chunk size, overlap strategy and splitting method all affect retrieval quality significantly.
2. Embedding generation
Each chunk is converted into a vector embedding using a model like OpenAI's text-embedding-ada-002. These embeddings capture semantic meaning, enabling similarity-based retrieval rather than keyword matching alone.
3. Vector store
Embeddings are stored in a vector database such as Pinecone, Weaviate or Azure AI Search. The vector store supports fast similarity search across millions of documents.
4. Retrieval and generation
When a user asks a question, the system embeds the query, retrieves the most similar chunks from the vector store and passes them to the LLM along with the user's question. The LLM generates a response grounded in the retrieved context.
A practical enterprise architecture
In production, a robust RAG system includes several additional components beyond the basics:
- Query preprocessing — rewriting user queries for better retrieval, including synonym expansion and intent classification
- Hybrid search — combining vector similarity with keyword search to improve recall across different query types
- Reranking — using a cross-encoder model to rerank retrieved chunks by relevance, significantly improving precision
- Source attribution — linking each claim in the response back to its source document for verifiability
- Guardrails — filtering out-of-scope queries and detecting attempts to extract sensitive information
Common pitfalls
Chunking strategy
Poor chunking is the most common cause of poor RAG performance. Chunks that are too small lose context. Chunks that are too large dilute relevance. The optimal chunk size depends on your document types — experiment with 500–1000 tokens and 10–20% overlap as a starting point.
Stale data
A RAG system is only as good as its knowledge base. Without a robust ingestion pipeline that keeps embeddings fresh, your system will retrieve outdated information. Implement incremental updates and track document freshness.
No evaluation framework
Teams often build RAG systems without a systematic way to measure retrieval quality. Use metrics like context precision and context recall to evaluate whether your system is retrieving the right documents. Without measurement, you're optimising blind.
Ignoring access control
Enterprise knowledge isn't universally accessible. Your RAG system must respect document-level permissions — a user asking a question should only receive answers grounded in documents they're authorised to see. This requires filtering at the retrieval stage, not just the generation stage.
When to use RAG
RAG excels when you need domain-specific answers grounded in your organisation's data — internal knowledge bases, policy documents, technical documentation and customer support. It's less suitable for tasks requiring complex multi-step reasoning over data, where fine-tuning or agentic approaches may perform better.
The key to a successful RAG implementation is iteration. Start simple, measure relentlessly and refine each component — chunking, retrieval, reranking — based on real query performance data.
