A RAG demo takes an afternoon. A RAG system that answers correctly against 200,000 messy internal documents, with permissions enforced and a latency budget, is a proper engineering project, and the gap between the two is where most teams stall. The demand side is not in question: Menlo Ventures found 51% of enterprise generative AI implementations use RAG, while Databricks measured vector database usage growing 377% in a single year.
What is a RAG system? Retrieval-augmented generation answers questions by first fetching relevant passages from your own documents, usually through vector database search, then passing those passages to a large language model as context. The model grounds its answer in your data rather than its training memory, which cuts hallucination and keeps answers current without retraining anything.
Retrieval sets your quality ceiling, not the model
The most common misdiagnosis in RAG system design is blaming the LLM for wrong answers when the retriever never surfaced the right passage. If the correct chunk is not in the context window, no model can recover it. The pipeline is nearly universal: parse documents, chunk them, embed and index the chunks with metadata, retrieve the nearest ones at query time, then rerank and generate. A 90% success rate at each of five stages leaves 59% end to end, so production RAG work is mostly measuring and fixing individual stages. The retriever defines what the system can possibly say; the model only phrases it.
Chunking strategy decides what your system can find
Chunking looks like preprocessing, but it fixes the unit of retrieval, which makes it a product decision. The classic failure: you split a financial report every 500 tokens and one chunk reads "the company's revenue grew by 3% over the previous quarter". Which company? Which quarter? The chunk lost its context, its embedding is ambiguous, and retrieval misses it. Anthropic's contextual retrieval work targets exactly this: prepend a short, LLM-generated context sentence to each chunk before embedding, and the top-20 retrieval failure rate falls 35% on that change alone.
A few rules hold up across corpora: split on headings and paragraphs before falling back to token counts, keep chunks in the few-hundred-token range with 10 to 20% overlap, and attach metadata such as source, timestamp and access permissions, since filtering and citations depend on it later. There is no universal best chunk size, only the best size for your corpus, which is why evaluation has to exist before you tune.
Hybrid search plus a reranker does the heavy lifting
Dense retrieval is the backbone of modern knowledge search: embed the query, run an approximate nearest neighbour search over an HNSW index, and you get semantically similar chunks even when the wording differs entirely. Its known blind spot is exact identifiers, since error codes, SKUs and invoice numbers get smeared into semantic mush by embeddings, while a plain BM25 keyword index matches them instantly. Serious systems therefore run hybrid search, vector and keyword retrieval in parallel with fused results, which engines like Weaviate support in a single query. The final layer is a reranker: retrieve wide, then let a cross-encoder score each candidate against the query and keep the best 20.
The chart above, from Anthropic's published results, shows the full stack: standard embeddings fail on 5.7% of retrievals, and layering contextual embeddings, contextual BM25 and reranking brings that to 1.9%, a 67% reduction. None of those steps involved a better LLM. Retrieval engineering beats model upgrades for search quality.
Permissions, freshness and evaluation are the real work
On infrastructure, the short version is that if you are under a few million vectors and already run Postgres, pgvector is the sensible starting point, and a dedicated engine such as Qdrant earns its place only when you measure a bottleneck. Two other problems consume half the engineering time in real deployments.
Permissions first. Your knowledge base has access controls; your vector index, by default, does not. If a compensation memo gets embedded and any employee's query can retrieve it, you have built a data leak with a chat interface. Document-level ACLs must travel with chunks as metadata, and every retrieval must filter on the caller's entitlements inside the database query. This is where RAG becomes genuine backend APIs work: authenticated retrieval endpoints, tenant isolation and audit logs.
Freshness is the second. Documents change, so you need incremental ingestion that re-embeds changed documents and deletes stale vectors. Then measure: build a golden set of 50 to 200 real questions paired with the passages that answer them, score retrieval with recall@k and MRR, score generation separately for faithfulness, and run the suite on every pipeline change. With the RAG market projected to grow 49.1% annually through 2030, an evaluation suite is the only safe way to adopt the tooling churn ahead.
FAQ
Do I need a dedicated vector database for RAG?
Not at first. Small corpora work fine with pgvector inside Postgres, and a dedicated engine earns its place only at millions of vectors, strict latency targets or heavy write throughput.
Is RAG better than fine-tuning for knowledge search?
For injecting factual knowledge, yes. RAG is cheaper, updatable in minutes and auditable, which is reflected in 51% enterprise adoption versus 9% for fine-tuning. Fine-tuning suits style and reasoning patterns, and mature systems often combine both.
Your first production RAG build, in order
Pick one high-value corpus, curate it, and collect 50 or more real questions as a golden evaluation set before tuning anything.
Chunk on document structure with overlap, and attach source, section, timestamp and permission metadata.
Start with pgvector on Postgres; adopt a dedicated engine only for measured scale or latency needs.
Ship hybrid search and a reranker from day one; exact-match queries are more common than you expect.
Enforce permissions inside the retrieval query, build incremental ingestion with proper deletes, and wire the evaluation suite into CI.
Treat RAG as a search engineering problem with an LLM at the end, not an LLM problem with search bolted on. BeyondPixl Studio designs and ships production RAG systems, from corpus strategy and retrieval architecture to evaluation pipelines. If your prototype has stalled short of production, talk to our engineering team about a retrieval quality audit.