Retrieval-augmented generation is deceptively easy to prototype. Embed some documents, store the vectors, embed the query, take the top five matches, stuff them in a prompt. It works impressively well on a curated document set and a handful of questions you wrote yourself. Then real users arrive, ask things you never anticipated, and the same system starts confidently answering from the wrong document, or missing an answer that is plainly sitting in the corpus.
Almost every RAG failure in production is a retrieval failure, not a generation failure. The model is usually capable of answering correctly if it is given the right context. This article is about the patterns that reliably get it that context — and the trade-offs each one carries.
Why Naive Vector Search Falls Over
Dense embeddings capture semantic similarity. That is genuinely useful and genuinely insufficient, for a few structural reasons.
Exact terms get washed out. Embeddings are lossy compressions of meaning. A query for error code SR-4419 or part number MX2200-B may retrieve documents about similar-sounding codes because the embedding has no special respect for exact string identity. In technical corpora this is a constant source of wrong answers.
Similarity is not relevance. A chunk can be topically close to the query while containing none of the information needed to answer it. Top-k by cosine distance optimizes for the former and only incidentally delivers the latter.
Chunks lose their context. A paragraph that reads “This limit does not apply to enterprise accounts” is meaningless without knowing which limit. Split naively, that chunk will be retrieved for the wrong questions and will actively mislead the model.
Recency and permissions are invisible. Pure vector search has no idea that one document is the superseded 2023 policy and another is the current one, or that the asking user is not allowed to see either.
Each pattern below addresses one of these.
Chunking: The Decision That Constrains Everything Downstream
Chunking is the highest-leverage and least glamorous part of a RAG system. Get it wrong and no amount of reranking rescues you.
Chunk on Structure, Not Character Count
Fixed-size chunking with a character window is the default in most tutorials and the wrong default for most real corpora. Documents have structure — headings, sections, list items, table rows — and that structure usually corresponds to semantic boundaries. Splitting on headings first, then subdividing oversized sections, produces chunks that stand alone far better than arbitrary 1000-character windows.
For structured or semi-structured content the right unit is often obvious: one chunk per FAQ entry, per product spec, per policy clause, per API endpoint. When the unit is obvious, use it.
Attach Context to Every Chunk
The single most effective cheap improvement to retrieval quality is prepending contextual headers to each chunk before embedding it: document title, section path, effective date, document type. A chunk that begins “Refund Policy > Enterprise Accounts > Exceptions (effective 2025-06-01)” embeds into a very different neighborhood than the same paragraph alone, and it retrieves far more precisely.
A related technique is to generate a short, model-written summary of what each chunk is about in the context of its parent document, and embed that alongside the raw text. It costs one inexpensive model call per chunk at indexing time and meaningfully improves recall on corpora where individual paragraphs are ambiguous.
Decouple What You Embed From What You Return
There is no rule that the embedded text and the text handed to the model must be identical. A common and effective pattern is small-to-big: embed small, precise chunks for accurate matching, but when a chunk is retrieved, return the surrounding parent section to the model. Retrieval gets precision; generation gets context. The cost is a slightly more complex index.
Retrieval: Hybrid, Then Rerank
Combine Dense and Sparse
Running both a vector search and a keyword search (BM25 or equivalent) and merging the results addresses the exact-term weakness directly. Sparse retrieval nails identifiers, error codes, product names, and rare terminology. Dense retrieval handles paraphrase and conceptual similarity. Reciprocal rank fusion is a simple, tunable way to merge the two ranked lists without having to calibrate scores across different scales.
If you implement one thing beyond naive vector search, make it this. The engineering cost is modest — most managed search services support both — and it removes an entire class of embarrassing misses.
Retrieve Wide, Then Rerank Narrow
First-stage retrieval should be optimized for recall, not precision. Pull a generous candidate set — dozens of chunks, not five — then use a cross-encoder reranker to score each candidate against the query directly and keep only the best handful.
The reason this works is architectural. Bi-encoders (what vector search uses) embed the query and document separately, so the model never sees them together and cannot reason about their interaction. Cross-encoders process the pair jointly and are substantially better judges of relevance. They are also far too slow to run over an entire corpus, which is exactly why the two-stage design exists.
Reranking adds latency and cost per query. It is usually worth it, and it is the second thing to add after hybrid search.
Filter on Metadata Before You Rank
Permissions, tenancy, document status, date ranges, and language should be pre-filters applied at query time, not post-hoc filters applied to results. Filtering after retrieval means your top-k is polluted with documents the user cannot see, and you silently lose recall. It also means an access-control bug becomes a data leak rather than an empty result.
Design the metadata schema at ingestion time. Retrofitting tenant isolation onto a vector index that was built without it is painful and, in multi-tenant products, occasionally alarming.
Rewrite the Query When the Input Warrants It
Users ask short, underspecified, context-dependent questions — especially in a chat interface, where “what about for annual plans?” depends entirely on the previous turn. A cheap model call that rewrites the user’s message into a standalone, retrieval-ready query resolves pronouns, expands abbreviations, and can split a compound question into multiple searches.
Use this selectively. Rewriting adds a round trip, and on already-explicit queries it can make retrieval worse by drifting from the user’s actual wording. Gate it on conversation state or query length rather than running it unconditionally.
Generation: Grounding and Honest Refusal
Once the right chunks are in context, most of the remaining work is instruction design.
Require citations. Ask the model to attribute each claim to a specific retrieved chunk, and render those citations in the interface. This does two things: it gives users a way to verify, and it measurably reduces the model’s willingness to assert things the sources do not support.
Make “I don’t know” an acceptable answer. The default behavior of a helpful model given weak context is to synthesize something plausible. An explicit instruction to say when the retrieved material is insufficient — combined with a relevance threshold that skips generation entirely when nothing scores well — is more reliable than hoping the model self-censors.
Order context deliberately. Models attend unevenly across long contexts, and material at the very beginning and end tends to be weighted more heavily than material buried in the middle. Put the highest-scoring chunks at the edges, and keep the total context tight rather than padding it with marginal matches.
The distinction between retrieving facts and generating language is worth keeping clear in your own head as well as in the architecture — our comparison of generative AI and traditional AI covers why the two failure modes need different mitigations.
Evaluation: Measure Retrieval Separately
The most common RAG debugging mistake is treating the pipeline as one box. When an answer is wrong, the first question should always be: was the correct chunk in the retrieved context at all?
If it was not, the problem is retrieval — chunking, embedding model, hybrid weighting, filters. If it was, the problem is generation — prompt, context ordering, model choice. These have completely different fixes, and conflating them wastes weeks.
Build a Question-to-Chunk Test Set
Assemble a set of real questions paired with the chunk IDs that genuinely contain the answer. Then measure recall at your first-stage k (did we retrieve it at all) and precision after reranking (did it make the final cut). These two numbers tell you where to invest.
Generating this set is less work than it sounds: sample real queries from logs, retrieve a wide candidate set, and have a human or a model-assisted reviewer mark which chunks are actually sufficient.
Test the Corpus, Not Just the Questions
RAG quality is bounded by content quality. If the answer genuinely is not in the corpus, or exists in three contradictory versions from different years, retrieval cannot save you. A surprising share of RAG projects are really content-cleanup projects wearing a machine learning costume — deduplication, deprecating stale documents, and establishing document ownership often move accuracy more than any model change.
When RAG Is the Wrong Answer
RAG is not universal. Skip it when:
- The corpus is small and stable. A few thousand tokens of policy that rarely changes can go directly in the prompt. Retrieval adds infrastructure, latency, and failure modes for no benefit.
- The question is structured. “How many orders shipped late last quarter?” is a database query. Text-to-SQL over a governed schema, or a purpose-built API, will be more accurate and cheaper than semantic search over report documents.
- You need exhaustive coverage. Top-k retrieval is a sampling strategy. Questions requiring every matching record — compliance sweeps, full audits — need deterministic queries.
- Freshness must be immediate. Index lag is real. If the answer must reflect the state of a system a second ago, call the system.
Most mature deployments end up as hybrids: retrieval for prose and policy, tool calls for live and structured data, and a router deciding which path a given question takes. Getting that routing right is an AI and ML architecture problem more than a prompting one, and it usually depends on data pipelines and quality checks being in place first.
Conclusion
The gap between a RAG demo and a RAG product is almost entirely in retrieval engineering: chunk on structure and enrich with context, run hybrid search, rerank a wide candidate set, filter on metadata before ranking, and instruct the model to cite sources and admit uncertainty. Then measure retrieval separately from generation so you know which half to fix.
None of these are exotic techniques. They are ordinary information-retrieval discipline applied to a new generation layer — and they are what keeps the system standing once users start asking questions you did not write yourself.
