RAG is not enough: retrieval architecture in 2026

Every failed AI feature we have been asked to rescue had the same architecture: chunk the docs, embed them, cosine similarity, stuff the top five into a prompt.

Why it plateaus Embeddings capture topical similarity, not precision. Ask for an invoice number, a SKU, or a clause reference and dense vectors will happily hand you five documents about the right topic and the wrong record.

What we replace it with - Hybrid search: BM25 plus dense vectors, fused with reciprocal rank fusion - A cross-encoder reranker over the top 50 candidates - Query rewriting so conversational follow-ups become standalone queries - Metadata filters applied before search, not after

Chunking is a product decision Chunk on document structure — sections, clauses, table rows — not a fixed token count. Carry the parent heading into every chunk so retrieved fragments keep their context.

Measure retrieval separately Track recall@k on the retrieval layer independently of answer quality. If the right chunk is not in the context window, no amount of prompt engineering saves the answer.