RAG: the technique that makes LLMs reliable in enterprise

Introduction

An LLM (Large Language Model) like GPT, Claude, or Gemini knows nothing about your internal documents. It was trained on the public web, not on your emails, Notion notes, or confidential reports. RAG (Retrieval-Augmented Generation) fills this gap without retraining the model. It's the technique that all serious organizations use to connect an LLM to their private data, and it's also the one that fails most silently in production when implemented too naively.

This article details how RAG actually works: how to transform documents into searchable objects, how to find the right passages, and why the "tutorial" version of this architecture doesn't hold up against real users.

RAG in brief: why this technique is becoming standard

Two concepts to master before going further:

• An LLM predicts the next word from input text. Nothing more, nothing less.
• A vector (or embedding) is a list of numbers that represents the meaning of text. Two texts similar in meaning have mathematically close vectors.

Before RAG, two options existed for exploiting private data: fine-tuning (retraining a model, costly and rigid) or simply having no business context whatsoever. RAG offers a third way: dynamically inject relevant passages retrieved on-the-fly from a document repository into the prompt.

This architecture is at the core of features like Azure OpenAI On Your Data, Microsoft Graph connectors used by Copilot, or solutions built with LangChain or LlamaIndex.

Indexing: transforming documents into searchable vectors

Indexing is the work done once, upstream, to make a document repository queryable.

Chunking (splitting)

Embedding models have an input size limit (often a few hundred to a few thousand tokens). Documents must therefore be split into pieces, or chunks:

• Too small (around 50 words): the chunk loses its context and becomes an isolated sentence without inherent meaning.
• Too large (several thousand words): the resulting vector mixes too many different ideas, making search imprecise.
• Practical size: between 500 and 1,000 words per chunk, split on natural boundaries (end of paragraph, section).

A common technique is to add overlap between consecutive chunks, for example one chunk from 0 to 500 words, the next from 450 to 950 words. This prevents an idea spanning a boundary from being lost.

1{
2  "chunk_size": 800,
3  "chunk_overlap": 100,
4  "splitter": "recursive_character"
5}

More sophisticated approaches exist, such as semantic chunking, which splits not on a fixed length but when the subject actually changes in the text.

Transformation into embeddings

An embedding model (such as OpenAI's text-embedding-3 or BGE as open source) doesn't generate text: it takes a chunk as input and produces a vector, typically 1,536 dimensions for text-embedding-3-small. Each dimension captures an abstract facet of meaning (topic, tone, technicality, etc.).

1from openai import OpenAI
2client = OpenAI()
3 
4response = client.embeddings.create(
5    model="text-embedding-3-small",
6    input=chunk_text
7)
8vector = response.data[0].embedding  # list of 1536 floats

Storage in a vector database

Vectors are stored in a vector database optimized for quickly finding vectors close to another vector: Pinecone, Chroma, Weaviate, or in the Microsoft ecosystem, Azure AI Search with its native support for vector search.

Vector search: finding the right passages

Once the user's question is transformed into a vector (using exactly the same embedding model as the one used to index documents), the closest chunks are found using k-nearest neighbors (k-NN) search, where k is typically 3 to 5.

The most commonly used proximity measure is cosine similarity: instead of measuring distance, you measure the angle between two vectors. A score close to 1 means similar texts, close to 0 means unrelated texts, close to -1 means opposite texts.

Answer generation: the augmented prompt

Once relevant chunks are retrieved, they are injected into a structured prompt:

1Answer the following question based only on the context below.
2 
3Context:
4{retrieved chunks}
5 
6Question: {user question}

The complete flow of vanilla RAG (or basic RAG) consists of six steps:

1. The user asks a question.
2. The question is vectorized.
3. The closest chunks are searched for in the vector database.
4. Chunks are inserted into a prompt.
5. The prompt is sent to the LLM.
6. The generated answer is returned.

This architecture works technically, but it regularly fails once it leaves the demo context.

Why basic RAG fails in production

Three breaking points appear systematically:

How to make RAG reliable: query translation, reranking, and self-correction

Reformulating the question before searching

Query translation consists of having the LLM rewrite the question from multiple angles before searching (multiquery technique), then combining the results of each reformulation.

A more efficient variant is called HyDE (Hypothetical Document Embedding). The LLM invents a hypothetical answer to the question, even if incorrect. This false answer is never shown to the user: it only serves as a search key, because its length and vocabulary resemble a real document more than a five-word question. Cosine similarity search becomes much more precise.

Reranking with a cross-encoder

Embedding-based search is fast but coarse: question and document are vectorized separately, never read together. Reranking corrects this in two passes:

1. Classic embedding-based search: retrieve 50 candidates from millions of documents.
2. A cross-encoder reads the question and each candidate simultaneously, word by word, and assigns a fine relevance score. Only the 3 to 5 best are kept.

This cascade costs more computationally, but it greatly reduces noise in the final context sent to the LLM.

Corrective RAG (Corrective RAG / Self-RAG)

Corrective RAG, also called adaptive RAG, transforms the linear chain into a verification loop:

• Are the retrieved documents relevant? If not, reformulate the question or switch to web search.
• Is the generated answer truly anchored in the provided documents, or is the model hallucinating? If so, regenerate.

This approach is documented in widely cited research papers, notably Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection and Corrective Retrieval Augmented Generation. It forms the basis of RAG architectures deployed in production by teams dealing with real user volumes.

Vanilla RAG versus production RAG

For further technical details, the official documentation on Azure AI Search vector search and the Azure OpenAI On Your Data guide detail the mechanisms for ingestion, chunking, and hybrid search available natively in the Microsoft ecosystem.

Key takeaways

• RAG relies on three stages: indexing (chunking, embedding, vector storage), retrieval (k-NN, cosine similarity), and generation (augmented prompt).
• A single embedding model must be used end-to-end, for both indexing and search.
• Vanilla RAG fails at three specific points: poorly formulated questions, non-relevant chunks, LLM hallucinations.
• Proven fixes are HyDE/multiquery for reformulation, reranking with cross-encoder for precision, and corrective RAG for self-verification.
• Before any production deployment, test system behavior on real ambiguous questions, not just well-formulated demo questions.

If your organization is considering a RAG project on Azure, start by validating the Azure AI Search service tier required for vector search, then measure retrieval quality before adding reranking and correction layers.