RAG Architecture

Large language models can answer questions, summarize content, and generate useful text, but their internal knowledge may be outdated, incomplete, or unrelated to an organization's private information. They may also produce plausible statements without reliable supporting evidence.

Retrieval-Augmented Generation, or RAG, addresses this limitation by searching approved external knowledge before generation. The model receives the user's request together with relevant evidence and is instructed to answer from that context.

What Is RAG Architecture?

RAG is an AI design pattern that combines retrieval and generation. Retrieval identifies useful information from an external source; generation uses that evidence to create a relevant response.

Knowledge remains in documents, databases, APIs, or search indexes that can be updated independently of the model. RAG does not teach the model permanently; it supplies information for the current request.

When RAG Is Useful

• Answers must use current policies, manuals, products, or operational information.
• Users need access to private, domain-specific, or organization-owned knowledge.
• Responses should include inspectable supporting sources.
• Knowledge changes more frequently than model retraining would be practical.
• The task is primarily finding and explaining existing information rather than learning a new behavior.

RAG is not automatically the right choice for every task. Structured queries may be better for exact database facts, tools may be required for actions, and fine-tuning may be more suitable for changing model behavior or output style.

Why Is RAG Important?

• Provides access to recent and private information
• Supports answers grounded in visible source material
• Allows knowledge updates without retraining the language model
• Can reduce unsupported claims when retrieval and generation are well controlled
• Preserves document permissions when authorization is designed correctly
• Enables focused assistants for enterprise and domain-specific workflows

RAG reduces some causes of hallucination but cannot eliminate them. Retrieval may miss the right evidence, retrieve misleading evidence, or provide correct evidence that the model misinterprets. The complete system must be evaluated.

Two Main RAG Pipelines

Offline Knowledge Ingestion

Approved sources -> Connectors -> Parse/OCR -> Clean -> Chunk
-> Add metadata and permissions -> Embed/index -> Search store
-> Quality checks, lineage, freshness, and deletion

Online Question Answering

User question -> Authenticate -> Understand/rewrite query -> Retrieve
-> Permission filter -> Rerank -> Build context -> LLM -> Validate
-> Answer with sources or safe fallback -> Logs, metrics, and feedback

Core Components

Source Systems

Knowledge can come from document repositories, manuals, websites, wikis, support systems, databases, APIs, and file uploads. Each source needs an owner, approved purpose, access rules, update schedule, retention policy, and deletion path.

Connectors and Parsing

Connectors detect source changes and collect authorized content. Parsers extract text, headings, tables, links, page numbers, and other structure from formats such as HTML, PDF, office documents, and scanned images. Parsing quality sets an upper limit on retrieval quality.

Chunking

Large documents are divided into retrievable units. Chunks should preserve meaningful boundaries and enough surrounding context. Very small chunks may lose meaning, while very large chunks add irrelevant text and consume the context budget.

Metadata and Permissions

Attach stable source identifiers, title, section, location, owner, dates, language, version, tags, sensitivity, and access-control metadata. Permission filters must be enforced before content enters the model context.

Embeddings and Indexes

Embedding models convert text into vectors that represent semantic relationships. Vector indexes support similarity search, while keyword or lexical indexes are strong for exact names, codes, dates, and uncommon terms. Many systems combine both.

Retriever and Reranker

The retriever produces candidates using vector, keyword, structured, or hybrid search. A reranker can then score the smaller candidate set more precisely. Filters, diversity, recency, and neighboring context may improve the final evidence set.

Context Builder

The context builder removes duplicates, orders evidence, respects token limits, labels sources, and combines it with trusted instructions. It should never treat instructions found inside retrieved content as trusted system policy.

Generator and Output Controls

The language model receives the question and selected evidence. Instructions should require supported answers, explicit uncertainty, and a fallback when evidence is insufficient. Output validation can enforce structure, safety, citations, and business rules.

Retrieval Strategies

The best approach depends on the corpus and question patterns. Evaluate retrieval with representative queries instead of selecting a vector database or embedding model by popularity.

Chunking and Context Design

• Prefer semantic boundaries such as headings, paragraphs, procedures, and table sections.
• Keep parent document and section metadata with every chunk.
• Use overlap only when it improves measured retrieval; excessive overlap increases duplication and cost.
• Handle tables, code, lists, images, and scanned content with format-aware processing.
• Retrieve adjacent or parent context when a matched chunk is incomplete.
• Fit evidence within a controlled context budget and remove low-value duplicates.

Citations and Grounded Answers

Citations help users inspect evidence, but a source link alone does not prove that every statement is supported. Use stable source identifiers and locations, preserve the exact evidence supplied to the model, and evaluate whether claims are entailed by the cited passages.

• Show document titles and relevant locations such as page or section.
• Link only to sources the current user is authorized to open.
• Distinguish direct evidence from model interpretation.
• Avoid answering when sources conflict or evidence is insufficient.
• Offer a review or search fallback for consequential questions.

Security and Privacy

A RAG system can expose sensitive information if it retrieves across tenant or role boundaries. It can also ingest malicious instructions hidden in documents. Security must cover both ingestion and query time.

Evaluating a RAG System

Evaluate individual stages and the end-to-end user outcome. A strong generator cannot repair missing evidence, and excellent retrieval does not guarantee a faithful answer.

Build a versioned evaluation set from real question patterns, including answerable, unanswerable, ambiguous, adversarial, permission-sensitive, and time-sensitive cases. Review failures by category.

A Simple Analogy

Imagine asking a librarian a question. Instead of relying only on memory, the librarian clarifies the request, checks which materials you may access, searches the catalog, selects relevant pages, compares the evidence, and explains the answer while pointing to the sources.

A RAG system follows a similar pattern. Its quality depends not only on the explanation, but also on cataloging, permissions, search strategy, source quality, and the decision to say that no reliable answer was found.

Python Example

This small example demonstrates keyword retrieval and a grounded fallback. Real systems use stronger search, authorization filters, an LLM, citations, observability, and extensive evaluation.

documents = [
    {"id": "refund-policy", "text": "Customers may request a refund within 30 days."},
    {"id": "shipping-policy", "text": "Standard shipping takes three to five days."},
]

def retrieve(question: str):
    terms = set(question.lower().split())
    ranked = sorted(
        documents,
        key=lambda doc: len(terms & set(doc["text"].lower().split())),
        reverse=True,
    )
    return ranked[0] if terms & set(ranked[0]["text"].lower().split()) else None

question = "What is the refund policy?"
source = retrieve(question)

if source:
    print({"answer": source["text"], "source": source["id"]})
else:
    print({"answer": "I could not find supporting information.", "source": None})

Common Failure Modes

Scaling and Reliability

• Process ingestion asynchronously with durable queues and idempotent jobs.
• Update changed documents incrementally rather than rebuilding everything.
• Partition indexes by tenant, domain, geography, or policy where appropriate.
• Cache approved queries or retrieval results without crossing permission boundaries.
• Use timeouts, bounded retries, circuit breakers, and graceful no-answer behavior.
• Monitor index freshness, queue depth, embedding failures, retrieval latency, and model limits.
• Maintain deletion workflows that remove content from source, chunks, indexes, caches, and backups according to policy.

Best Practices

• Begin with representative user questions, trusted sources, and measurable acceptance criteria.
• Assign owners and lifecycle policies to every knowledge source.
• Preserve document structure, metadata, lineage, versions, and permissions during ingestion.
• Choose chunking, embedding, search, and ranking strategies through evaluation.
• Apply authorization before retrieval and preserve tenant boundaries everywhere.
• Treat retrieved content as untrusted data and defend against prompt injection.
• Require answers to use supplied evidence, show sources, and abstain when support is insufficient.
• Evaluate ingestion, retrieval, generation, safety, service, user, and cost outcomes separately and end to end.
• Monitor freshness and real questions, categorize failures, and improve the weakest stage.
• Document model, prompt, embedding, index, corpus, configuration, owners, and fallback versions.