Case Studies

After learning about requirements, models, RAG, agents, cloud infrastructure, security, scalability, governance, and recovery, the next question is how those concepts work together in a real solution.

Case studies develop architectural judgment by connecting a business problem to design decisions and measurable outcomes. The examples in this lesson are realistic teaching scenarios rather than claims about a particular company's production system.

What Is an Architecture Case Study?

An AI architecture case study is a structured analysis of how a solution might be designed, delivered, operated, and improved for a defined problem and context.

• Business problem, users, workflow, and measurable baseline
• Functional and non-functional requirements
• Available data, quality, permissions, and ownership
• Architecture options and the selected design
• Model, retrieval, agent, infrastructure, and integration decisions
• Security, privacy, compliance, reliability, and human oversight
• Costs, risks, assumptions, limitations, and alternatives
• Evaluation, rollout, monitoring, support, and improvement results

A strong case study distinguishes measured facts from assumptions and proposed outcomes. It does not report success before a pilot or production evaluation provides evidence.

Why Study Case Studies?

• Connects individual technical concepts into complete systems
• Builds skill in requirements, trade-offs, and decision records
• Shows why different use cases require different architectures
• Reveals operational, security, cost, and human factors omitted by simple demos
• Provides practice for design reviews, interviews, and project proposals
• Encourages learning from failure modes and rejected alternatives

A Repeatable Review Framework

Case Study 1: Customer Support Assistant

Business Problem

A retailer receives many repetitive questions about shipping, returns, warranties, and order status. Customers wait for agents, while support staff spend time finding policy information across multiple documents and systems.

Requirements

• Answer policy questions from current approved sources with citations.
• Retrieve order information only for an authenticated customer.
• Escalate ambiguous, sensitive, complaint, refund-exception, or low-evidence cases.
• Meet a defined response-time and availability objective.
• Preserve conversation and action records according to policy.
• Measure resolution, answer support, escalation, satisfaction, latency, safety, and unit cost.

Proposed Architecture

Customer -> Web/chat channel -> API and identity -> Intent/risk routing
                                            |-> Policy RAG -> LLM -> Citations
                                            |-> Order tool with customer-scoped access
                                            |-> Human support queue

Approved documents -> Ingestion -> Permission metadata -> Search index
All paths -> Output controls, audit events, metrics, feedback, and support

Key Decisions and Trade-Offs

Evaluation and Failure Modes

• Build test cases for supported, unsupported, conflicting, outdated, adversarial, and permission-sensitive questions.
• Measure retrieval recall, groundedness, citation support, containment, escalation quality, P95 latency, and cost per resolved conversation.
• Test prompt injection in customer input and documents, cross-account order access, provider outage, stale policy ingestion, and agent handoff.
• Require the assistant to abstain when evidence is insufficient rather than invent a policy.

Case Study 2: Product Recommendation System

Business Problem

An online store wants to help customers discover relevant products without overwhelming them or promoting unavailable, inappropriate, or repetitive items.

Requirements

• Rank eligible products for a defined page and customer context.
• Respect consent, regional availability, inventory, age, and business constraints.
• Support new users and new products with limited interaction history.
• Return recommendations within the page latency budget.
• Avoid unacceptable concentration, repetition, and feedback-loop effects.
• Demonstrate incremental value through controlled experiments.

Proposed Architecture

Catalog + inventory + permitted interaction events
        |
Validated batch/stream pipelines -> Feature store -> Candidate generation
                                             |
Web/app request -> Context features -> Ranking model -> Eligibility rules
                                             |
                               Recommended items -> Events/experiments
                                             |
                         Quality, drift, latency, fairness, and revenue metrics

Key Decisions and Trade-Offs

Evaluation and Failure Modes

• Offline evaluation includes ranking quality, coverage, novelty, diversity, subgroup and catalog-segment behavior.
• Online evaluation includes engagement, conversion, margin, returns, satisfaction, latency, and guardrail metrics.
• Test stale inventory, feature delay, hot products, event duplication, cold start, feedback loops, manipulation, and model rollback.
• Monitor whether short-term clicks harm longer-term user experience or seller and product exposure.

Case Study 3: Internal Document Search Assistant

Business Problem

Employees spend too much time searching policies, project documents, and operating procedures across repositories. Existing keyword search struggles with natural-language questions, but content permissions must remain intact.

Requirements

• Answer from approved internal sources and display usable citations.
• Never retrieve content the employee cannot access in the source system.
• Show freshness, conflict, and uncertainty where relevant.
• Support document updates and deletion within defined objectives.
• Provide a conventional search fallback and source opening.
• Measure task completion and time saved, not only answer fluency.

Proposed Architecture

Document repositories -> Permission-aware connectors -> Parse/OCR -> Chunk
-> Metadata + ACLs -> Embedding/keyword indexes -> Freshness/deletion jobs

Employee + SSO -> Query service -> ACL filter -> Hybrid retrieval -> Rerank
-> Context builder -> LLM -> Grounded answer + authorized source links

Key Decisions and Trade-Offs

Evaluation and Failure Modes

• Evaluate parsing, retrieval recall, ranking, groundedness, citation support, permission isolation, freshness, and end-to-end latency.
• Include unanswerable, ambiguous, conflicting, multilingual, scanned, deleted, malicious, and permission-boundary questions.
• Monitor no-result rate, stale sources, index lag, source openings, user corrections, security violations, and cost per useful answer.
• Test prompt injection inside documents and prevent retrieved instructions from controlling tools or system policy.

Case Study 4: Invoice Processing Assistant

Business Problem

A finance team manually reads invoices, enters fields, matches purchase orders, and routes exceptions. The process is slow and transcription errors create rework, but incorrect payments would be consequential.

Requirements

• Extract supplier, invoice number, dates, currency, totals, tax, and line items.
• Match approved vendors and purchase orders through controlled systems.
• Detect duplicates and validate totals deterministically.
• Route low-confidence, conflicting, high-value, or policy-exception cases to finance staff.
• Never issue payment solely from unreviewed model output.
• Preserve source documents, decisions, approvals, and corrections for required records.

Proposed Architecture

Secure upload/email -> Malware scan -> Object storage -> Queue
-> OCR/document model -> Typed extraction -> Validation and duplicate checks
-> Vendor/PO read tools -> Confidence and policy routing -> Human review
-> Approved accounting-system draft -> Separate payment authorization

Audit trail, metrics, model versions, retention, and exception feedback across all stages

Key Decisions and Trade-Offs

Evaluation and Failure Modes

• Measure field-level precision and recall, total consistency, duplicate detection, review rate, correction time, throughput, and cost per accepted invoice.
• Evaluate by supplier, layout, language, scan quality, currency, table complexity, and value band.
• Test malicious attachments, prompt-like invoice text, duplicate uploads, missing purchase orders, service outages, retries, and access violations.
• Use idempotency and transaction state so retries cannot create duplicate accounting entries.

Architecture Comparison

The cases use different patterns because their output types, failure consequences, data, latency, and human workflows differ. Reusing platform capabilities is valuable, but forcing every problem into a chatbot or agent architecture is not.

Common Cross-Cutting Decisions

API or Self-Hosted Model

Compare measured capability, privacy, latency, traffic, total cost, model control, provider dependency, and team operations. The answer can differ by model task within one solution.

Synchronous or Asynchronous

Keep interactive paths within a clear latency budget. Put long, bursty, retryable work behind durable queues with status, cancellation, idempotency, and recovery.

Model Decision or Deterministic Rule

Use models where uncertainty and flexible interpretation are valuable. Use deterministic code for authorization, calculations, eligibility, limits, schemas, transaction integrity, and consequential policy enforcement.

Automation or Human Review

Base oversight on impact, uncertainty, reversibility, and available evidence. Human review must have suitable context, time, authority, workload, and escalation—not merely a button added for appearance.

From Pilot to Production

Architecture Decision Records

An Architecture Decision Record captures a significant choice so future teams understand its context and trade-offs. It should be concise and updated or superseded when assumptions change.

Decision: Use permission-aware hybrid retrieval for internal document search
Status: Approved for pilot
Context: Natural-language queries plus exact policy codes; strict document ACLs
Options: Keyword only, vector only, hybrid
Choice: Hybrid retrieval with ACL filtering before search and reranking
Consequences: Better expected coverage; higher tuning, latency, and operations
Evidence: Retrieval test set, permission-negative tests, latency benchmark
Review trigger: Corpus doubles, permissions model changes, or P95 exceeds target
Owner: Knowledge assistant product team

A Simple Analogy

Designing several houses begins with different occupants, land, climate, budget, accessibility, and safety needs. A small apartment, farmhouse, and automated smart home share foundations and utilities, but copying one plan into every context would fail.

AI architecture is similar. Patterns are reusable, but the final design must come from the actual requirements and evidence.

Python Decision-Support Example

This simplified example maps a known project type to a starting pattern. Real architecture selection requires requirements, alternatives, measurements, risk review, and stakeholder decisions rather than a single conditional.

ARCHITECTURE_PATTERNS = {
    "customer_support": "RAG + scoped tools + human escalation",
    "recommendation": "Candidate generation + ranking + eligibility rules",
    "document_search": "Permission-aware hybrid RAG + citations",
    "invoice_processing": "Async extraction + validation + human approval",
}

def starting_pattern(project_type: str) -> str:
    return ARCHITECTURE_PATTERNS.get(
        project_type,
        "Run requirements analysis before selecting a pattern",
    )

print(starting_pattern("customer_support"))

How to Critique a Case Study

• Separate facts, measured results, estimates, and assumptions.
• Ask whether AI is necessary and compare a simpler baseline.
• Check whether metrics represent real user and business outcomes.
• Look for missing data rights, security, fairness, accessibility, and affected-user analysis.
• Trace normal, failure, degraded, recovery, and retirement paths.
• Calculate total cost and human workload, not only model price.
• Challenge claims of accuracy, scale, reliability, or value without evidence.
• Identify who owns each decision, risk, system, dataset, and incident.
• Ask what would cause the team to stop, roll back, redesign, or retire the solution.

Common Challenges

• Case studies omit failed experiments, operational effort, or negative outcomes.
• Business contexts and constraints are simplified or missing.
• Several valid architectures exist and no single answer is universal.
• Vendor names are mistaken for architecture decisions.
• Offline model metrics are presented as business success.
• Security, governance, cost, support, and recovery appear only as afterthoughts.
• A successful pilot is assumed to work at production scale.
• Correlation is presented as causal impact without a suitable experiment.
• Readers copy a final diagram without understanding the assumptions.

Best Practices

• Begin with the user and business problem, baseline, outcome, owner, and constraints.
• Document functional, quality, security, privacy, reliability, cost, and human-workflow requirements.
• Compare at least one simple baseline and plausible architecture alternatives.
• Explain why each major component exists and what trade-off it introduces.
• Include data lineage, permissions, trust boundaries, failure modes, fallback, and recovery.
• Define evaluation before reporting outcomes and use representative real-world cases.
• Measure model, service, user, business, risk, and economic results together.
• Capture decisions, assumptions, rejected options, limitations, owners, and review triggers.
• Pilot with controlled exposure, release gradually, and learn from failures and user feedback.
• Adapt reusable patterns to the context instead of copying an architecture unchanged.