Capstone Architecture Project

This project brings together requirement analysis, model selection, RAG, tools, cloud architecture, security, governance, compliance, scalability, performance, high availability, disaster recovery, evaluation, and cost estimation into one complete design.

You will act as the AI Solution Architect for a customer support assistant. The goal is not to list technologies or draw an attractive diagram; it is to create a traceable, evidence-driven design and explain why it is suitable for the business problem.

Project Scenario

A growing online retailer receives questions through its website and mobile application. Customers ask about shipping, returns, warranties, product information, and order status. Support agents manually search policy documents and business systems, causing long wait times and inconsistent answers.

The company wants an AI assistant that answers routine questions, retrieves current approved policies with citations, accesses customer-specific order information only after authentication, and transfers uncertain or exceptional cases to a human agent.

Business Goals

• Reduce customer waiting time for routine support questions.
• Improve the consistency and evidence behind policy answers.
• Allow support agents to focus on exceptions and complex cases.
• Maintain or improve customer satisfaction and resolution quality.
• Protect customer and company information.
• Keep cost per successfully resolved conversation sustainable.

These goals are hypotheses until validated against a baseline and pilot. Your design must state what will be measured and what result would justify continued investment.

Users and Stakeholders

Project Scope

In Scope for the Initial Release

• Answer approved shipping, return, warranty, and product-policy questions.
• Show citations that customers and agents can inspect.
• Retrieve authenticated order status through a read-only tool.
• Detect low-evidence, conflicting, sensitive, abusive, or exceptional requests.
• Transfer the conversation and relevant context to a support agent.
• Collect controlled user and agent feedback for evaluation.

Out of Scope for the Initial Release

• Automatically approving refunds or financial exceptions
• Changing delivery addresses, payment details, or account ownership
• Giving legal, medical, or financial advice
• Using unapproved public websites as policy sources
• Training provider models on customer conversations
• Autonomous actions without deterministic authorization and required approval

Step 1: Requirements Analysis

Create a requirements catalog with identifiers so later design choices, tests, and monitors can refer to the same requirement.

Replace illustrative targets with stakeholder-approved values. Record traffic, languages, regions, support hours, accessibility, retention, integration, provider, budget, and growth assumptions.

Step 2: Architecture Options

Recommended initial design: a RAG assistant with an authenticated read-only order-status tool and human escalation. Keep financial and account-changing actions outside the first release.

Step 3: High-Level Architecture

Customer web/mobile chat
        |
CDN/WAF -> API gateway -> Identity/session -> Conversation service
                                            |
                                  Intent and risk router
                                    |              |
                              Policy RAG      Order-status tool
                                    |              |
                           Context builder <- scoped customer data
                                    |
                          LLM -> Output controls -> Citation validation
                                    |
                    Customer response or human-support handoff

Approved documents -> Ingestion queue -> Parse/chunk/metadata/ACL -> Search indexes

Identity, secrets, policy, audit, metrics, traces, evaluation, cost, backups, and incident response span all layers.

Create a visual diagram for your final submission showing trust boundaries, data stores, network exposure, external providers, human systems, normal flow, degraded paths, and recovery environment.

Step 4: Data Flow Design

Policy Question Flow

Question -> Validate/minimize -> Retrieve authorized policy chunks -> Rerank
-> Build labeled context -> Generate -> Validate claims/citations/safety
-> Respond or abstain/escalate -> Safe telemetry and feedback

Order Status Flow

Authenticated customer -> Verify session and order relationship
-> Call read-only order API with scoped workload identity
-> Minimize fields -> Format status -> Respond -> Audit access

Knowledge Ingestion Flow

Approved repository -> Change event -> Scan/parse -> Structure-aware chunks
-> Source/version/owner/permission metadata -> Embeddings + keyword index
-> Quality checks -> Publish -> Freshness and deletion monitoring

For every flow, document source, format, purpose, owner, validation, sensitivity, transformation, destination, permissions, retention, deletion, lineage, and failure path.

Step 5: RAG Design

• Use only approved policy and product repositories with named owners.
• Preserve headings, tables, page or section location, stable source ID, version, and effective date.
• Choose chunking through retrieval evaluation rather than a fixed arbitrary size.
• Combine keyword and vector retrieval if representative questions benefit.
• Filter by region, product, language, effective date, and permissions before retrieval.
• Rerank a bounded candidate set and remove duplicates before generation.
• Require citations and a clear no-evidence or conflicting-evidence fallback.
• Treat documents as untrusted data and defend against embedded prompt injection.
• Define freshness and deletion objectives for source-to-index synchronization.

Step 6: Model Selection

Compare at least one simple or smaller candidate with a more capable model on the same versioned evaluation set. Document the decision, rejected alternatives, limitations, and review triggers.

Step 7: Security Architecture

Include a threat model, trust-boundary diagram, role and service permissions, data classification, secrets design, logging rules, security tests, incident kill switches, and residual-risk record.

Step 8: Governance and Compliance

• Assign business, product, system, data, model, security, privacy, and operational owners.
• Register the use case, models, providers, prompts, tools, data sources, and risk tier.
• Identify applicable legal, contractual, accessibility, privacy, records, and industry requirements with specialists.
• Trace each important obligation to a control, test, evidence, owner, and review trigger.
• Define human escalation, complaint, correction, and appeal where appropriate.
• Version architecture, data, model, prompt, policy, evaluation, approval, and notices.
• Time-limit exceptions and document accepted residual risk.
• Define material-change, incident, suspension, and retirement processes.

Step 9: Cloud and Infrastructure Design

Keep production and non-production accounts, identities, data, budgets, and access separate. Select provider-specific services only after comparing requirements, team skill, total cost, and exit needs.

Step 10: Scalability and Performance

• Estimate average, peak, growth, failure, region, and tenant traffic scenarios.
• Set P50, P95, and P99 end-to-end and first-token latency objectives.
• Scale application APIs, model endpoints, ingestion workers, indexes, and databases independently.
• Queue document ingestion and evaluation outside the interactive path.
• Use bounded context, streaming, connection reuse, batching, and safe caching where measured.
• Apply quotas, concurrency limits, backpressure, load shedding, and maximum model spend.
• Test load, stress, spikes, endurance, cold starts, dependency slowness, and scale-down.
• Optimize the measured bottleneck while preserving answer support and safety.

Step 11: Availability and Disaster Recovery

• Define user-workflow SLOs, error budgets, RTO, RPO, and minimum degraded capacity.
• Run application replicas across appropriate failure domains with readiness checks.
• Use deadlines, bounded retries with jitter, circuit breakers, and dependency bulkheads.
• Provide search-only, cached approved policy, queued response, or human-support degraded modes.
• Protect databases, source documents, models, prompts, configuration, keys, and audit evidence.
• Rebuild derived indexes from versioned governed sources.
• Validate fallback models for schema, quality, safety, privacy, latency, and cost.
• Test bad deployments, model outage, index failure, database restore, credential compromise, and region recovery.

Step 12: Evaluation Plan

Build a versioned evaluation set from real question patterns. Include normal, ambiguous, unanswerable, conflicting, stale, multilingual, adversarial, permission-sensitive, high-value, and accessibility cases. Keep an untouched release test set.

Step 13: Cost Estimate

Monthly cost
= model input/output and embedding usage
+ application/model compute and idle HA capacity
+ database, object, vector, cache, backup, and logs
+ network, gateway, security, monitoring, and support services
+ human knowledge ownership, review, support, incidents, and maintenance

Unit economics = total monthly cost / accepted resolved conversations

Create low, expected, high, peak, failure, and growth scenarios. Record users, requests, tokens, context, output, cache rate, retries, escalation, document updates, storage, retention, availability, review effort, unit prices, discounts, currency, date, and contingency.

Step 14: Delivery and Rollout Plan

Step 15: Operations Plan

• Define objectives, dashboards, actionable alerts, on-call ownership, and escalation.
• Create runbooks for model, provider, RAG, database, queue, identity, security, and deployment failures.
• Monitor model, prompt, corpus, permissions, provider, latency, safety, business, and cost changes.
• Review failed and escalated conversations using privacy-aware sampling.
• Reevaluate model, retrieval, and controls before material changes.
• Patch and rotate dependencies, images, credentials, keys, and certificates.
• Test backups, restores, fallback, rollback, incident response, and support handoff.
• Maintain user notices, documentation, inventories, decisions, exceptions, and retirement criteria.

Required Capstone Deliverables

Suggested Document Structure

1. Executive Summary
2. Business Context, Users, Baseline, and Goals
3. Scope, Assumptions, Constraints, and Requirements
4. Architecture Options and Decision Records
5. Context, Component, Data Flow, Deployment, and Recovery Diagrams
6. Model, RAG, Tool, and Data Design
7. Security, Privacy, Governance, Compliance, and Human Oversight
8. Scalability, Performance, Availability, and Disaster Recovery
9. Evaluation, Monitoring, Support, and Incident Response
10. Cost Estimate, Unit Economics, Risks, and Sensitivity
11. Delivery Roadmap, Acceptance Gates, and Retirement Plan
12. Open Questions, Limitations, and Appendices

Traceability Matrix

Architecture Decision Record Example

Decision: Use RAG plus a read-only order tool for the initial release
Status: Proposed for pilot
Context: Policies change frequently; order data is structured and customer-specific
Options: LLM only, fine-tuned model, RAG only, RAG plus tools, autonomous agent
Choice: Permission-aware RAG plus authenticated read-only order lookup
Consequences: Grounded current answers and exact status; added ingestion and integration
Risks: Retrieval failure, injection, data leakage, provider outage
Controls: ACL filters, citations, scoped identity, output validation, escalation
Evidence: Offline tests, authorization negatives, load/failure tests, pilot outcomes
Review trigger: New write action, new region, provider/model change, or target failure
Owner: Customer support AI product owner

Python Configuration Example

This simplified configuration makes important architectural choices visible. A real system would manage validated configuration, secrets, infrastructure, policies, models, and rollout state through secure version-controlled services.

architecture = {
    "pattern": "RAG with scoped read-only order tool",
    "knowledge": {"retrieval": "hybrid", "citations_required": True},
    "model": {"route": "evaluated primary", "fallback": "approved smaller model"},
    "security": {"authentication": True, "least_privilege": True},
    "human_oversight": {"low_evidence": "escalate", "financial_action": "not_allowed"},
    "operations": {"autoscaling": True, "monitoring": True, "tested_recovery": True},
}

print(architecture)

Design Review Checklist

• The problem, baseline, users, owner, scope, and measurable outcomes are clear.
• Architecture components trace to requirements and unnecessary complexity is removed.
• Data sources, permissions, quality, lineage, retention, and deletion are documented.
• Model and retrieval decisions use comparative evaluation and record limitations.
• Trust boundaries, threats, identities, tool permissions, and human approval are explicit.
• Compliance obligations trace to controls, tests, evidence, monitors, and owners.
• Normal, degraded, failure, rollback, recovery, escalation, and retirement paths exist.
• Evaluation covers quality, safety, important slices, service, users, business, and cost.
• Cost includes providers, infrastructure, people, reliability, support, growth, and contingency.
• The rollout has acceptance gates, kill switches, communication, training, and rollback thresholds.
• Assumptions, rejected alternatives, residual risks, open questions, and review triggers are honest.

Assessment Rubric

Common Capstone Mistakes

• Starting with cloud or model products before defining the problem and requirements
• Presenting a component list instead of interfaces, flows, failure paths, and ownership
• Using RAG without source governance, permission filtering, citations, or retrieval evaluation
• Adding an agent where a deterministic workflow is safer and simpler
• Claiming accuracy, scale, availability, compliance, or savings without evidence
• Treating authentication as the complete security architecture
• Omitting human workload, exceptions, support, incidents, recovery, and retirement
• Estimating only model-token cost and ignoring the lifecycle and people
• Designing every possible future feature instead of a controlled initial release
• Hiding assumptions and trade-offs rather than explaining them clearly

Portfolio Presentation

Present the capstone as a design review rather than a technology tour. Begin with the problem and stakes, explain the chosen architecture and its major decisions, walk through one normal and one failure scenario, show how outcomes will be evaluated, and finish with risks, cost, rollout, and open questions.

• Keep the main presentation concise and move detail into appendices.
• Use diagrams with readable labels and explicit trust boundaries.
• Explain why rejected alternatives did not fit the current requirements.
• Separate proposed targets from measured prototype or pilot results.
• State limitations and residual risks directly.
• Be ready to adapt the design when an interviewer changes a requirement.