An AI application may work well for ten test users and fail when thousands arrive. Requests queue, quotas are reached, accelerators run out of memory, retries multiply traffic, and costs grow. Scaling prepares the entire system for increased demand.

Scaling is not simply adding servers. A scalable LLM product manages capacity, queueing, data, tools, failure recovery, quality, security, and cost together.

What Is Scaling?

Serving scalability is the ability of an AI application to handle more traffic and workload while meeting defined objectives for quality, latency, availability, and cost. Workload includes request count, token length, concurrency, retrieval, and tool calls.

Training Scale vs Serving Scale

Vertical and Horizontal Scaling

Vertical Scaling

Vertical scaling gives one instance more memory, faster processors, stronger accelerators, or higher-bandwidth connections. It is simple but limited by available hardware and can create a larger single failure domain.

Horizontal Scaling

Horizontal scaling adds replicas or workers and distributes requests among them. It increases capacity and resilience but adds routing, state, deployment, and observability complexity. A model split across accelerators is distributed inference, which differs from adding independent replicas.

Hosted API Scaling

A hosted provider operates model infrastructure, but the application still manages quotas, timeouts, retries, fallbacks, privacy, and budgets. Use bounded backoff with jitter for retryable failures, never retry authorization errors, and protect side-effecting operations with idempotency controls.

Self-Managed Inference

Self-hosting provides infrastructure control and transfers responsibility for model files, accelerator scheduling, inference runtimes, batching, security patches, scaling, and incident recovery.

Client
  -> API gateway and authentication
  -> Rate limiter and admission control
  -> Bounded queue
  -> Load balancer
  -> Inference replicas or distributed workers
  -> Output validation
  -> Response and monitoring

Capacity Planning

Plan from measured workload rather than user count. Track requests per second, peak bursts, input and output token distributions, concurrency, generation time, tool fan-out, memory per request, and required failure headroom.

Python Estimate

This simplified use of Little's Law estimates average concurrency as arrival rate multiplied by average time in the system. It is not a replica count and cannot replace load testing.

def estimated_concurrency(requests_per_second, average_seconds):
    return requests_per_second * average_seconds

arrival_rate = 12.0
average_latency = 4.5
headroom = 1.4

baseline = estimated_concurrency(arrival_rate, average_latency)
planned = baseline * headroom

print(f"Average concurrency: {baseline:.1f}")
print(f"Capacity target with headroom: {planned:.1f}")

Queues and Backpressure

A queue absorbs short bursts, but an unlimited queue turns overload into extreme latency. Backpressure slows or rejects new work when downstream capacity is saturated.

• Set maximum queue depth and request deadlines.
• Separate interactive and offline workloads.
• Cancel work after a client disconnects or deadline expires.
• Apply tenant quotas so one customer cannot exhaust shared capacity.
• Return clear retry guidance when capacity is unavailable.
• Prioritize requests only through an authorized, documented policy.

Load Balancing, Batching, and Autoscaling

Load balancers distribute requests among healthy compatible replicas. Continuous batching improves accelerator utilization by grouping active sequences, trading throughput against queueing and tail latency.

Autoscaling should consider queue depth, pending tokens, active sequences, accelerator utilization, memory, and latency—not CPU alone. Because model loading can be slow, retain warm capacity and avoid rapid scaling oscillation.

Optimize Before Adding Hardware

• Remove duplicated prompt context and retrieve fewer, better chunks.
• Limit output to what the user needs.
• Route simple tasks to smaller models that pass evaluation.
• Use tested quantization on supported hardware.
• Cache safe repeated work with permission-aware keys.
• Use optimized runtimes and continuous batching.
• Run independent retrieval or tool calls concurrently where safe.

Caching at Scale

Caches may store retrieval results, embeddings, repeated prompt prefixes, or deterministic responses. Keys must include every value affecting output, such as permissions, source version, model, prompt, settings, and locale. Never share personalized or tenant-specific results through a global cache.

Resilience and Failure Handling

• Use readiness checks that verify real serving capability.
• Apply timeouts and limited retries with backoff and jitter.
• Use circuit breakers when dependencies repeatedly fail.
• Keep fallbacks compatible with quality and safety requirements.
• Deduplicate repeated jobs and make actions idempotent where possible.
• Prevent cascading failures by limiting fan-out and retry storms.
• Maintain rollback paths for models, prompts, runtimes, and deployments.

Security and Multi-Tenancy

• Authorize every document, cache, and tool operation.
• Isolate tenant data, adapters, logs, and usage accounting.
• Apply request, token, and spending limits per tenant and route.
• Keep model endpoints and credentials inaccessible to unauthorized clients.
• Test cross-tenant leakage under concurrency and failure conditions.
• Preserve privacy-safe audit trails.

Load and Stress Testing

• Use production-like prompt and output-length distributions.
• Test steady load, bursts, and long-running jobs.
• Increase concurrency until saturation.
• Measure queue time, TTFT, token speed, tail latency, errors, and cost.
• Include retrieval, tools, validation, retries, and fallbacks.
• Test dependency slowness, throttling, node loss, and restarts.
• Verify graceful degradation and recovery.

Monitoring and Cost

• Traffic, input and output tokens, and concurrency.
• Queue depth, wait time, cancellation, and rejection rates.
• TTFT and end-to-end p50, p95, and p99 latency.
• Success, timeout, retry, fallback, and invalid-output rates.
• Accelerator utilization, memory, batch size, and cache hit rate.
• Provider quotas, spend, and cost per successful task.
• Quality and safety by model, route, tenant, and version.

Common Mistakes

• Planning from user count without measuring token load.
• Using unlimited queues or retries.
• Autoscaling only from CPU usage.
• Adding hardware before removing unnecessary work.
• Ignoring model cold-start time.
• Sharing caches across permission boundaries.
• Optimizing throughput while tail latency becomes unusable.
• Failing over to a model that has not passed safety evaluation.

Best Practices

• Define quality, latency, availability, throughput, and cost objectives.
• Measure real workload distributions and critical paths.
• Optimize prompts, models, and workflows before adding hardware.
• Use bounded queues, backpressure, quotas, deadlines, and cancellation.
• Plan headroom for bursts, failures, and autoscaling delay.
• Test every fallback for quality, safety, and data compatibility.
• Load test end-to-end production-like workflows.
• Monitor continuously and rehearse rollback procedures.