A single container on one server is easy to understand but becomes fragile when traffic grows, hardware fails, updates are released, or workloads require specialized resources. Kubernetes coordinates containers across a cluster and continuously works to keep the declared application state running.

What Is Kubernetes?

Kubernetes, often abbreviated K8s, is an open-source platform for orchestrating containerized workloads. Teams declare the desired state—such as an image, replica count, resources, health checks, and network exposure—and Kubernetes controllers reconcile actual state toward it.

Kubernetes does not train models, guarantee application correctness, or automatically choose safe scaling rules. It supplies building blocks for scheduling, service discovery, rollout, recovery, configuration, security, and resource management.

Why Kubernetes Is Used for AI

• Runs several replicas of online inference services for availability and throughput.
• Schedules CPU-, memory-, and accelerator-intensive workloads onto suitable nodes.
• Restarts failed containers and replaces Pods when nodes disappear.
• Performs rolling, canary, blue-green, and shadow deployments with supporting tools and design.
• Scales services and workers in response to demand or queue signals.
• Provides common deployment patterns for APIs, batch jobs, training jobs, and pipelines.
• Centralizes configuration, policy, observability, and operational ownership across services.

Cluster Architecture

Managed Kubernetes services operate much of the control plane, but application teams still own workload configuration, access, resources, release safety, observability, cost, and model behavior.

Important Workload Resources

Pods Are Ephemeral

A Pod can be replaced during failure, scaling, scheduling, or deployment, and its IP and local writable filesystem can disappear. Store durable business data, model metadata, and important job state in appropriate external systems or persistent volumes rather than relying on a Pod's local filesystem.

A Model-Serving Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ticket-classifier
  labels:
    app: ticket-classifier
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ticket-classifier
  template:
    metadata:
      labels:
        app: ticket-classifier
        version: v1
    spec:
      containers:
        - name: api
          image: registry.example.com/ticket-classifier@sha256:<digest>
          ports:
            - name: http
              containerPort: 8000
          env:
            - name: MODEL_VERSION
              value: "17"
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
            limits:
              cpu: "2"
              memory: "2Gi"
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
            periodSeconds: 5
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            periodSeconds: 10
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true

Production manifests also need namespace, service account, Pod-level security settings, topology and disruption planning, configuration, observability, rollout, and organization-specific policy.

Expose the Deployment with a Service

apiVersion: v1
kind: Service
metadata:
  name: ticket-classifier
spec:
  selector:
    app: ticket-classifier
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

The Service gives clients a stable name and routes traffic to ready Pods matching its selector. A ClusterIP Service is internal; external traffic commonly enters through a managed load balancer, Ingress, or Gateway with TLS and authentication controls.

Essential kubectl Commands

Health Probes

Do not use liveness to test every downstream dependency; a temporary database outage could cause a restart storm. Readiness can require that the model is loaded and the instance is able to serve safely.

Resource Requests and Limits

Requests influence scheduling and reserved capacity; limits constrain certain resource use. Understated requests create contention, while excessive requests waste capacity and prevent scheduling. Measure real startup, idle, and peak behavior under representative load.

GPU Workloads

• Use nodes with the required accelerator, driver, runtime, and device plugin.
• Request accelerators explicitly and schedule with labels, affinity, or dedicated node pools.
• Use taints and tolerations when accelerator nodes should reject unrelated workloads.
• Track device memory, utilization, temperature, errors, and model loading time.
• Avoid one full model copy per process unless memory capacity supports it.
• Benchmark batching and concurrency; high GPU utilization does not automatically mean good user latency.
• Plan capacity for failure, rollout, and traffic spikes—not only average demand.

Node Selection

Autoscaling

CPU alone may be a weak signal for AI serving. Queue depth, concurrent requests, latency, accelerator utilization, and tokens or samples per second may better reflect load. Scale-up must account for model-download and warm-up time, and scale-down must not interrupt active work.

Horizontal Pod Autoscaler Example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ticket-classifier
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ticket-classifier
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

This is a starting example, not a universal policy. Load test the application and tune minimum capacity, thresholds, stabilization, and custom metrics against service objectives and cost.

Batch and Training Jobs

Use Jobs for finite work such as batch inference, evaluation, data validation, or training. Define retry, timeout, parallelism, completion, cleanup, output storage, and idempotency explicitly.

apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-evaluation-20260720
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 7200
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: evaluation
          image: registry.example.com/model-eval@sha256:<digest>
          args: ["--dataset-version", "support-2026-07-20"]
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"

CronJobs

CronJobs create Jobs on a schedule. Configure time zone when supported, concurrency policy, deadline for missed starts, job history, and idempotent output. Scheduled retraining should produce a candidate requiring evaluation and approval, not automatically replace production.

Configuration and Secrets

• Use ConfigMaps for non-sensitive runtime configuration and version changes with deployments.
• Use Secrets or an external secret provider for sensitive values.
• Kubernetes Secret data is encoding, not automatically sufficient encryption or access control.
• Restrict Secret read permissions and avoid broad environment-variable exposure where a mounted value or external provider is safer.
• Rotate credentials and restart or reload workloads through a controlled process.
• Never commit production secrets to Git or render them into logs and manifests.

Namespaces and Multi-Tenancy

Namespaces organize resources and support access, quotas, naming, and policy boundaries, but they are not automatically a complete security boundary. Use role-based access control, network policies, admission policy, quotas, and separate clusters when stronger isolation is required.

RBAC and Service Accounts

Each workload should use a dedicated service account with only the Kubernetes and cloud permissions it needs. Avoid default broad identities, cluster-admin access, and long-lived static cloud credentials. Audit both Kubernetes RBAC and external identity bindings.

Network Security

• Use NetworkPolicies with a supporting network implementation to restrict Pod communication.
• Default-deny traffic where operationally appropriate, then allow required flows.
• Encrypt external traffic with TLS and protect endpoints with authentication and authorization.
• Limit model-service egress to required data, registry, monitoring, and API destinations.
• Separate public gateways from internal services and administrative endpoints.
• Protect metadata and control-plane endpoints from workload access.

Pod and Container Security

• Run containers as non-root and prevent privilege escalation.
• Use read-only root filesystems and minimal writable volumes.
• Drop unnecessary Linux capabilities and avoid privileged containers.
• Use seccomp and other runtime controls according to platform standards.
• Do not mount the host filesystem, runtime socket, or broad credentials without a justified design.
• Enforce policy through namespace standards and admission controls.
• Scan and verify container images before admission.

Supply-Chain Security

Deploy images by immutable digest from an approved registry. Scan dependencies and images, generate provenance and software bills of materials where required, verify signatures or attestations, restrict who can push and deploy, and prevent untrusted CI code from accessing production credentials.

Rolling Updates

A Deployment can replace Pods gradually. Configure surge and unavailable capacity according to service needs, make readiness meaningful, and give Pods enough time for model loading and graceful termination.

• Set maxUnavailable low enough to preserve capacity.
• Use maxSurge only when cluster and accelerator capacity can support it.
• Configure progress deadlines and observe rollout status.
• Stop new traffic before termination and complete or checkpoint active work.
• Ensure old and new versions are compatible with requests, features, caches, and data schemas.

Canary, Blue-Green, and Shadow Releases

Kubernetes primitives support replicas and routing, but advanced traffic splitting and model comparison may require an ingress controller, service mesh, model-serving platform, or application-level routing.

Availability and Disruptions

• Run enough replicas across independent nodes or zones for the required availability.
• Use topology spread or anti-affinity so replicas do not all share one failure domain.
• Create Pod disruption budgets for voluntary maintenance while recognizing they do not prevent every outage.
• Reserve capacity for rollout, failure, and traffic spikes.
• Test node loss, zone loss, dependency outage, and recovery procedures.
• Back up and restore stateful dependencies; replacing Pods is not a data backup.

Observability

Emit structured logs to standard output, expose metrics, propagate trace IDs, and attach model and image versions to telemetry. A healthy Pod does not prove that the model is accurate or useful.

Cost Management

• Right-size requests using measured workload behavior.
• Track cost by namespace, workload, model, team, and successful outcome where feasible.
• Use autoscaling carefully and preserve minimum warm capacity for latency objectives.
• Schedule interruptible batch work on suitable lower-cost capacity with checkpointing.
• Share accelerators only through supported, measured isolation strategies.
• Detect idle GPU nodes, oversized Pods, excessive replicas, and failed jobs that retain resources.
• Balance cost optimization with availability, rollout, and recovery capacity.

GitOps and Configuration Management

GitOps stores desired Kubernetes configuration in a reviewed repository and uses an automated controller to reconcile the cluster. Use templates or overlays to separate environments, validate manifests in CI, protect production branches, and keep secrets in an appropriate encrypted or external system.

Git change
-> Schema, policy, security, and manifest validation
-> Peer review and approval
-> Merge desired state
-> GitOps controller reconciles cluster
-> Rollout health and AI quality checks
-> Promote, pause, or revert reviewed configuration

Kubernetes in the MLOps Lifecycle

Approved model and serving code
-> Build, scan, and register immutable image
-> Update reviewed Kubernetes configuration
-> Deploy to staging
-> Test health, compatibility, performance, and model behavior
-> Canary or shadow in production
-> Monitor service, model, data, outcomes, and cost
-> Scale, roll back, retrain, or retire

Troubleshooting Workflow

Check desired resource and rollout status
-> Inspect Pod status and events
-> Read current and previous container logs
-> Verify image, command, configuration, Secret access, and service account
-> Check probes, endpoints, DNS, network policy, and dependencies
-> Inspect CPU, memory, GPU, storage, and scheduling constraints
-> Correlate application traces and model metrics
-> Fix root cause, test, deploy safely, and document

Common Failure States

When Kubernetes Is the Wrong Choice

Kubernetes has significant operational complexity. A managed model endpoint, serverless container service, batch platform, or simpler virtual machine deployment may be better for a small team, low traffic, few services, or limited operational requirements. Choose it when scheduling, scaling, portability, policy, or platform standardization justifies the cost.

Practical Exercise

• Package a small validated prediction API as a non-root immutable image.
• Deploy it locally or to a safe development cluster with a Deployment and Service.
• Add startup, readiness, and liveness probes.
• Set measured resource requests and limits.
• Scale replicas and observe load distribution and termination behavior.
• Trigger a failed container and confirm recovery and alerting.
• Perform a rolling update and rollback while sending test traffic.
• Add logs, metrics, model version labels, a network policy, and a service account.
• Document troubleshooting, security, cost, and production gaps.

Best Practices

• Deploy scanned immutable images by digest with recorded model compatibility.
• Set meaningful startup, readiness, liveness, and graceful-shutdown behavior.
• Measure and configure resource requests, limits, concurrency, and accelerator capacity.
• Scale on signals that represent real AI workload demand and account for cold starts.
• Use least-privilege service accounts, secure Secrets, network policy, and restricted containers.
• Spread replicas across failure domains and plan disruptions, rollout capacity, backup, and recovery.
• Release gradually with quality and safety gates plus a tested rollback path.
• Monitor cluster, service, model, data, outcome, security, and cost layers together.