AI systems change frequently: code is fixed, prompts are revised, model artifacts are retrained, dependencies receive security updates, and infrastructure evolves. CI/CD turns those changes into a repeatable path from reviewed source to tested, traceable, and safely deployed artifacts.

What Is CI/CD?

Continuous delivery retains an explicit production release decision; continuous deployment automates that final step. Higher-risk AI systems often require approvals even when all technical checks pass.

Why AI CI/CD Is Different

A Complete MLOps Delivery Flow

Developer change
-> Pull request and review
-> Code, schema, data-contract, security, and AI evaluation checks
-> Build immutable serving artifact
-> Register approved model and metadata
-> Deploy same artifact to staging
-> Integration, compatibility, smoke, and load tests
-> Approval and progressive production release
-> Monitor service, model, data, outcomes, and cost
-> Promote, pause, roll back, or create a new candidate

Continuous Integration

• Integrate small coherent changes frequently rather than maintaining long-lived divergent branches.
• Require pull requests, peer review, and passing status checks on protected branches.
• Run fast checks early and expensive tests only when relevant inputs change.
• Fail clearly and preserve logs and artifacts needed to diagnose the result.
• Keep CI configuration under version control and review it as production code.
• Ensure a passing pipeline cannot be bypassed by an unreviewed production change.

Common CI Checks

Testing Pyramid for AI Services

AI Evaluation Gates

A model or prompt should pass predefined gates on a versioned evaluation set before promotion. Gates must reflect the actual decision and include important slices and guardrails, not only one aggregate accuracy score.

evaluation_gates:
  schema_valid_rate:
    min: 0.995
  overall_f1:
    min: 0.90
    max_regression_from_production: 0.01
  high_risk_recall:
    min: 0.97
  unsupported_claim_rate:
    max: 0.005
  safety_suite:
    required: pass
  latency_p95_ms:
    max: 300
  cost_per_1000_requests:
    max: 5.00

These numbers are illustrative. Real thresholds must be selected from the use case, error costs, baseline, uncertainty, and risk. A candidate can improve one metric while causing an unacceptable regression elsewhere.

Evaluation Data

• Version test cases, labels, scoring code, rubrics, and model or judge configuration.
• Include common, difficult, rare, adversarial, and high-impact scenarios.
• Keep evaluation data independent from training and prompt-tuning data.
• Protect sensitive cases and use approved sanitized or synthetic examples where possible.
• Measure reviewer agreement when judgments are subjective.
• Add confirmed production failures as regression tests under data-governance rules.

Data Pipeline CI

• Validate source and feature schemas and backward compatibility.
• Test missingness, ranges, categories, duplicates, freshness, and volume.
• Check labels and train, validation, and test split integrity.
• Detect target, temporal, entity, and test-data leakage.
• Verify transformation reproducibility and training-serving consistency.
• Record lineage, dataset version, and checksums for candidate training runs.

Continuous Training Is Not Continuous Promotion

A retraining pipeline may run on a schedule, after enough new labels, or after investigated drift. Its output is a candidate. That candidate still needs data validation, reproducibility, evaluation, registry status, approval, staged release, and production monitoring before replacing the current model.

Approved retraining trigger
-> Validate and version new data
-> Train reproducible candidates
-> Evaluate against baseline and production champion
-> Register candidate with lineage and metrics
-> Human or automated approval according to risk
-> Shadow or canary deployment
-> Promote only if production guardrails pass

Immutable Artifacts

Build once and promote the same immutable artifacts through environments. Do not rebuild different production binaries from a branch name or mutable tag. Record content digests for containers, model versions, configuration, and deployment manifests.

A Generic CI Pipeline

name: ai-service-ci

on:
  pull_request:
  push:
    branches: [main]

jobs:
  verify:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Check out reviewed source
        uses: actions/checkout@<pinned-version-or-sha>
      - name: Set up runtime
        uses: actions/setup-python@<pinned-version-or-sha>
        with:
          python-version: "3.11"
      - name: Install locked dependencies
        run: pip install --require-hashes -r requirements.txt
      - name: Static checks
        run: ./scripts/lint-and-typecheck.sh
      - name: Unit and integration tests
        run: ./scripts/test.sh
      - name: AI evaluation gates
        run: ./scripts/evaluate.sh
      - name: Security checks
        run: ./scripts/security-checks.sh

This illustrates pipeline structure. Actions, dependencies, permissions, runners, and commands must be selected and secured for the actual platform. Pin third-party automation, minimize permissions, and do not expose secrets to untrusted pull-request code.

Build Pipeline

Verified main commit
-> Build container in controlled runner
-> Create software bill of materials and provenance
-> Scan dependencies, secrets, and image
-> Sign or attest artifact when required
-> Push immutable digest to approved registry
-> Store build logs and test evidence
-> Update candidate release metadata

Continuous Delivery

Continuous delivery automatically prepares a release for production but retains an explicit deployment decision. This is useful when release timing, change windows, model risk, stakeholder approval, or regulatory controls require human authorization.

Continuous Deployment

Continuous deployment automatically releases every eligible change that passes the complete pipeline. It is appropriate only when automated tests and production safeguards are strong enough for the workflow's risk. High-impact model changes may remain continuously delivered rather than automatically deployed.

Environment Promotion

Keep environment-specific configuration outside the artifact. Promote the same artifact while injecting reviewed configuration and protected secrets for each environment. Staging should be representative without copying unrestricted production data.

Database and Feature-Schema Changes

• Prefer backward-compatible expand-and-contract migrations.
• Deploy readers and writers in an order that supports mixed versions during rollout.
• Back up or checkpoint state and test migration and rollback on realistic volume.
• Version feature and request schemas and check model compatibility before traffic shifts.
• Do not assume application rollback can undo an irreversible data migration.
• Monitor data quality and transformation results during and after migration.

Release Strategies

Deployment Gates

• All required source, test, evaluation, security, and policy checks pass.
• Artifacts are immutable, traceable, scanned, and stored in approved registries.
• Model, preprocessing, API, feature, and infrastructure versions are compatible.
• Staging smoke, integration, load, failure, and rollback tests pass.
• Monitoring, alerts, dashboards, and runbooks are ready.
• Change risk, approvals, rollout size, guardrails, and rollback thresholds are recorded.
• On-call or operational ownership is confirmed for the release window.

Post-Deployment Verification

Deploy canary
-> Confirm instances are healthy and ready
-> Run synthetic prediction and end-to-end checks
-> Compare latency, error, saturation, and cost
-> Compare prediction distribution, quality sample, safety, and fallback
-> Check business and customer guardrails
-> Promote gradually, pause, or roll back
-> Record final release state and evidence

Rollback and Roll Forward

Rollback restores the previous compatible application, model, prompt, configuration, or infrastructure version. Roll forward deploys a corrective new version. Choose based on compatibility, data changes, incident severity, and recovery speed.

• Keep the last known-good artifacts and configuration available.
• Define automatic and manual rollback thresholds before release.
• Test rollback in staging and through operational exercises.
• Pause harmful external actions while deciding.
• Reconcile predictions, records, messages, or decisions already produced.
• Document when rollback is unsafe because of schema or state changes.

Secrets and Credentials

• Use platform secret stores or workload identity rather than plaintext pipeline variables.
• Grant each job and environment only the permissions it needs.
• Use short-lived credentials and protected deployment environments where possible.
• Prevent untrusted fork or pull-request code from receiving secrets.
• Mask secrets in logs but do not rely on masking as the primary control.
• Rotate credentials and audit use after suspected exposure.

CI/CD Supply-Chain Security

• Protect source branches, tags, release approvals, and registry permissions.
• Pin third-party actions, images, and build dependencies to reviewed versions or digests.
• Use isolated ephemeral runners or carefully hardened managed runners.
• Avoid privileged builds and broad cloud credentials.
• Generate provenance and software bills of materials when required.
• Scan source, dependencies, infrastructure, containers, models, and artifacts.
• Sign or attest releases and verify policy before production admission.
• Audit who changed, approved, built, and deployed each release.

Pipeline Permissions

CI jobs that only test source should not receive production deployment credentials. Separate build, staging, and production jobs with narrowly scoped identities and protected approvals. A compromised test dependency should not automatically gain production control.

Caching and Performance

• Cache immutable dependencies using keys based on lock-file content.
• Do not cache secrets or unverified outputs from untrusted changes.
• Parallelize independent tests while respecting data and compute limits.
• Use change detection to avoid unnecessary expensive training or evaluation.
• Separate fast pull-request gates from longer scheduled robustness suites.
• Track queue time, execution time, failure rate, flakiness, and cost by job.

Flaky Tests

A flaky test passes and fails without a relevant behavior change. Quarantining may be necessary briefly, but repeatedly retrying until green hides real risk. Fix nondeterminism, external dependencies, timing, random seeds, shared state, and inadequate tolerances, and track flaky-test ownership.

Handling Nondeterministic AI Evaluations

• Control model, parameters, tools, retrieval corpus, prompt, and evaluation version where possible.
• Use repeated trials or statistical thresholds when output variability is material.
• Prefer deterministic schema and policy checks before subjective scoring.
• Use human review for borderline, high-impact, or disagreement cases.
• Monitor variance and judge-model changes rather than hiding them behind broad tolerances.
• Record raw outputs and scoring evidence with privacy safeguards.

Pipeline Observability

Deployment and Model Monitoring

A pipeline is not complete when deployment reports success. Feed post-release health, latency, prediction quality, drift, fairness, safety, cost, and business guardrails into release decisions. Automated rollback should use robust signals and avoid oscillation or reacting to noisy metrics.

Approvals and Separation of Duties

• Match approval depth to model and deployment risk.
• Ensure reviewers see evaluation, security, compatibility, and rollout evidence.
• Verify approvers are authorized for the environment and decision.
• Separate artifact creation from sensitive production promotion where required.
• Record approval, timestamp, artifact identifiers, and conditions.
• Use time-limited approvals so a stale candidate is not deployed much later without revalidation.

GitOps

GitOps keeps desired deployment state in a reviewed repository and uses an agent to reconcile the environment. It strengthens auditability and drift detection but still requires secret management, policy validation, progressive release, and secure access to both source and cluster.

Common Failure Patterns

A Practical Project

• Create a FastAPI prediction service with an approved model artifact.
• Add formatting, types, unit, integration, contract, and AI evaluation tests.
• Build a non-root container and scan source, dependencies, secrets, and image.
• Record the Git commit, dataset, model, evaluation, and image digest.
• Deploy the immutable image to a local or safe staging environment.
• Run smoke, compatibility, failure, and load tests.
• Canary a new model version with explicit quality and latency gates.
• Trigger a failure and demonstrate rollback and output reconciliation.
• Create a dashboard and runbook for pipeline and production health.

Pipeline Checklist

[ ] Protected source and reviewed change
[ ] Reproducible locked environment
[ ] Static, unit, integration, and contract checks
[ ] Versioned data and AI evaluation gates
[ ] Secret, dependency, container, and policy scans
[ ] Immutable traced artifacts and provenance
[ ] Production-like staging validation
[ ] Approved progressive release
[ ] Health, model, outcome, safety, and cost monitoring
[ ] Tested rollback, recovery, and reconciliation
[ ] Auditable ownership and release record

Best Practices

• Integrate small reviewed changes and provide fast, trustworthy feedback.
• Test code, data, features, schemas, model behavior, security, and infrastructure.
• Treat retraining output as a candidate and require promotion gates.
• Build once, generate provenance, and promote immutable artifacts through environments.
• Use least-privilege short-lived identities and isolate untrusted code from secrets.
• Release gradually with compatibility checks, service and AI guardrails, and a kill switch.
• Test rollback, roll-forward, state recovery, and output reconciliation.
• Measure pipeline reliability and speed alongside production quality, outcomes, safety, and cost.