Security

Protect data, models, software supply chains, endpoints, infrastructure, and users across the AI lifecycle.

Machine learning systems combine valuable datasets, executable model artifacts, dependency-heavy software, powerful cloud infrastructure, and interfaces that influence real decisions. A compromise can expose private data, poison training, replace a model, steal intellectual property, manipulate predictions, interrupt service, or trigger harmful downstream actions.

Security is a lifecycle responsibility rather than a feature added after deployment. Controls should match the use case, threat actors, data sensitivity, business impact, deployment environment, and applicable obligations. Security and responsible-AI risk management overlap, but neither replaces the other.

Security Objectives

Objective	Meaning for AI	Example failure
Confidentiality	Only authorized actors access data, weights, prompts, secrets, and outputs	Training records or proprietary models are exposed
Integrity	Inputs, artifacts, configuration, and decisions are not changed improperly	A registry model is replaced with a backdoored version
Availability	Authorized users receive dependable service	Abusive traffic exhausts an inference endpoint
Authenticity	Identity and artifact origin can be verified	A pipeline deploys an image posing as an approved build
Accountability	Sensitive actions are attributable and auditable	Production traffic changes without an actor record
Resilience	Failure is contained and the system recovers safely	One dependency outage disables every prediction path

Threat Modeling

Threat modeling identifies assets, trust boundaries, actors, entry points, abuse paths, impact, controls, detection, and residual risk. Perform it before implementation and repeat it when models, data sources, tools, permissions, vendors, or deployment patterns change.

Inventory datasets, labels, code, notebooks, models, images, credentials, endpoints, logs, and downstream actions.
Map data and control flows across users, applications, pipelines, registries, clouds, providers, and human review.
Consider external attackers, malicious insiders, compromised dependencies, abusive users, and accidental misuse.
Analyze data theft, poisoning, model replacement, extraction, denial of service, cross-tenant access, and unauthorized tool execution.
Rank threats by likelihood and impact, then assign mitigations, tests, detection, response, owners, and accepted residual risk.

Output

Asset -> Threat actor -> Entry point -> Abuse path -> Impact
-> Preventive control -> Detection -> Response -> Owner

Example
Model registry -> compromised CI identity -> replace approved model
-> manipulated predictions
-> short-lived identity + least privilege + artifact provenance + approval gate
-> integrity alert
-> stop rollout, revoke identity, restore known-good release, investigate predictions

Secure the Data Lifecycle

Collect only data needed for an approved purpose and document its source, authority, sensitivity, retention, and owner.
Authenticate producers, validate schemas and ranges, scan uploaded files, and quarantine malformed or untrusted data.
Encrypt data in transit and at rest and restrict access by workload, role, environment, and tenant.
Version datasets and transformations, preserve lineage, and protect training, validation, test, and feedback sets from silent modification.
Separate identifiers where appropriate and use tokenization, aggregation, or privacy-enhancing methods based on risk.
Apply access, retention, deletion, backup, restoration, export, and legal-hold controls to data, features, labels, logs, and predictions.

Data Poisoning and Backdoors

Attackers or faulty sources can add, remove, relabel, duplicate, or selectively alter training and feedback data. A backdoor may make a model behave normally except when a particular trigger appears. Authenticate sources, monitor data quality and distributions, govern labels, quarantine feedback, preserve reproducible dataset versions, and test candidate models across important slices and adversarial cases.

New feedback should not automatically become training data. Validate provenance, remove duplicates and manipulation, review sensitive labels, and require the retrained candidate to pass the normal approval and release process.

Protect Models and Artifacts

Treat models, tokenizers, preprocessing objects, feature definitions, adapters, prompts, and environments as controlled release artifacts.
Store immutable versions in restricted registries and record hashes, provenance, lineage, evaluation evidence, ownership, and approval state.
Verify artifact identity, integrity, expected origin, and policy before training, loading, or deployment.
Avoid unsafe deserialization of untrusted model files; use approved formats and constrained loading environments.
Separate create, approve, deploy, and delete authority to reduce the blast radius of one compromised identity.
Retain a complete last known-good release for rollback and securely retire obsolete artifacts.

Software and AI Supply Chain

Use reviewed sources and pin packages, base images, models, datasets, plugins, and CI actions to immutable versions or digests.
Generate software and model inventories and preserve build provenance.
Scan source, dependencies, containers, secrets, licenses, infrastructure definitions, and artifacts with suitable tools.
Build in isolated ephemeral environments with minimal network access and short-lived credentials.
Sign or attest releases and verify policy before deployment.
Patch according to exploitability and impact, then rebuild and retest instead of modifying production manually.
Review provider security, model and data terms, deprecations, and incident-notification processes.

Identity, Access, and Secrets

Authenticate humans and workloads separately. Prefer federated identity and multi-factor authentication for people, plus managed workload identity and short-lived credentials for services. Apply least privilege at the narrowest practical scope and separate development, approval, deployment, security, and audit duties.

Store passwords, API keys, certificates, and signing keys in an approved secret or key manager.
Retrieve secrets at runtime; never hardcode them in code, notebooks, models, images, logs, or client applications.
Use separate identities and credentials by workload and environment.
Audit secret access and permission changes, review stale access, and practice rotation and emergency revocation.
Authorize both the caller and ownership of the requested tenant, model, data, record, and administrative action.

Network and Runtime Security

Keep registries, storage, databases, training systems, and internal endpoints private unless public access is explicitly required.
Segment networks by trust boundary and allow only required inbound and outbound communication.
Use TLS with correct certificate verification and control egress to reduce data exfiltration.
Harden images and hosts, minimize installed software, run containers as non-root, use read-only filesystems where practical, and remove unnecessary capabilities.
Patch operating systems, runtimes, drivers, orchestrators, and accelerators through tested immutable releases.
Protect cloud metadata services and block server-side request forgery paths to workload credentials.

Secure CI/CD and Training Pipelines

Pipelines can modify datasets, registries, infrastructure, and production, so treat them as privileged production systems. Require reviewed source changes, protect branches, use ephemeral runners and short-lived identities, and never run untrusted contribution code with production secrets or network access.

Validate pipeline parameters and pin actions, dependencies, images, and environments.
Enforce code, data, model, security, policy, and artifact-integrity gates before promotion.
Require authorized approval for high-risk releases and prevent a notebook or training job from promoting itself.
Record builds, approvals, registry changes, deployments, traffic changes, configuration, and rollbacks in protected audit logs.

Secure Model Endpoints

Control	Purpose
Authentication and authorization	Establish identity and enforce model, action, tenant, record, and admin scope
Schema and file validation	Reject malformed, oversized, unsupported, or dangerous inputs
Rate, concurrency, token, and cost limits	Reduce abuse, denial of service, and denial of wallet
Timeouts and bounded retries	Prevent resource exhaustion and retry storms
Output validation	Block malformed or unsafe output before downstream execution
Network controls	Limit public routes, origins, protocols, and dependency access
Safe errors	Avoid exposing internals, secrets, stack traces, or unauthorized record existence
Abuse monitoring	Detect probing, extraction, scraping, enumeration, and overload patterns

AI-Specific Threats

Threat	What happens	Defensive direction
Evasion or adversarial input	Crafted input causes an incorrect prediction	Validate domain constraints, test robustness, detect abuse, and add human review for high impact
Model extraction	Queries approximate or steal model behavior	Authenticate, rate limit, monitor, restrict output detail, and protect artifacts
Membership inference	An attacker estimates whether a record was in training	Minimize sensitive data, test leakage, restrict outputs, and apply privacy techniques where appropriate
Model inversion	Outputs reveal sensitive attributes or representative data	Limit confidence detail, test leakage, control access, and reduce memorization
Poisoning and backdoors	Training or feedback manipulates future behavior	Authenticate sources, validate data and candidates, preserve lineage, and gate releases
Denial of wallet	Expensive requests exhaust financial budgets	Enforce request, token, tool, concurrency, user, and spend limits
Training-serving skew	Production transformations differ from training	Share or version transformations and continuously verify parity

Generative AI, RAG, and Agent Security

Treat prompts, retrieved documents, webpages, email, uploaded files, tool results, and model output as untrusted content—not privileged instructions. Prompt injection can arrive directly from users or indirectly through retrieved material.

Keep system rules and authorization outside untrusted content; never use the model as the authorization boundary.
Enforce document and tenant permissions before retrieval and filter through authoritative access metadata.
Scan and safely parse files in isolated environments without sensitive credentials or unrestricted networks.
Allowlist tools and destinations, validate typed arguments, minimize tool privileges, and require approval for consequential actions.
Constrain agent steps, time, tokens, spend, recursion, and external side effects.
Validate structured output before executing commands, database changes, code, URLs, or messages.
Test injection, exfiltration, secret discovery, unsafe files, malicious links, cross-tenant retrieval, and tool misuse.
Version and monitor provider models, prompts, retrieval indexes, tools, policies, and evaluators.

Treat Model Output as Untrusted

A model can return malformed JSON, unsafe code, invented identifiers, hostile markup, SQL fragments, or unauthorized instructions. Validate against strict schemas and business rules, encode for the destination context, authorize external actions independently, and use sandboxing or human approval when consequences are significant.

Security Monitoring

Authentication failures, authorization denials, unusual privilege changes, and dormant account use.
Unexpected model, dataset, image, prompt, tool, policy, registry, or configuration changes.
Artifact checksum, signature, provenance, or approved-source verification failures.
Systematic probing, confidence scraping, extraction patterns, high-cost requests, and rate-limit pressure.
Cross-tenant access attempts, abnormal exports, suspicious secret access, and unusual egress.
Prompt-injection indicators, unauthorized tool attempts, agent loops, and blocked side effects.
Logging gaps, disabled controls, missing audit events, and security-monitoring pipeline failures.

Security Testing

Test	Purpose
Static and dependency analysis	Find insecure code, exposed secrets, and vulnerable libraries
Container and infrastructure scanning	Find unsafe images, permissions, networks, and configuration
Dynamic endpoint testing	Exercise authentication, authorization, validation, limits, and safe errors
Tenant-isolation testing	Prove one tenant cannot access another tenant's data or models
Artifact verification	Reject tampered, unapproved, or unexpected model and image versions
Adversarial ML evaluation	Assess evasion, leakage, extraction, poisoning, and backdoor risks
GenAI red teaming	Test injection, exfiltration, tool misuse, harmful output, and excessive agency
Resilience and incident exercise	Verify overload handling, detection, containment, rollback, notification, and restoration

Incident Response

Output

Detect suspicious event
-> Triage affected users, data, artifacts, models, regions, and actions
-> Preserve evidence and establish incident ownership
-> Contain: revoke identity, isolate workload, stop pipeline, disable tool, block traffic, or roll back
-> Remove compromised code, data, artifact, identity, or configuration
-> Restore from verified known-good assets and validate service + model behavior
-> Identify and reconcile affected predictions and external actions
-> Notify required stakeholders and improve tests, controls, monitoring, and training

Restoring endpoint availability is not enough. Determine which predictions were produced, which decisions or external actions followed, whether data or models were poisoned, and whether affected users require correction, appeal, or notification. Back up required state and regularly test recovery against explicit recovery objectives.

Common Security Mistakes

Mistake	Risk	Better practice
Hardcoded key	Credential leaks through code or images	Use workload identity and a secret manager
Broad administrator role	One compromise reaches every resource	Use scoped least privilege and separated duties
Public storage for convenience	Data or models become exposed	Default to private access and explicit sharing
Unverified model file	Tampering or unsafe deserialization	Use trusted registry, provenance, immutable versions, and integrity checks
Raw feedback becomes training data	Poisoning and privacy violations	Quarantine, validate, govern, and approve it
Model decides authorization	Probabilistic output bypasses policy	Enforce deterministic authorization outside the model
Prompt filter is the only defense	Indirect injection and tool misuse remain	Layer identity, retrieval, tool, output, limit, and monitoring controls
Unlimited inference	Denial of service or wallet	Apply request, token, tool, concurrency, cost, and user limits
Sensitive production logs	Secrets and private data leak through telemetry	Minimize, redact, restrict, audit, and retain by purpose
No recovery exercise	Controls fail during a real incident	Test revocation, isolation, rollback, restore, and reconciliation

Practical Security Exercise

Draw a prediction API's data flow from client through features, model, logs, and downstream decision.
List assets, trust boundaries, threat actors, abuse cases, impact, controls, detections, and owners.
Replace static keys with a least-privilege workload identity and secret manager.
Pin, scan, attest, and verify a container and model artifact before deployment.
Add request schemas, file and size limits, authorization, rate limits, timeouts, safe errors, and output validation.
Test cross-tenant access, tampered artifacts, poisoning indicators, extraction-like queries, overload, and secret leakage in logs.
Simulate a compromised deployment identity: revoke it, stop rollout, restore a known-good release, and identify affected predictions.
Document residual risk, dashboards, alerts, escalation, evidence handling, recovery, and improvement actions.

Production Security Checklist

Output

[ ] Purpose, impact, data sensitivity, assets, actors, trust boundaries, and threats documented
[ ] Data sources authenticated; schemas, files, labels, lineage, poisoning, retention, and deletion controlled
[ ] Models, images, code, prompts, tools, and configs versioned, scanned, provenanced, approved, and verified
[ ] Human and workload identities separated; MFA, short-lived credentials, least privilege, and duties enforced
[ ] Secrets externalized, access audited, rotation and emergency revocation tested
[ ] Networks segmented; private access, TLS, controlled ingress and egress, hardened runtime, and patching applied
[ ] CI/CD isolated; untrusted code separated; tests, attestations, approvals, and audit records required
[ ] Endpoints enforce authentication, authorization, validation, quotas, rate, concurrency, timeout, and safe output
[ ] Poisoning, evasion, extraction, inversion, injection, tool misuse, and denial-of-wallet risks assessed
[ ] Detection, protected audit logs, incident containment, rollback, reconciliation, and restoration tested
[ ] Inventory, governance, residual-risk acceptance, review cadence, and retirement ownership maintained