When a prediction fails, a model behaves unexpectedly, or a deployment changes production behavior, teams need evidence. Logs provide a chronological record of important events such as application startup, data validation, model loading, requests, external calls, predictions, errors, approvals, and configuration changes.

Logging and monitoring are related but different. Logs are detailed event records used for investigation and audit. Monitoring turns logs, metrics, traces, and model evidence into trends, dashboards, objectives, and alerts. Good logs make monitoring and incident diagnosis possible, but logging everything without purpose creates privacy, security, cost, and search problems.

What Makes a Useful Log Entry?

Use UTC timestamps with a consistent high-resolution format. Keep stable fields and event names so dashboards and queries do not break whenever message wording changes.

Structured Logging

Prefer structured JSON or another machine-readable schema over free-form text. Structured fields can be filtered, aggregated, validated, and joined consistently. Keep the human message concise and place queryable facts in dedicated fields.

{
  "timestamp": "2026-07-20T08:42:11.184Z",
  "severity": "INFO",
  "event": "prediction.completed",
  "service": "credit-risk-api",
  "environment": "production",
  "request_id": "req_8f19",
  "trace_id": "tr_7b02",
  "model_version": "risk-v17",
  "release_version": "2026.07.4",
  "input_schema_version": "application-v3",
  "prediction_class": "manual_review",
  "duration_ms": 83,
  "status": "success"
}

Log Levels

Level meanings should be consistent across services. A warning should not page by itself unless a monitoring rule determines that its frequency or impact is urgent. Disable verbose debug logging in normal production unless it is carefully sampled and protected.

Where Logging Belongs in MLOps

Prediction Logging

Prediction records should support traceability and outcome analysis without becoming an uncontrolled copy of sensitive inputs. Record the prediction ID, timestamp, model and preprocessing versions, schema version, safe feature metadata, prediction or score where allowed, decision path, latency, fallback, and a reference that can later connect verified ground truth.

• Distinguish the model output from the final human or business decision.
• Record whether the system abstained, fell back, or required human review.
• Keep enough information to identify predictions affected by a faulty release or data source.
• Use privacy-preserving cohort or hashed identifiers only when necessary and approved.
• Store high-volume analytical prediction records separately from operational application logs when their access and retention needs differ.

Generative AI and Agent Logging

For generative AI, record the provider and model deployment, prompt-template version, retrieval and index versions, tool definitions, safety-policy version, token counts, latency, structured-output validation, refusal or fallback, and evaluation signals. Agent systems also need a trace of selected tools, authorized arguments, results, loop limits, approvals, and external side effects.

Do not record raw prompts, retrieved passages, model responses, tool payloads, or chain-of-thought by default. They may contain personal data, secrets, proprietary documents, or adversarial content. Prefer safe metadata, redacted samples, policy-approved review stores, and concise decision or error summaries.

{
  "event": "generation.completed",
  "request_id": "req_a182",
  "model_deployment": "support-model-v6",
  "prompt_version": "answer-v12",
  "retrieval_index_version": "kb-2026-07-18",
  "input_tokens": 1240,
  "output_tokens": 186,
  "citations_count": 3,
  "schema_valid": true,
  "safety_action": "allow",
  "tool_calls": 1,
  "duration_ms": 920
}

Correlation Across Services

A single prediction may pass through a gateway, application, feature service, model endpoint, database, queue, and human-review system. Propagate a request or trace context through every hop and include it in logs. Use a separate prediction or operation ID when work continues asynchronously after the original request.

request_id=req_8f19
API received request
-> feature service returned version fs-42
-> model endpoint used risk-v17
-> decision service routed to manual review
-> outcome service later attached verified label

One identifier connects the complete operational story.

Errors and Exceptions

• Log the failed operation, safe error type and message, component, dependency, and retryability.
• Include request, trace, job, release, and model identifiers needed for correlation.
• Capture stack traces at the boundary that owns diagnosis without duplicating the same exception at every layer.
• Separate expected validation failures from internal faults so error-rate dashboards remain meaningful.
• Do not expose stack traces or internal details in user-facing responses.
• Never place secrets, authorization headers, database strings, or complete sensitive payloads in exception logs.

Privacy and Security

Logs are production data and may be more sensitive than the application database because they combine identifiers, errors, internal structure, and user activity. Apply data minimization before collection rather than relying only on later deletion.

Operational Logs and Audit Logs

Operational logs explain runtime behavior and failures. Audit logs answer who performed a sensitive action, what resource changed, when, from where, and whether it succeeded. Model approvals, deployments, traffic changes, access-policy edits, secret access, data exports, and deletions usually require stronger protection and longer or policy-specific retention than routine debug events.

Audit records should be difficult for the subject of an investigation to alter. Restrict deletion and configuration privileges, centralize important records, synchronize time, and monitor logging-pipeline failures.

From Logs to Metrics and Alerts

Logs can produce counters and rates such as validation failures, fallback usage, model-load errors, safety events, or tool failures. Avoid using logs for every high-volume numeric signal when a native metric is more efficient. Alerts should operate on sustained, actionable conditions and link to a filtered log view and runbook.

Structured log event: prediction.failed
-> Aggregate by model_version + error_type
-> Calculate failure rate over sufficient traffic
-> Compare with service objective
-> Alert responsible on-call with dashboard and query link
-> Investigate correlated trace and safe event details

Centralized Log Pipeline

Applications + model servers + training jobs + pipelines + cloud audit sources
-> structured stdout or logging agents
-> buffering and transport
-> parsing, validation, enrichment, redaction, and sampling
-> protected centralized storage and indexes
-> search, dashboards, metrics, alerts, audit review, and archives
-> retention expiration and verified deletion

Design the pipeline to tolerate a logging-backend slowdown without taking down inference. Buffer within safe limits, apply backpressure or sampling, and define which security or audit events must never be silently dropped. Monitor the logging system itself for ingestion delay, dropped events, parser errors, storage failures, and unexpected cost.

Sampling and Volume Control

• Keep all critical errors, security events, release changes, and required audits.
• Sample repetitive successful requests when full retention is unnecessary.
• Use dynamic sampling to retain unusual latency, rare cohorts, new releases, and failed traces at higher rates.
• Avoid high-cardinality indexed fields unless their diagnostic value justifies cost.
• Aggregate numeric behavior into metrics and store detailed examples separately.
• Apply retention by log category rather than one period for every event.

Retention and Lifecycle

Retention should follow a documented purpose. Debug logs might need only a short window, operational records may need enough time for incident discovery, prediction records may follow label and model-analysis needs, and audit logs may follow contractual or regulatory requirements. Define hot searchable storage, lower-cost archive, legal hold, export, deletion, and restoration procedures. Longer retention is not automatically safer or more useful.

Python Structured Logging Example

import json
import logging
import time
import uuid

logger = logging.getLogger("prediction_api")
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(message)s"))
logger.addHandler(handler)
logger.setLevel(logging.INFO)

MODEL_VERSION = "risk-v17"

def predict(model, features):
    request_id = str(uuid.uuid4())
    started = time.perf_counter()

    try:
        prediction = model.predict([features])[0]
        logger.info(json.dumps({
            "event": "prediction.completed",
            "request_id": request_id,
            "model_version": MODEL_VERSION,
            "duration_ms": round((time.perf_counter() - started) * 1000, 2),
            "status": "success"
        }))
        return prediction
    except ValueError as exc:
        logger.warning(json.dumps({
            "event": "prediction.rejected",
            "request_id": request_id,
            "model_version": MODEL_VERSION,
            "error_type": type(exc).__name__,
            "status": "invalid_input"
        }))
        raise

The example deliberately avoids recording the raw feature vector. In a real service, configure handlers once during startup, use a production structured-logging library or framework integration, propagate trace context, and ensure uncaught internal exceptions are safely captured.

Logging Schema Governance

• Define required fields, types, allowed values, naming conventions, and event ownership.
• Version breaking schema changes and keep consumers compatible during migration.
• Validate events in tests and monitor parser or schema rejection rates.
• Maintain a catalog describing each event, its purpose, sensitivity, retention, and downstream consumers.
• Review new fields for privacy, security, cardinality, and cost before release.
• Test that secret and sensitive-field redaction works, including exception and dependency paths.

Common Logging Mistakes

Practical Exercise

• Define a JSON schema for application lifecycle, prediction success, validation rejection, and internal failure events.
• Add request, trace, model, release, schema, environment, status, and duration fields to a prediction API.
• Propagate one request ID through an API, feature lookup, model call, and asynchronous outcome update.
• Create tests proving passwords, tokens, authorization headers, raw personal fields, and unsafe control characters are not logged.
• Generate log-based error and fallback metrics and build a filtered incident query.
• Simulate a failed candidate model load and reconstruct the timeline from deployment, application, and audit logs.
• Document access roles, sampling, retention, archive, deletion, and logging-pipeline failure behavior.
• Estimate daily volume and cost, then remove low-value events or fields.

Production Logging Checklist

[ ] Stable structured event names, schema, levels, UTC timestamps, and owners defined
[ ] Request, trace, job, prediction, model, release, data, prompt, and config versions included where relevant
[ ] Application, data, training, evaluation, registry, deployment, inference, and audit events covered
[ ] Raw sensitive content avoided; secrets, headers, personal data, and untrusted text safely handled
[ ] Service accounts, access, tenant isolation, encryption, integrity, and query or export auditing enforced
[ ] Central pipeline buffering, parsing, redaction, sampling, indexing, and failure behavior tested
[ ] Log-derived metrics, dashboards, alerts, and incident queries linked to runbooks
[ ] Category-specific retention, archive, legal hold, deletion, and restoration documented
[ ] Volume, cardinality, ingestion delay, dropped events, and logging cost monitored
[ ] Schemas, redaction rules, access, retention, and diagnostic value reviewed regularly