mlops

MLOps applies DevOps discipline to machine learning, but the reason it became its own field rather than just DevOps-for-models is that ML systems accumulate a kind of technical debt ordinary software doesn't. Sculley and colleagues at Google (2015) identified this directly in "Hidden Technical Debt in Machine Learning Systems": a model's behavior depends not just on its code but on the data feeding it, and changing any upstream input — a feature, a data source, a preprocessing step — can silently change the model's behavior in ways no code review would catch, a problem they called CACE: changing anything changes everything. Ordinary software has mostly stable, explicit dependencies between components; ML systems have implicit, data-mediated dependencies that conventional software engineering practices were never built to track.

The lifecycle is a loop, not a pipeline with an end — retraining feeds back into the same stages it started from. Data flows through training, then validation checks the resulting model against held-out metrics before it's accepted into a registry, then deployment pushes it to production, then monitoring watches its live behavior — and when monitoring detects drift or degradation, that triggers retraining, which re-enters the loop at the start rather than the system simply running one model forever. Treating this as a one-time pipeline that ends at deployment is the most common structural mistake in MLOps: a model that performed well at launch degrades as the real-world data distribution it sees in production drifts away from what it was trained on, and without the monitoring-to-retraining loop closed, nothing catches that degradation until users notice it.

Canary deployment exists because an ML model's failure mode is rarely a crash — it's quietly wrong predictions, which a traditional health check can't detect. A new software version that's broken usually throws errors or fails obviously; a new model version that's subtly worse just makes slightly worse predictions, with no exception to catch and no immediate symptom. Canary deployment addresses this by routing a small, growing fraction of real traffic to the new model while monitoring its actual prediction quality against the existing model, rather than trusting offline validation metrics alone — moving from 5 percent of traffic, to 50, to 100 only as the live comparison keeps looking good, which catches the exact failure mode offline validation is most likely to miss: a model that looked fine on a static test set but behaves differently on the live, shifting distribution of real production traffic.

The model registry is the artifact that makes "which exact model is in production, and what produced it" an answerable question instead of institutional memory. A registry — MLflow, built at Databricks and now the most widely used open-source option, or Kubeflow's Kubernetes-native registry, or a cloud provider's managed equivalent like SageMaker's or Vertex AI's — stores each model version alongside its lineage: which dataset version, which code commit, which hyperparameters, and which evaluation metrics produced it. This matters for more than organization; when a production incident traces back to a specific model version, the registry is what lets a team answer exactly what changed between the version that worked and the version that didn't, rather than reconstructing the answer from scattered logs, Slack threads, and whoever happens to remember.

Shadow deployment tests a model against real traffic with zero user-facing risk, at the cost of needing somewhere to send the predictions that never affect anyone. The new model receives a full copy of live production traffic and generates real predictions, but those predictions are logged and compared against the current production model's outputs rather than ever being shown to a user — full exposure to real-world data distribution with none of the risk a canary's partial live traffic still carries, which makes it the preferred validation step before even a canary rollout for changes where a bad prediction reaching a real user would be costly.

Read MLOps as the discipline built specifically around the fact that a model's correctness depends on data it doesn't control as tightly as it controls its own code: every practice — the registry's lineage tracking, canary's live-metric gating, shadow deployment's zero-risk validation, the monitoring loop that feeds back into retraining — exists to make an implicit, data-mediated dependency into something explicit enough to version, test, and roll back, which is exactly the CACE problem Sculley's paper identified as uniquely ML's to solve.