the pipeline was green. the model was wrong.

The pipeline was green. It had been green for six weeks. Every commit triggered the build. Every build passed the tests. Every deployment completed. The Slack notification said "deploy successful" with a small rocket emoji.

The model had been quietly wrong for most of those six weeks.

Not wrong in a way that threw exceptions. Not wrong in a way that spiked the error rate. Not wrong in a way that any of the alerts I had configured would have caught. Wrong in the way that matters most and is hardest to see: the predictions were becoming less accurate every day. The world had kept moving. The training data had not.

Nobody noticed because everything was green.

This is the specific way DevOps fails at AI. Not because DevOps engineers are bad. Because DevOps was built for a world where the same code produces the same output. And in that world, green means good. A test passes or it doesn't. A service is up or it isn't. An artifact deployed to staging is the exact same artifact that reaches production. The CI/CD pipeline is a deterministic machine operating on deterministic software.

Machine learning is not deterministic software.

A model is trained on historical data. The moment it ships, that history starts aging. Users change their behavior. New patterns emerge. Old correlations break. The data distribution your model learned from diverges from the distribution it now serves. This happens without any code change. Without any deployment. Without any human action at all. The world simply keeps moving.

Your pipeline stays green. Your model keeps degrading.

The failure mode does not announce itself. There are no 500 errors. There is no latency spike. The service health dashboard shows 99.9% uptime. The model is answering every request. It is just answering them worse than it was in week one, and better than it will be in week twelve, and nobody knows.

"But we have monitors on the—" On what? On latency. On error rate. On request volume. On infrastructure health. On all the things DevOps taught you to watch. None of those metrics tell you whether the predictions are still any good. That requires ground truth. Ground truth requires knowing what actually happened after the model made its recommendation. That requires a feedback loop. DevOps does not build feedback loops into production by default. You have to add them. Most teams do not.

I did not. For six weeks.

Here is the second way DevOps fails at AI, and it is worse than the first. Rollback.

In traditional software, rollback is the escape hatch. Something breaks. You revert to the last known good version. The code from two weeks ago still works because code does not degrade. It is deterministic. Yesterday's version and today's version of the code produce the same outputs for the same inputs. Roll back and you are safe.

Roll back a model and you are back to a version that was wrong in a slightly different way. The model from two weeks ago was trained on data that is now a further two weeks older. It has not aged better in the artifact store. The world has not helpfully paused so your old model could stay relevant. Rollback in MLOps is not a fix. It is a retreat to a different, earlier failure state.

The mental model is wrong. DevOps engineers learn to think of deployment as an endpoint. Ship it. Monitor it. If it breaks, roll back. If it doesn't break, done. The artifact is stable.

An AI platform engineer knows that deployment is not an endpoint. It is the beginning of degradation. The model starts becoming less relevant from the moment it hits production. Not catastrophically. Not immediately. Slowly. Inevitably. The question is not whether it will degrade. It is how fast and whether you will notice.

This changes the entire operating model.

In DevOps you deploy code and monitor infrastructure health. In an AI platform you deploy a model and monitor prediction quality, data distribution shift, ground truth feedback latency, and training data freshness. Those are completely different instruments measuring completely different things. The Datadog dashboard your DevOps team built tells you the pods are running. It does not tell you whether the pods are running a model that still makes good decisions.

I spent three months watching pods run a model that was making increasingly bad decisions. The Datadog dashboard was excellent. Very informative about pod health.

The third failure is ownership.

A traditional software service has an owner. The team that writes it runs it. They wrote the business logic. They understand the edge cases. When something breaks they know where to look. DevOps amplified this by pushing ownership to the team level and giving them the tools to deploy and monitor themselves. Clear owner. Clear accountability. Works.

A machine learning model in production has fractured ownership by design. The data scientist built it. They understand the architecture, the training process, the evaluation metrics, the known failure modes. They do not own production. The platform team owns production. The data engineer owns the pipeline that feeds training data. The product team owns the feature that surfaces model outputs to users. Nobody owns the intersection of all four. When the model degrades, the incident falls into the gap between them.

"But we have an on-call rota that—" For what? For incidents the alerting system knows to look for. Model degradation is not a page. It is a gradual trend in a metric nobody configured an alert for, visible to a person who had the judgment to look for it and understood what they were seeing. In most organizations that person does not exist at 2am. Sometimes they do not exist at all.

I was the person who noticed. I noticed because I was manually sampling outputs on a Thursday afternoon for an unrelated reason. Not because a system told me to look. Because I happened to look.

The AI platform discipline exists to close these three gaps systematically. Not with more YAML. Not with better Kubernetes operators. With a different set of primitives built for the actual problem.

Continuous training. Not just continuous deployment. Automated pipelines that detect data drift above a threshold and trigger a new training run. Distributional monitoring that compares the embedding space of production inputs this week against the training distribution. Ground truth pipelines that collect outcome feedback and use it to evaluate whether predictions were actually correct, not just whether they were returned without a 500.

Model registries with performance lineage. Not just "version 1.2.3 is deployed." Version 1.2.3, trained on data through this date, evaluated at this accuracy on this test set, showing this drift rate in production, with these ground truth outcomes logged. A complete artifact record that lets you answer the question "is this model still any good" rather than the question "is this service still running."

Shadow deployment. Run the new model candidate in parallel with the production model, routing a fraction of traffic to both, comparing prediction quality under identical conditions before promoting. Not A/B testing for user experience. A/B testing for model correctness. Different goal. Different infrastructure. Most teams do not build it because DevOps does not require it.

The DORA 2025 report said something important. AI amplifies the quality of the engineering system it operates within. Teams with mature DevOps ship AI faster. Teams without it deploy models into chaos.

What it did not say loudly enough: DevOps maturity is necessary but not sufficient. The practices that make software delivery excellent do not automatically make AI deployment trustworthy. You need both. They are not the same discipline. They share tools and they share culture and they share almost nothing else at the layer where AI actually fails.

DevOps taught us how to know when software is broken. The service crashes. The test fails. The error rate climbs. The alert fires.

AI fails without breaking. It fails while everything monitors as healthy. It fails while the pipeline stays green and the dashboard shows uptime and the deployment log says successful.

That is a different kind of failure. It needs a different kind of engineering.

the pipeline was green.

i had built a good pipeline. tested, automated, observable, everything a devops engineer is supposed to build.

the model was wrong.

those two facts coexisted for six weeks without contradiction because i was measuring the wrong things. devops taught me to measure whether the system is running. what i needed to measure was whether the system was right.

those are not the same question.

the rocket emoji fired. the predictions rotted. the dashboard said nothing.