AINEWS
Most enterprise machine learning systems perform best on the day they ship, then drift because they cannot learn from the outcomes they cause. Smeet H. Patel has named the structural problem the closed loop gap.
Most enterprise machine learning systems perform best on the day they ship. After that, they stop getting better, because the outcomes they cause land in a data warehouse the model that triggered them never reads.
The pattern is structural. The model is fine. A churn model (a system that predicts which customers are about to leave) trains on a past world, scores customers, triggers an intervention (a retention offer, a call, a discount), and the resulting outcome (did the customer stay, did they spend more) is logged for analysts and dashboards. The model that caused the intervention never sees that result. It runs the same scoring formula tomorrow, next month, next year, regardless of whether the offers it triggered actually worked. In practice, this looks like a retention team mailing discount offers to customers who would have stayed anyway, while the customers who actually needed the offer have already canceled.
Smeet H. Patel, a practitioner who has worked on prescriptive retention at a large enterprise telecommunications provider described in the source as operating across tens of millions of accounts, calls the structural flaw the "closed-loop gap" (OneIndia). The term distinguishes systems that learn from their own consequences from systems that learn only from training data and decay.
The underlying phenomenon is not unique to retention. Peer-reviewed research published in Scientific Reports documents "temporal quality degradation" in AI models, in which performance decays as the relationship between inputs and ground-truth labels shifts over time (PMC9270447). Independent industry data from the ML observability vendor NannyML puts the empirical scale of the problem at 91% of deployed models degrading after launch (NannyML). A separate peer-reviewed venue carries a research article reaching the same conclusion from a different angle (ScienceDirect). Secondary commentary has been reinforcing the picture for at least a year (AIGL, Swept AI).
Patel adds the closed-loop reading. Most enterprise pipelines are not just drifting. They are not architecturally capable of learning from their own outputs at all. A churn model produces a probability score; a separate treatment system decides what to offer; a separate channel optimizer picks email, SMS, or call; a separate attribution team tries to measure whether the offer worked. These five workstreams usually have five owners, five dashboards, and five release cadences. Structurally, they are one system. Breaking them apart opens the loop.
Patel formalized that reading in 2025 as RetentionOS, a framework that integrates churn prediction, uplift-based treatment assignment (picking offers based on how the customer will respond rather than how likely they are to leave), channel optimization, causal outcome attribution (a method for estimating whether the intervention actually caused the customer to stay), and feedback-loop integration into a single self-improving loop (OneIndia). He describes it, "to the best of his knowledge," as the first peer-reviewed contribution to treat those five stages as one continuously self-improving system rather than as disconnected components.
The portable diagnostic is not "buy RetentionOS" or "buy a new model." It is a question any reader can take into a vendor pitch or an internal review: can this system learn from its own outcomes, or only from training data? If only from training data, it is by design drifting the day after it ships, regardless of how good it looked on day one. The strongest counterargument, that some enterprise teams are already wiring these loops and just not talking about it, survives the diagnostic by pointing at the org chart, not the tools: the teams running retention, treatment, channel, attribution, and feedback as separate workstreams with separate budgets are the teams whose systems will keep drifting no matter which model they swap in.
The next milestone worth watching is whether any large enterprise publicly reorgs its ML pipeline to put those five stages under one owner with one feedback budget. The teams that do will be the ones that, a year from now, do not need a vendor to tell them their models are degrading.