The Past, Present, and Future of MLOps

Many Machine Learning Projects Fail

Many organizations are investing heavily in AI and machine learning. However, that does not mean organizations are reaping the value of machine learning within their organizations. According to VentureBeat, 87% of data science projects do not make it to production. In their keynote at VentureBeat’s Transform 2019, Deborah Leff, CTO for Data Science and AI at IBM, and Chris Chap, Senior Vice President of Data and Analytics at Gap, argue that many teams lack the right leadership and support for conditions that lead to success in these projects.

Leff and Chap describe data quality, data democratization, and organizational flow issues as key hurdles that many organizations do not have the proper tools or structure to overcome. Data teams that cannot access high-quality and relevant data or work in the correct collaborative environment are not able to succeed in these projects, regardless of their competence. They also describe issues with model decay where some organizations implement models that stop working without noticing for over three years. This was especially clear with the onset of Covid-19, where lockdowns forced completely different behaviors onto consumers and many retail analytics models broke in production.

Another important consideration in ML projects is the rise in the use of open source technologies for testing and producing models. These include open-source programming languages like Python, and its host of libraries like Scikit-learn, pandas, PySpark, and more. Changing the version of one of these in the production environment can greatly impact the success of the project or make it stop working entirely. Managing these versions and open source frameworks requires an organized system to be effective at scale.

The need for MLOps

Machine Learning Model Operationalization Management (MLOps) is a way of standardizing and streamlining a machine learning project’s lifecycle management. In their book, Dataiku discusses three key factors why MLOps is becoming so important to enterprises: there are many dependencies, organizations lack data literacy and common data language, and data scientists don’t have engineering skills.

Many dependencies: Data is a reflection of real-world behaviors and processes, and naturally changes over time (e.g., toilet paper buying habits during Covid-19). As data changes over time, which happens constantly, models in production must continuously adapt to these changes so that organizations keep solving the original problem they intended to solve with these models.

No common data language: In the machine learning life cycle, projects involve people from a diverse set of teams with different backgrounds. This includes data scientists, data engineers, but also functional subject matter experts. These groups do not share the same skills or data fluency to communicate about data in the same way, and their alignment is crucial to deploy models in production and evaluate their success.

Data scientists do not have engineering skills: Data scientists typically specialize in building models and evaluating them, not in writing applications to be deployed. As organizations rely on more models and more complex models, data scientists are facing a trade-off between developing and iterating on models and maintaining them in production.

Why MLOps is Useful

Machine learning models are being increasingly deployed within business processes. This increase in volume necessitates the use of MLOps to mitigate risks associated with these models, deploy these models responsibly, and use them at scale.

There are many risks associated with deploying a model:

• The model can be unavailable for some time. Technology can potentially be unreliable in the production environment. If an organization relies heavily on these models, it can face significant challenges when the technology becomes unavailable.

• The model can return poor results for a sample. It may perform very well for most groups but can give poor results for one group. This needs to be corrected or understood to avoid making poor business decisions based on model outputs for the underperforming group. This poor performance can also lead to ethical issues related to AI models underperforming for some protected variables (such as race, age, gender, etc.)

• The model can become less fair or accurate in the future. As the world changes, the data that was used in training becomes less representative of the world the model is attempting to understand. This can lead to poor performance on future data.

• Data science talent may leave the organization over time. Maintaining these models requires technical skills and bandwidth data scientists may not always have to ensure perfect reliability of these models.

These risks scale depending on the impact the model has on the organization and the probability of one of these risk events occurring. Many of these risks can be mitigated by ensuring the training data is a good reflection of data that will be seen in the production environment and ensuring the production environment is stable as models are very sensitive to changes. MLOps is increasingly necessary for mitigating risks associated with the increasing number of machine learning models being deployed for important decisions.

As the use of machine learning models in decision-making becomes more prevalent, it is increasingly important to ensure they are deployed responsibly. There are a lot of causes of algorithmic bias that need to be avoided when decisions are automated through AI systems. Failure to successfully minimize bias can lead to punishment through government regulation or loss of demand and profits for a brand. To responsibly deploy AI systems, models must be trained on compliant and unbiased data sources, have interpretable and explainable results, and accountability throughout the organization where it is easy to find where in the development pipeline something potentially went wrong to fix it effectively. To deploy AI models responsibly, organizations require a streamlined ML model life cycle.

Deploying more and more models requires MLOps to be successful. It is necessary to have a streamlined process to keep track of versioning, to understand model performance versus retrained model performance, and to ensure that a model continues to perform well over time. Without these MLOps, it is very difficult to successfully add value and business impact with these models.

The Roles of an MLOps Program

While MLOps is nascent and is still evolving rapidly, it is necessary to have strong people in an organization in the following roles: Subject Matter Experts, Data Scientists, Data Engineers, and Software and DevOps Engineers.

Subject matter experts are business-oriented individuals who understand and inform how to measure and continually evaluate the success of models during training and after deployment. For a successful MLOps program, they must develop an easy-to-understand business performance measurement and a feedback loop for distinguishing models that are not performing relative to their business expectations. Additionally, risk subject matter experts are an important part of the MLOps program. Their goal is to minimize and clarify the risk a company faces due to the deployment of ML models in production.

Course Recommendations: Understanding Data Topics

Data scientists are responsible for building the models which address the business questions generated by the subject matter experts. These models should be deployable in production environments with production data. Data scientists should ensure these models are easily and safely deployable in production, have tests for quality assurance and potential improvements, and have easy visibility of all deployed models from one location.

Course Recommendations: Machine Learning Scientists Career Track (R/Python), Image Processing in Python, Machine Learning with PySpark

Data engineers are responsible for creating data pipelines for these models. For a successful MLOps program, performance for all deployed models must be easily visible as well as full detail for each pipeline to see where addressable issues may exist.

Finally, software and DevOps engineers are in charge of helping with the successful integration of these models with the other non-machine learning applications. For a successful MLOps practice, they must integrate data engineering best practices to ensure that these models can work in parallel with existing applications reliably, continuously, and seamlessly.

Course Recommendations: Data Engineer with Python Career Track

There are a lot of key roles that must be successfully utilized for a successful MLOps program. Successful deployment of hundreds of models, at scale, requires members outside of the data experts, including subject matter experts and software engineers, to generate positive business outcomes. Organizations striving to develop successful MLOps programs must aim to build a common data language across all stakeholders through upskilling and continuous learning.

Learn more about DataCamp for Business:

DataCamp for Business provides an interactive learning platform for companies that need to upskill and reskill their people on data skills. With topics ranging from data literacy, and data science to data engineering and machine learning, over 1,600 companies trust DataCamp for Business to upskill their talent.