Monitoring Machine Learning in Python
This course covers everything you need to know to build a basic machine learning monitoring system in Python
Start Course for Free3 Hours11 Videos38 Exercises
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.Training 2 or more people?Try DataCamp For Business
Loved by learners at thousands of companies
Course Description
Learn how to monitor your ML Models in Python
Monitoring machine learning models ensures the long-term success of your machine learning projects. Monitoring can be very complex, however, there are Python packages to help us understand how our models are performing, what data has changed that might have led to a drop in performance, and give us clues on what we need to do to get our models back on track. This course covers everything you need to know to build a basic monitoring system in Python, using the popular monitor package, nannyml.Understand the optimal monitoring workflow
Model monitoring is not only about simply calculating model performance in production. Unfortunately, it is not that easy. Especially when labels are hard to come by. This course will teach you about the optimal monitoring workflow. It will ensure that you always catch model failures, avoid alert fatigue, and quickly get to the root of the issue.Learn how to find the root cause of model performance issues
Another important component to model monitoring is root cause analysis. This course will dive into how to use data drift detection techniques to get to the root cause of model performance issues. You will learn how to use both univariate and multivariate data drift detection techniques to uncover potential root causes of model issues.For Business
Training 2 or more people?
Get your team access to the full DataCamp library, with centralized reporting, assignments, projects and moreIn the following Tracks
Machine Learning Engineer
Go To Track- 1
Data Preparation and Performance Estimation
FreeIn this chapter, you will be introduced to the NannyML library and its fundamental functions. Initially, you will learn the process of preparing raw data to create reference and analysis sets ready for production monitoring. As a practical example, you will investigate predicting the tip amount for taxi rides in New York. Toward the end of the chapter, you will also discover how to estimate the performance of the tip prediction model using NannyML.
What is NannyML?50 xpKey features of NannyML50 xpLoad the dataset100 xpData preparation for NannyML50 xpReference or analysis period?100 xpLoading and splitting the data100 xpCreating reference and analysis set100 xpPerformance estimation50 xpSpecify the algorithm and problem type50 xpInterpreting results50 xpCBPE and DLE workflow100 xpPerformance estimation for tip prediction100 xp - 2
Monitoring Performance and Business Value
In this chapter, you will be introduced to realized performance calculators used when ground truth becomes available. You will learn about the more advanced methods for handling results, including filtering, plotting, converting them to data frames, chunking, and establishing custom thresholds. Lastly, you'll apply this knowledge to calculate the business value of a model trained on the hotel booking dataset.
When labels are available50 xpWhen performance estimation is off50 xpComparing estimated and realized performance100 xpWorking with calculated and estimated results50 xpDifferent chunking methods100 xpModifying the thresholds100 xpInteracting with results100 xpBusiness value calculation and estimation50 xpBusiness value calculation50 xpDrop in monetary value50 xpBusiness calculation for hotel booking dataset100 xp - 3
Root Cause Analysis and Issue Resolution
Having detected the performance degradation in the hotel booking model, you will now learn how to identify the underlying issue causing it. In this chapter, you will be introduced to multivariate and univariate drift detection methods. You will also learn how to identify data quality issues and how to address the underlying problems you detect.
Multivariate drift detection50 xpIdentifying relevant drifts50 xpDrift in hotel booking dataset100 xpUnivariate drift detection50 xpUnivariate drift detection for hotel booking dataset100 xpRanking the univariate results100 xpVisualizing drifting features100 xpData quality and statistic checks50 xpData quality checks100 xpSummary statistics100 xpIssue resolution50 xpWhat is the resolution?50 xpShould you do nothing or not?50 xpImplementing a monitoring workflow100 xpCongratulations50 xp
For Business
Training 2 or more people?
Get your team access to the full DataCamp library, with centralized reporting, assignments, projects and moreIn the following Tracks
Machine Learning Engineer
Go To TrackCollaborators
Prerequisites
Machine Learning Monitoring ConceptsHakim Elakhrass
See More
Co-founder and CEO of NannyML
Hakim is one of the co-founders of nannyML, one of the most popular open source machine learning model monitoring libraries. He has almost a decade of data science experience. Hakim holds a Masters Degree in Bioinformatics from the KU Leuven.
Maciej Balawejder
See More
Data Scientist at NannyML
Maciej is a data scientist at NannyML with a background in math and mechanical engineering.
What do other learners have to say?
Join over 13 million learners and start Monitoring Machine Learning in Python today!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.