Discrimination and bias in a model is unethical and the impact can be catastrophic to a business. Unfortunately, this can simply be that when you built your model that you didn't see bias in certain demographics because you didn't have enough of them in your data. Over time, more and more of a certain demographic enters your data that the model can't properly make good decisions for. That is extremely detrimental from a financial and business perspective, because if your model is discriminating against a certain segment, then you're obviously not doing the best for the company. Worst of all, it’s not fair to the people you're making predictions about.
Actually putting models into production is what will set you apart as a data scientist. it's an important skill that, unfortunately, not many data scientists have. They should also really have a grasp of the business impact of the model. A model is more than just its performance or technical metrics. Why are you building this model? What value does it add and how is that value changing over time? How is it impacting other departments? Obviously data scientists need technical skills, but they must also have a deep intuition about why they are doing what they are doing.
Whether or not you know what actually happens in the real world after the prediction, understanding model performance is still challenging from both an engineering perspective and a data science perspective.
Data scientists need to cultivate a thorough understanding of a model’s potential business impacts as well as the technical metrics of the model.
Making machine learning tools open source builds trust with users and enables a community-based approach for getting feedback.