<p>Over the last few years, AI has seen significant growth in utilization and effectiveness in organizations. Its applications range from predictive analytics for more efficient and accurate business operations, to machine vision for direct interactions with the world, and natural language processing (NLP) for more effective interaction with technology through language. <a href="https://www.accenture.com/us-en/insights/artificial-intelligence/ai-investments">Accenture reports</a> that 84% of surveyed executives believe they need to leverage AI to achieve their growth objectives.</p>
<p>With this fast growth in a relatively new and maturing technology comes a lot of responsibility and ethical concerns. <a target="_blank" href="https://www.capgemini.com/news/ai-and-ethics-2020-report/">Capgemini found</a> that 70% of customers expect transparency and fairness in their AI interactions. Additionally, regulators are beginning to tighten regulations around AI transparency and fairness. The European Union recently released a set of AI regulations that aim to minimize the harmful impacts of AI systems, and the FTC announced plans to go after companies selling biased algorithms (<a target="_blank" href="https://www.technologyreview.com/2021/04/21/1023254/ftc-eu-ai-regulation-bias-algorithms-civil-rights/">MIT Tech Review</a>).</p>
<p>The increases in ethical concerns are justified and driven by AI use cases that increasingly involve highly sensitive settings such as healthcare, judicial systems, and finance. Without proper tuning and understanding of the potential ethical risks, AI can lead to the reproduction and amplification of historical bias and discrimination. For example, <a target="_blank" href="https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm">ProPublica investigated the COMPAS algorithm</a> which was used to inform sentencing for criminals by predicting the likelihood of an individual to re-offend. They discovered that black defendants were significantly more likely to receive false-positive recommendations suggesting they would re-offend when they would not while white defendants were more likely to receive false-negative recommendations suggesting they would not re-offend. </p>
<p>Similarly, OpenAI’s <a target="_blank" href="https://github.com/openai/gpt-3">GPT-3</a> NLP transformer model with 175 billion parameters has demonstrated significant bias in generating sentences that include <a target="_blank" href="https://arxiv.org/pdf/2101.05783v1.pdf">religion</a>. The model has also shown bias toward other groups, including gender and race which is discussed in the <a target="_blank" href="https://arxiv.org/pdf/2005.14165.pdf">original paper</a>.  </p>
<p>As AI becomes more ubiquitous, it’s crucial to ensure the responsible use of AI. If we are to trust these systems to inform important and sensitive decisions, they cannot be biased or include historical stereotypes.</p>
<h2 id="what-is-algorithmic-bias-">What is Algorithmic Bias?</h2>
<p>Algorithmic bias is pretty prominent in the news, as seen in <a target="_blank" href="https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G">gender bias</a> in hiring decisions favoring men, online advertisements that reproduce redlining on a map (<a target="_blank" href="https://www.theguardian.com/technology/2019/mar/28/facebook-ads-housing-discrimination-charges-us-government-hud">housing discrimination</a>), and healthcare algorithms that include <a target="_blank" href="https://science.sciencemag.org/content/366/6464/447">racial bias</a>. All of these examples clearly show a group being treated differently by the model than another group.</p>
<p>Many metrics exist for formally defining algorithmic bias. The basic definition is that an algorithm treats groups in the data differently. The groups we are concerned about, such as race, gender, religion, and pregnancy or veteran status are called protected or sensitive characteristics. When analyzing an algorithm, it’s necessary to understand what groups are potentially most vulnerable to harmful impacts of the model.</p>
<p>There are two categories of metrics to determine the bias and fairness of an algorithm: fairness by representation and fairness by error.</p>
<h3 id="fairness-by-representation">Fairness by Representation</h3>
<p>These metrics evaluate the outcomes of the model for different groups. For example, these metrics could look at the proportion of men who get called back for an interview versus women for a hiring algorithm.</p>
<h3 id="fairness-by-error">Fairness by Error</h3>
<p>Fairness by error metrics observe the difference in the proportion of error rates made for each group. For example, in the COMPAS algorithm bias described above, the false-positive rate for black defendants was significantly higher than other groups, and the false-negative rate for white defendants was significantly lower.</p>
<h3 id="causes-of-algorithmic-bias">Causes of Algorithmic Bias</h3>
<p>Bias in an algorithm is caused by the underlying data. Simple models like linear regression and more complex transformer models are not biased on their own; they produce outputs based on the data provided to them. Below are five causes of bias in data.</p>
<ol>
<li><strong>Skewed dataset:</strong> A dataset may have different proportions of data for each group or training examples for each group which may lead to biased model outputs.</li>
<li><strong>Tainted examples:</strong> The data may have unreliable labels or historical bias may exist in the data.</li>
<li><strong>Limited Features:</strong> The collected features may be unreliable or not informative for specific groups due to bad data collection practices.</li>
<li><strong>Samples Size:</strong> AI algorithms struggle to perform well with smaller datasets.</li>
<li><strong>Proxy Features:</strong> Some features can leak information about protected variables, even if the protected variable is not included in the data. <a target="_blank" href="https://arxiv.org/pdf/1707.08120.pdf">Research has shown</a> that variables like zip code, sports activity, and the university a person attended can all indirectly inform a model about protected variables which can lead to algorithmic bias.</li>
</ol>
<h2 id="what-are-mitigation-techniques-that-can-be-used-">What are mitigation techniques that can be used?</h2>
<p>We now have a good formal understanding of what algorithmic bias is and what it is caused by. There are two potential avenues for mitigating algorithmic bias: technical techniques and AI governance. Most likely a combination of these two techniques will yield the best results in reducing the presence of bias in an algorithm.</p>
<h3 id="technical-mitigation-techniques">Technical Mitigation Techniques</h3>
<p>Within technical techniques, there are three approaches: pre-processing techniques which reduce bias in the original data before training, in-processing methods that mitigate bias during model training, and post-processing algorithms that reduce bias by modifying predictions.</p>
<p><a target="_blank" href="https://core.ac.uk/download/pdf/81728147.pdf">This paper</a> describes four common pre-processing techniques used in the literature in the introduction:</p>
<ul>
<li><strong>Suppression:</strong> remove a variable that correlates most with a sensitive variable</li>
<li><strong>Massaging:</strong> Relabel some observations in the data in an efficient way using a ranker to remove discrimination in the input data</li>
<li><strong>Reweighting:</strong> Assign weights to each tuple in the training data to remove discrimination</li>
<li><strong>Sampling:</strong> Sample from each group to get a more balanced representation of each group using either uniform sampling or preferential sampling by duplicating or skipping observations.</li>
</ul>
<p>Other techniques like <a target="_blank" href="https://krvarshney.github.io/pubs/CalmonWVRV_nips2017.pdf">this paper’s</a> probabilistic formulation and <a target="_blank" href="http://sorelle.friedler.net/papers/kdd_disparate_impact.pdf">this paper</a>, which describes two “repair procedures,” provide good methods of removing bias as well in the pre-processing stage.</p>
<p>In-processing techniques focus on removing bias during training. These techniques typically include fairness constraints which impact the optimization problem and lead to more fair results. Examples discussing in-process methods include this one looking at <a target="_blank" href="https://arxiv.org/pdf/1806.06055.pdf">fairness constraints</a> and this one looking at <a target="_blank" href="https://www.kamishima.net/archive/2012-p-ecmlpkdd-print.pdf">fairness-aware classification</a>.</p>
<p>Finally, post-processing algorithms modify model outputs to reduce bias. These algorithms can either change predictions for privileged or unprivileged groups as seen by <a target="_blank" href="https://arxiv.org/pdf/1709.02012.pdf">Pleiss et al.</a> and <a target="_blank" href="https://arxiv.org/pdf/1610.02413.pdf">Hardt et al.</a> or they reduce bias when an algorithm output is close to the decision boundary (where it is less sure about the result) as seen in <a target="_blank" href="https://mine.kaust.edu.sa/Documents/papers/ICDM_2012.pdf">this paper</a> on decision theory.</p>
<p>For any of these techniques, it is essential to discuss with all stakeholders involved the trade-offs in accuracy and the risks being taken by changing the underlying data or model results. These techniques can lead to predicting outcomes that have not been seen historically or are not seen in the data.  </p>
<h3 id="governance-and-accountability">Governance and Accountability</h3>
<p>There are a lot of benefits to using AI in organizations. However, as seen throughout the blog post, ethical risks posed by AI systems lead to organizational risks such as reputational damage, which lead to loss of revenue. No firm wants to have a headline saying they used a biased algorithm to make business decisions that negatively impacted a group of customers or employees.</p>
<p>Proper AI governance is conducted through comprehensive risk management. The specific use case of the AI decision and the associated impact assessment for each stakeholder needs to be identified. Understanding the potential monetary and human risks associated with the use case is important. Once the magnitude of each risk is accurately assessed for each stakeholder, the likelihood for these repercussions should be calculated.</p>
<p>AI governance best practices include comprehensive checklists for stakeholders, automated and manual testing, business standards compliance testing, detailed reports, and assertions from individuals and groups who are directly responsible for the model. For more, details make sure to download our white paper.</p>
<h2 id="responsible-ai-relies-on-data-literacy">Responsible AI Relies on Data Literacy</h2>
<p>Scaling responsible AI in organizations requires the development of data literacy within the organization. Successful AI governance includes multiple stakeholders engaging with and adopting a single framework. Understanding the scope of projects, how they are deployed and governed, and their impact and potential risks is a requirement for responsible AI for all involved stakeholders. <a target="_blank" href="https://www.accenture.com/_acnmedia/PDF-92/Accenture-AFS-Responsible-AI.pdf">Accenture argues</a> that data literacy builds trust and allows for responsible AI by including diverse perspectives which lead to consistently better outcomes.</p>
<p>By increasing baseline data literacy in an organization, subject matter experts who do not consistently interact with AI systems can include their expertise in these systems and communicate with technical stakeholders more effectively. This improved communication leads to more trust throughout the organization for all stakeholders involved.</p>
<p>The important aspects of data literacy for promoting responsible AI in an organization include an understanding of the following areas: the data science and machine learning workflow, different data roles, the flow of data through an organization, the distinction between various types of AI systems, and evaluation metrics for machine learning models.</p>
<h2 id="increase-baseline-data-literacy-to-scale-responsible-ai">Increase Baseline Data Literacy to Scale Responsible AI</h2>
<p>Training all stakeholders in an organization is required to promote responsible AI. Avoiding the associated risks with algorithmic bias is necessary to allow an organization to successfully take advantage of the many benefits of leveraging AI in business contexts. Data literacy is the key to successful communication between stakeholders, trust within the organization, and diverse solutions to be explored for responsible AI. The EU is allocating €1.1 trillion to their <a target="_blank" href="https://ec.europa.eu/social/BlobServlet?docId=22832&amp;langId=en">Skills Agenda</a>, with a heavy emphasis on AI skills and data literacy for regulators, demonstrating the importance of these tools in organizations. Many programs teach the fundamentals of data literacy. For more details on how data literacy can fuel responsible AI — check out our latest white paper.  </p>
<br><p><center><a target="_blank" href="https://www.datacamp.com/business/demo"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603718763/banner_style_2_m573ef.png" /></a></p>


https://www.datacamp.com/community/blog/the-case-for-responsible-ai

We recently released a report co-written by DataRobot’s VP of Trusted AI Ted Kwartler, Global AI Ethicist Haniyeh Mahmoudian, and Managing Director of AI Ethics Sara Khatry. Here’s a run-down of what to expect.

The Case for Responsible AI

<p>The concept of data-driven organizations has been a staple of the technology industry. Tech giants like Amazon, Google, and Netflix operate with data at the core of their business model, and they epitomize how data can be leveraged to drive decision-making towards successful business growth.</p>
<p>With growing data availability and significant decreases in costs for data processing and storage, opportunities to harness data to solve business challenges have extended beyond the technology industry. Finance, healthcare, and insurance companies have recognized this and have embarked on data transformation journeys to establish competitive business advantages.</p>
<p>While leveraging cutting edge-machine learning techniques and bridging the data science talent gap remains top of mind, an equally crucial aspect of a data strategy is the creation of a data governance plan. This article aims to dive into the definition of data governance, as well as the tools and technology that can drive successful governance programs.</p>
<h2 id="what-is-data-governance-">What is Data Governance?</h2>
<p>In a <a target="_blank" href="https://www.datacamp.com/resources/webinars/how-data-governance-enables-scalable-data-science">recent webinar</a>, Aaren Stubberfield, Data Scientist at Microsoft, outlined the opportunities and best practices of data governance, where he started by addressing the fundamental question of what data governance is.
Data governance, as defined by the <a target="_blank" href="https://www.dama.org/">Data Management Association (DAMA)</a>, is the exercise of authority and control (<strong>planning</strong>, <strong>monitoring</strong>, and <strong>enforcement</strong>) over the management of data assets.</p>
<p><strong>(1) Planning:</strong>  Creating rules that datasets need to conform to (e.g. format standardization of date fields e.g. YYYY-MM-DD)</p>
<p><strong>(2) Monitoring:</strong> Measuring compliance to rules set in place in the Planning phase (e.g. tracking the percentage of ‘Date’ fields with missing values)</p>
<p><strong>(3) Enforcement:</strong> Remediation actions in the event of data rule breaches based on level of urgency e.g. immediate corrective action if critical data fields such as social security number are found to contain major discrepancies</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1623443176/survey_1_nyksns.png" /></center>


<h2 id="what-does-a-data-governance-framework-look-like-">What does a Data Governance Framework look like?</h2>
<p>There are different types of data governance frameworks out there, but they tend to have these four common elements: Data Policies and Procedures, Datasets, Data Stewards and Stakeholders, and Technology.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1623443176/survey_2_zuwgvr.png" /></center><br>

<p>Data stewards are the people who develop governance policies and procedures for the datasets in the organization. Data governance tools are vital enablers for the stewards to do their best work, and they form a key part of the governance framework.</p>
<h2 id="what-are-the-tools-and-technology-used-for-data-governance-">What are the Tools and Technology used for Data Governance?</h2>
<p>Data governance tools are necessary to automate governance operations and data stewardship efforts since the large volumes of data render it impossible for these activities to be done manually. They are also able to integrate with different IT products and datasets along with comprehensive data cataloging, thereby extending their capabilities across an organization’s entire data management system.
To meet the burgeoning demand for these tools, there has been a proliferation of data quality solutions in the market. A <a target="_blank" href="https://www.gartner.com/en/documents/3988016/magic-quadrant-for-data-quality-solutions">review by Gartner</a> analyzed the data governance market and came up with Magic Quadrants to classify vendors offering data governance solutions.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1623443176/survey_3_pxeg3p.png" /></center><br>

<p>Given the growing importance of data governance, numerous data management tools offer governance capabilities as part of their systems. SolutionsReview released the <a target="_blank" href="https://solutionsreview.com/data-management/data-management-vendor-map-a-guide-to-the-best-data-management-tools/&amp;sa=D&amp;source=editors&amp;ust=1622129720378000&amp;usg=AOvVaw0Sld2arQpyWTWtd6n2BAiX">Data Management Vendor Map</a>, where they classified data management solutions into <strong>three categories</strong>: Data Quality Tools, Master Data Management, and Data Management for Analytics.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1623443176/survey_4_vo8yu6.png" /></center><br>

<h4 id="-1-data-quality-tools">(1)   Data Quality Tools</h4>
<p>Data quality is defined as the overall utility of the data and its ability to be easily processed and analyzed for other purposes. In order to achieve and sustain good data quality, the governance tools need to perform key functions such as standardizing, profiling, parsing, cleansing, and monitoring. Besides keeping the data clean and well-organized, these tools are also expected to support data processing across the organization’s entire data system.
<p>Examples:<br>
(i)          <a target="_blank" href="https://www.talend.com/products/data-integrity-governance/">Talend</a><br>
Talend’s flagship product is the <strong>Talend Data Fabric</strong>, which includes its data integrity and governance capabilities as part of a single unified cloud platform for trusted data. This includes metadata management to data lineage, as well as collaborative data stewardship solutions.</p>
<p>(ii)          <a target="_blank" href="https://www.sas.com/en_us/software/data-quality.html">SAS</a><br>
The <strong>SAS Data Quality</strong> product provides data quality management capabilities across different kinds of databases and data architecture deployments. It includes the essential functions of data cleansing, entity resolution, as well as a unified web-based console to monitor data quality jobs.</p>
<p>(iii)             <a target="_blank" href="https://www.informatica.com/products/data-quality/informatica-data-quality.html">Informatica</a><br>
Informatica offers an expansive portfolio of data tools in various deployments. Their <strong>Informatica Data Quality</strong> product provides users with a rich set of data transformation capabilities, while also allowing them to build and review business rules without relying on IT.</p>
<h4 id="-2-master-data-management">(2)   Master Data Management</h4>
<p>Master data refers to the consistent and uniform collection of core company-wide data points and comprises key components such as customers, leads, suppliers, employees, accounts, and more. Underpinning an effective data quality control strategy is the setup of solid master data management. It is only then that the enterprise’s growing data assets (along with its metadata) can be properly structured and consolidated in a central repository to support optimal decision making.
<p>Examples<br>
(i)             <a target="_blank" href="https://www.ataccama.com/platform/master-data-management">Ataccama</a><br>
The <strong>Ataccama Platform</strong> product serves as an augmented data management platform that includes MDM as one of its modules. It also includes other relevant modules like data quality and metadata management, and is fully integrated for different types of deployment.</p>
<p>(ii)            <a target="_blank" href="https://www.winshuttle.com/products/enterworks/master-data-management/">Enterworks</a><br>
The <strong>EnterWorks Platform</strong> is a multi-domain platform that creates a central repository of reliable, up-to-date master data consolidated across all enterprise applications with powerful tools to improve data quality and governance. Furthermore, all administrative and governance functions across the multiple domains can be achieved from a single user interface.</p>
<p>(iii)          <a target="_blank" href="https://www.riversand.com/products/mdm/">Riversand</a><br>
Riversand’s <strong>Master Data Experience Management</strong> platform provides a multi-domain cloud-native, unified software-as-a-service (SaaS) platform for all MDM use cases. It aims to eliminate data silos and support compliance with data governance rules by creating a single, accurate, trustworthy source of master data, along with comprehensive views of business-critical data.</p>
<h4 id="-3-data-management-for-analytics">(3)   Data Management for Analytics</h4>
<p>With the multitude of governance activities involved in maintaining data quality, enterprises may want to look for comprehensive integrated data platforms instead of standalone solutions. Data management for analytics solutions are comprehensive systems that integrate with analytics software to oversee data analytics such as relational and non-relational analytical processing, business intelligence, and machine learning.
<p>Examples<br>
(i)             <a target="_blank" href="https://www.collibra.com/data-governance">Collibra</a>
The <strong>Collibra Data Governance</strong> product is a cloud-based platform that helps enterprises establish a common data asset understanding and collaborate in a central location. This includes a suite of services such as policy manager, reference data and business glossaries. All these help to generate trusted data for powerful business analytics to be built upon.</p>
<p>(ii)          <a target="_blank" href="https://erwin.com/solutions/metadata-management/">erwin</a>
The erwin Data Intelligence Suite (<strong>erwin DI</strong>) serves as a unified software platform that lets users create automated and curated enterprise data catalogs complete with data models and on-demand lineage. This drives agile and well governed data preparation and analytics, with integrated business glossaries for organization-wide data literacy.</p>
<p>(iii)            <a target="_blank" href="https://www.alation.com/product/platform/">Alation</a>
The <strong>Alation Platform</strong> empowers analysts and business users with an open and intelligent platform that supports a wide variety of metadata management applications to data catalog and governance. Furthermore, its integrated Analytics product serves as a one-stop shop for productive self-service analytics.</p>
<h2 id="how-to-maximize-the-value-of-these-tools-">How to maximize the value of these tools?</h2>
<p>It is important to remember that tools are just one lever in the strategy towards data democratization. To make full use of these data governance tools, the people in the organization are key.
This is because good data governance is a team effort that requires humans to design and implement the agenda and policies and then leverage technology to automate and monitor these procedures optimally.
To achieve this, organizations need to start focusing on data literacy upskilling so that everyone has the skills to work with data to do their best work.</p>
<p>Learn more about <a target="_blank" href="https://www.datacamp.com/groups/business">DataCamp for Business</a>:<br>
<i>DataCamp for Business provides an interactive learning platform for companies that need to upskill and reskill their people on data skills—on everything from data literacy and data science to data engineering and machine learning. Join 1,600+ companies and 80% of the Fortune 1000 who use DataCamp to upskill their teams.</i></p>
<br><p><center><a target="_blank" href="https://www.datacamp.com/business/demo"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603718763/banner_style_2_m573ef.png" /></a></p>


https://www.datacamp.com/community/blog/a-survey-into-data-governance-tools

In a recent webinar, we discussed how data governance is a key component of an organization’s data strategy and enables it to harness the full value of data. For a data governance plan to succeed, it is important that the right tools and technology are employed.

In a recent webinar, we discussed how data governance is a key component of an organization’s data strategy and enables it to harness the full value of data.

A Survey Into Data Governance Tools

<p><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1623167783/spotlight-iryna-datacamp_vjypef.png" /></center><br></p>
<p>Once an accountant in Ukraine, Iryna’s move to Belgium and a new culture inspired a fresh start. After learning web design from <a target="_blank" href="http://www.hackyourfuture.be/">Hack Your Future Belgium</a>, one of DataCamp Donates’ partner organizations, she now works with us full-time as a frontend developer. Read on to find out how, for one woman, data got democratized.</p>
<h4 id="let-s-start-with-the-present-what-do-you-do-at-datacamp-">Let’s start with the present: what do you do at DataCamp?</h4>
<p>I started last March as an intern in software development, and incidentally, today is my first day working in this role full-time. Right now, I work with the commercial team on all DataCamp’s signed-out pages: that is, when you visit the website all the pages you can reach from there without signing in is our work.</p>
<h4 id="what-does-your-job-typically-involve-">What does your job typically involve?</h4>
<p>I start my day by checking new tickets (requests from colleagues that relate to my job). Then we have our daily meeting—although this gets repeated later in the day because of the time gap between us, the London team, and the team in the US. Usually, I have a couple of other meetings with other team members to ask work-related questions—other than that, I press on with all the jobs on my plate.
Working from home was a bit strange at first since I started my internship completely remotely, so I saw two people from the office on my first day when I went in to pick up my laptop, and have never seen my teammates in real life since. Still, the communication and everything else works so well here at DataCamp that it never really feels lonely.</p>
<h4 id="going-back-in-time-where-are-you-from-">Going back in time, where are you from?</h4>
<p>I’m from Ukraine and I moved to Belgium one year ago. That actually inspired my career change. Originally in Ukraine I was an accountant, and I moved to Belgium since my husband was working there. When I moved here it was a new beginning, and I asked myself if I would like to transition into something else. In the end, I chose to transition to web development, and I’m really happy that I made that choice—I love this job, there’s so much variety to it, so much room to be creative and so much room for me to learn!</p>
<h4 id="where-did-your-interest-in-tech-start-">Where did your interest in tech start?</h4>
<p>When I moved I was kind of out of my depth. My English wasn’t perfect, I didn’t speak any Flemish or French, so when I went to an integration course to learn these languages I also had an opportunity to take courses in tech too—HackYourFuture ran one. They would later become one of DataCamp Donates’ main partners. They were offering tech courses to newcomers and refugees in Belgium for free—I didn’t really know if it was for me, but I decided to give it a try. I ended up spending the rest of the year in that course, and afterward decided to pursue it as a career. I came across the internship on DataCamp, and now that’s me working in development full-time! My first project, incidentally, was DataCamp Donates itself, when they were working on launching their own site. Effectively, my role was using Squarespace to make the site match the main site’s design.</p>
<h4 id="is-there-anything-you-ll-be-working-on-in-the-future-that-particularly-excites-you-">Is there anything you’ll be working on in the future that particularly excites you?</h4>
<p>I really like the fact that I work with DataCamp Donates to expand what they do to more countries. For me, that quite literally hits close to home. But every day there are new projects coming in, and the main thing that excites me is that I work with such amazing people from whom I can learn a lot.</p>
<h4 id="any-advice-for-how-datacamp-donates-can-better-itself-since-you-ve-experienced-what-they-offer-as-a-learner-">Any advice for how DataCamp Donates can better itself since you’ve experienced what they offer as a learner?</h4>
<p>Keep doing what you do. There are a lot of countries like, for example, Ukraine itself where development is beginning to take root. So, the education and lifestyle are great in the capital but in smaller cities and the countryside not so much. As a result people really have an interest in upgrading their skills, and parents are determined to ensure their children get the best education.</p>
<p><i>Would you like to join Iryna at DataCamp? We’re currently hiring in New York, London, Leuven, Belgium, and remote locations. Check out our <a target="_blank" href="https://www.datacamp.com/jobs">open roles</a>.</i></p>
<p><i>Hack Your Future Belgium is an open education nonprofit that prepares newcomers for a career in web development, with a special focus on refugees and women. They facilitate integration and address the shortage of qualified workforce in the IT sector. Learn more and get involved at <a target="_blank" href="https://hackyourfuture.be/">www.hackyourfuture.be</a>.</i></p>

https://www.datacamp.com/community/blog/new-hire-spotlight-iryna

Learn how a DataCamp Donates participant landed a job with DataCamp.

New Hire Spotlight: Iryna

<p>Click the image below to download our infographic on data maturity within your organization. For a deeper dive into data maturity within your organization, explore our <a target="_blank" href="https://www.datacamp.com/resources/whitepapers/your-organizations-guide-to-data-maturity">exclusive guide</a>.</p>
<br><center><a target="_blank" href="https://s3.amazonaws.com/assets.datacamp.com/email/other/Your+Organization&#39;s+Guide+to+Data+Maturity_infographicc.pdf"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1622825612/Your_Organization_s_Guide_to_Data_Maturity_infographicc_qknvlp.pdf" /></a></center>

<br><p><center><a target="_blank" href="https://www.datacamp.com/business/demo"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603718763/banner_style_2_m573ef.png" /></a></p>


https://www.datacamp.com/community/blog/infographic-your-organizations-guide-to-data-maturity

Our infographic explains why your organization’s data maturity matters. Learn actionable steps to take to go from data reactive to data fluent. 

Your Organization's Guide to Data Maturity [Infographic]

https://www.datacamp.com/community/blog/the-past-and-present-of-data-science

Sergey Fogelson deep dives into how data science has evolved over the past decade, the largest challenges facing data teams today, and the importance of learning SQL and data democratization. 

The Past and Present of Data Science

<p>In January 2021, the World Economic Forum released a report on upskilling for shared prosperity. It was a call to action stating that closing the skills gap could add up between $5-6.5 trillion USD to global GDP by 2030.</p>
<p>Now more than ever, the learning and development function is critical to companies who want to develop organization-wide data literacy. This May, DataCamp hosted a panel discussion with Kerrie O’Sullivan, EMEA Training Manager, and Sheil Naik, Senior Technical Trainer from the Bloomberg Global Data Team, alongside Marcus Robertson, Global Curriculum Lead at NatWest Group, to discuss the state of L&amp;D for data-driven organizations, best practices designing digital-first learning experiences, and how they foster a learning culture in their organizations.</p>
<br><center><a target="_blank" href="https://www.datacamp.com/resources/webinars/learning-and-development-for-the-data-driven-age"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1622224538/image_47_uqpsuy.png" /></a></center><br>

<p>In the panel discussion, two broad themes emerged. One on the strategic importance of L&amp;D when providing organizations insights on skill (and data skills) transformation, and second on best practices for operationalizing data skill transformation. Here are some key takeaways from the panel:</p>
<h2 id="the-strategic-importance-of-learning-and-development">The strategic importance of learning and development</h2>
<ul>
<li><p><strong>On the role L&amp;D plays in the organization —</strong> The panel highlighted how learning and development teams can drive the skills transformation vision throughout the organization. This includes collaborating with business groups to provide guidance and thought leadership on the skills of the future, and designing learning environments that are flexible and equitable to all learners. This is especially important as jobs are quickly transforming, and organizations and communities are starting to engage in what can be considered the biggest reskilling and upskilling programs we’ve seen in decades.</p>
</li>
<li><p><strong>The role that data skills play in upskilling and reskilling efforts —</strong> Data skills sit amongst the top key competencies to upskill and reskill on for L&amp;D leaders on the panel. For example at both Bloomberg and NatWest, these skills can range from upskilling on Python, SQL, and business intelligence tools, to data literacy and understanding where to access data and how to make sense of it. It’s important to note that, data skills are a methodology for solving business problems, so it’s highly important that data skills are injected into already existing subject matter expertise. These data skills whether conceptual or technical, allow subject matter experts to break down organizational siloes by becoming conversational with data experts or creating analysis for themselves.</p>
</li>
<li><p><strong>How building and sustaining data skills is different from other skills —</strong> Since data skills are relatively nascent and can be daunting at first, creating a narrative for data upskilling to ensure learners are excited about what they can achieve with data is a prerequisite to foster data skills. Moreover, as organizations upskill on data, there are various roles with differing data literacy that need to interact with data. As such both Bloomberg and NatWest rolled out learning programs that emphasize what is possible to achieve with data and how to integrate data into daily work. Marcus highlighted the importance of providing a safe installation-free environment for experiential learning on technical tools like Python and R such as DataCamp. Interestingly, since data skills are technically oriented and are applied with tools such as Jupyter Notebooks and IDEs, the impact of learning outcomes is more easily measurable than other skills. An example could be drawn from how <a target="_blank" href="https://www.datacamp.com/resources/webinars/thrive-in-a-data-driven-age">Sheil leveraged data analysis with Python</a>, to analyze the effectiveness of his blended learning program on data analysis with Python, which uses DataCamp for foundational material alongside in-house training delivered and administered by Sheil.</p>
</li>
</ul>
<h2 id="best-practices-for-operationalizing-data-skill-transformation">Best practices for operationalizing data skill transformation</h2>
<ul>
<li><p><strong>The switch to a digital-first L&amp;D function —</strong>  Like most teams and organizations, both Bloomberg and NatWest L&amp;D teams transitioned to a digital workplace throughout 2020. For example for Bloomberg, this meant redesigning data upskilling programs to be digital-first by making them shorter, more interactive, and by encouraging learners to interact with experts within the organization. Moreover, transitioning to digital meant the breaking down of barriers between regional L&amp;D teams and their learners within both organizations. This means regional L&amp;D teams sharing best practices and the development of a culturally richer training experience for facilitators and learners alike.</p>
</li>
<li><p><strong>Best practices creating a learning culture —</strong> A key theme mentioned throughout the panel is the importance of senior leadership when galvanizing excitement around learning. While this includes senior executive sponsorship for data upskilling, it also includes middle-management having conversations with direct reports around their upskilling opportunities and what that means for their teams. Marcus mentioned that a balancing act middle-managers face is that they are asked to deliver results, while also facilitating the skills development of their team. A key point to get across, however, is that teams can only deliver results if they develop their skills. Moreover, Kerrie highlighted the importance of promoting and providing structured training programs that are discoverable, paced, and clearly, articulate learning outcomes, can jumpstart the conversation, and create excitement around learning.</p>
</li>
</ul>
<p>This only scratches the surface of what was covered in the panel — in the <a target="_blank" href="https://www.datacamp.com/resources/webinars/learning-and-development-for-the-data-driven-age">on-demand recording</a>, Sheil, Kerrie, and Marcus deep-dive into best practices aligning business objectives with learning outcomes, how to conduct a thorough needs analysis for skills, how they envision L&amp;D programs will be affected by the return to the office, and more. Make sure to register to catch it all.</p>
<p><i>Learn more about <a target="_blank" href="https://www.datacamp.com/groups/business">DataCamp for Business</a>:<br>
DataCamp for Business provides an interactive learning platform for companies that need to upskill and reskill their people on data skills. With topics ranging from data literacy, and data science to data engineering and machine learning, over 1,600 companies trust DataCamp for Business to upskill their talent.</i></p>


https://www.datacamp.com/community/blog/learning-development-data-driven-age

We recently hosted a panel with Bloomberg and NatWest Group on the importance of data literacy, and how to go about operationalizing data skill transformation programs. 

Learning and Development for the Data Driven Age

Event Recap: Learning and Development for the Data Driven Age



<p>The&nbsp;<a href="http://pandas.pydata.org">Pandas</a>&nbsp;library is one of the most preferred tools for data scientists to do data manipulation and analysis, next to <a href="http://matplotlib.org">matplotlib</a> for data visualization and <a href="http://www.numpy.org">NumPy</a>, the fundamental library for scientific computing in Python on which Pandas was built.&nbsp;</p>

<p>The&nbsp;fast, flexible, and expressive Pandas data structures are designed to&nbsp;make real-world data analysis significantly&nbsp;easier, but this might not be immediately the case for those who are just getting started with it. Exactly because there is so much functionality built into this package that the options are overwhelming.</p>

<p>That&#39;s where this Pandas cheat sheet might come in handy.&nbsp;</p>

<p>It&#39;s a quick guide through the basics of Pandas that you will need to get started on wrangling your data with Python.&nbsp;</p>

<p>As such, you can use it as a handy reference if you&nbsp;are just beginning their data science journey with Pandas or, for those of you who already haven&#39;t started yet, you can just use it as a guide to make it easier to learn about and use it.&nbsp;</p>

<p><a href="http://datacamp-community-prod.s3.amazonaws.com/f04456d7-8e61-482f-9cc9-da6f7f25fc9b" target="_blank"><img alt="Pandas Cheat Sheet for Data Science in Python" src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1621255989/Pandas_Cheat_Sheet_for_Data_Science_in_Python_hqirhp.png" /></a>

<p>The Pandas cheat sheet will guide you through the basics of the Pandas library, going from the data structures&nbsp;to I/O, selection, dropping indices or columns, sorting and ranking, retrieving basic information of the data structures you&#39;re working with to applying functions and data alignment.</p>

<p>In short, everything that you need to kickstart your data science learning with Python!</p>

<p>Do you want to learn more? Start the <a href="https://www.datacamp.com/courses/intermediate-python-for-data-science/">Intermediate Python For Data Science</a> course for free now or try out our <a href="https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python/">Pandas DataFrame tutorial</a>!&nbsp;</p>

<p>Also, don&#39;t miss out on our <a href="https://www.datacamp.com/community/blog/pandas-cheat-sheet-python">Pandas Data Wrangling cheat sheet</a> or our other <a href="https://www.datacamp.com/community/data-science-cheatsheets">data science cheat sheets</a>.&nbsp;</p>

<p><strong>(Click above to download a printable version or read the online version below.)</strong></p>
<h2>Python For Data Science Cheat Sheet: Pandas Basics</h2>
<p>Use the following import convention:</p>
<pre><code>&gt;&gt;&gt; import pandas as pd</code></pre>
<h3>Pandas Data Structures</h3>
<div>
<style scoped>
  table, th, td {
   border: 1px solid black;
    color: black;
    font-size: 12px;
max-width:25%;
white-space:nowrap;
}

h3 {font-weight:normal;
}
h4 {
	font-weight: lighter;
}
  </style>
<h3>Series</h3>
<p>A one-dimensional labeled array capable of holding any data type </p>
<pre><code>&gt;&gt;&gt; s = pd.Series([3, -5, 7, 4],  index=['a',  'b',  'c',  'd'])</code></pre>

<table class="fixed">
<tr>
  <td>A</td>
  <td>3</td>
  <tr>
</tr>
<td>B</td>
<td>5</td>
<tr>
</tr>
<td>C</td>
<td>7</td>
<tr>
</tr>
<td>D</td>
<td>4</td>
<tr>
</tr>
</table>

<h3>DataFrame</h3>
<p>A two-dimensional labeled data structure with columns of potentially different types</p>



<pre><code>&gt;&gt;&gt; data = {'Country': ['Belgium',  'India',  'Brazil'],

'Capital': ['Brussels',  'New Delhi',  'Brasilia'],

'Population': [11190846, 1303171035, 207847528]}</code>
<code>&gt;&gt;&gt; df = pd.DataFrame(data,columns=['Country',  'Capital',  'Population'])</code></pre>
<br>
<table>
<tr>
  <th></th>
  <th>Country</th>
<th>Capital</th>
<th>Population</th>
</tr>
<tr>
    <td>1</td>

  <td>Belgium</td>
<td>Brussels</td>
<td>11190846</td>
</tr>
<tr>
  <td>2</td>
<td>India</td>
<td>New Delhi</td>
<td>1303171035</td>
</tr>
<tr>
  <td>3</td>
<td>Brazil</td>
<td>Brasilia</td>
<td>207847528</td>
</tr>
</table>


<p>Please note that the first column <strong>1</strong>,<strong>2</strong>,<strong>3</strong> is the index and <strong>Country</strong>,<strong>Capital</strong>,<strong>Population</strong> are the Columns.
<h3>Asking For Help</h3>
<pre><code>&gt;&gt;&gt; help(pd.Series.loc)</code></pre>


<h2>I/O</h2>
<h3>Read and Write to CSV</h3>
<pre><code>&gt;&gt;&gt; pd.read_csv('file.csv', header=None, nrows=5)
&gt;&gt;&gt; df.to_csv('myDataFrame.csv')</code></pre>


<h3>Read multiple sheets from the same file</h3>
<pre><code>&gt;&gt;&gt; xlsx = pd.ExcelFile('file.xls')
&gt;&gt;&gt; df = pd.read_excel(xlsx,  'Sheet1')</code></pre>


<h3>Read and Write to Excel</h3>
<pre><code>&gt;&gt;&gt; pd.read_excel('file.xlsx')
&gt;&gt;&gt; df.to_excel('dir/myDataFrame.xlsx',  sheet_name='Sheet1')</code></pre>


<h3>Read and Write to SQL Query or Database Table</h3>
<p>(read_sql()is a convenience wrapper around read_sql_table() and read_sql_query())</p>
<pre><code>&gt;&gt;&gt; from sqlalchemy import create_engine
&gt;&gt;&gt; engine = create_engine('sqlite:///:memory:')
&gt;&gt;&gt; pd.read_sql(SELECT * FROM my_table;, engine)
&gt;&gt;&gt; pd.read_sql_table('my_table', engine)
&gt;&gt;&gt; pd.read_sql_query(SELECT * FROM my_table;', engine)</code></pre>



<pre><code>&gt;&gt;&gt; df.to_sql('myDf', engine)</code></pre>


<h2>Selection</h2>
<h3>Getting </h3>
<p>Get one element</p>
<pre><code>&gt;&gt;&gt; s['b']
-5</code></pre>

<p>Get subset of a DataFrame</p>
<pre><code>&gt;&gt;&gt; df[1:]
Country     Capital   Population
1  India    New Delhi 1303171035
2  Brazil   Brasilia  207847528</code></pre>

<h3>Selecting', Boolean Indexing and Setting</h3>
<h3>By Position</h3>
<p>Select single value by row and and column</p>
<pre><code>&gt;&gt;&gt; df.iloc([0], [0])
'Belgium'
&gt;&gt;&gt; df.iat([0], [0])
'Belgium'</code></pre>


<h3>   By Label</h3>
<p>Select single value by row and column labels</p>
<pre><code>&gt;&gt;&gt; df.loc([0],  ['Country'])
'Belgium'
&gt;&gt;&gt; df.at([0],  ['Country'])
'Belgium'</code></pre>

<h3>By Label/Position</h3>
<p>Select single row of subset of rows</p>
<pre><code>&gt;&gt;&gt; df.ix[2]
Country      Brazil
Capital    Brasilia
Population  207847528</code></pre>

<p>Select a single column of subset of columns</p>
<pre><code>&gt;&gt;&gt; df.ix[:, 'Capital']
0     Brussels
1    New Delhi
2     Brasilia</code></pre>

<p>Select rows and columns</p>
<pre><code>&gt;&gt;&gt; df.ix[1, 'Capital']
'New Delhi'</code></pre>


<h3>Boolean Indexing</h3>
<p>Series s where value is not &gt;1</p>
<pre><code>&gt;&gt;&gt; s[~(s &gt; 1)]</code></pre>
<p>
s where value is &lt;-1 or &gt;2 </p>
<pre><code>&gt;&gt;&gt; s[(s &lt; -1) | (s &gt; 2)]</code></pre>
<p>Use filter to adjust DataFrame</p>
<pre><code>&gt;&gt;&gt; df[df['Population']&gt;1200000000]</code></pre>
<h3>
  Setting</h3>
<p>Set index a of Series s to 6</p>
<pre><code>&gt;&gt;&gt; s['a'] = 6</code></pre>

<h2>Dropping</h2>
<p>Drop values from rows (axis=0)</p>
<pre><code>&gt;&gt;&gt; s.drop(['a',  'c'])</code></pre>
<p>
Drop values from columns(axis=1)</p>
<pre><code>&gt;&gt;&gt; df.drop('Country', axis=1) </code></pre>

<h3>Sort and Rank</h3>
<p>Sort by labels along an axis</p>
<pre><code>&gt;&gt;&gt; df.sort_index()</code></pre>
<p>Sort by the values along an axis</p>
<pre><code>&gt;&gt;&gt; df.sort_values(by='Country') </code></pre>

<p>Assign ranks to entries</p>
<pre><code>&gt;&gt;&gt; df.rank()</code></pre>



<h2>Retrieving Series/DataFrame Information</h2>
<h3>Basic Information</h3>
<p>(rows, columns)</p>
<pre><code>&gt;&gt;&gt; df.shape</code></pre>

<p>
Describe index </p>
<pre><code>&gt;&gt;&gt; df.index</code></pre>

<p>
Describe DataFrame columns</p>
<pre><code>&gt;&gt;&gt; df.columns</code></pre>
<p>
Info on DataFrame</p>
<pre><code>&gt;&gt;&gt; df.info()</code></pre>
<p>
Number of non-NA values</p>
<pre><code>&gt;&gt;&gt; df.count()</code></pre>


<h3>Summary</h3>
<p>Sum of values</p>
<pre><code>&gt;&gt;&gt; df.sum()</code></pre>
<p>
Cumulative sum of values </p>
<pre><code>&gt;&gt;&gt; df.cumsum()</code></pre>
<p>
Minimum/maximum values</p>
<pre><code>&gt;&gt;&gt; df.min()/df.max()</code></pre>


<p>Minimum/Maximum index value</p>
<pre><code>&gt;&gt;&gt; df.idxmin()/df.idxmax() </code></pre>


<p>Summary statistics</p>
<pre><code>&gt;&gt;&gt; df.describe()</code></pre>
<p>
Mean of values</p>
<pre><code>&gt;&gt;&gt; df.mean()</code></pre>
<p>
Median of values</p>
<pre><code>&gt;&gt;&gt; df.median()</code></pre>


<h2>Applying Functions</h2>
<pre><code>&gt;&gt;&gt; f = lambda x: x*2</code></pre>

<p>Apply function</p>
<pre><code>&gt;&gt;&gt; df.apply(f)</code></pre>

<p>
Apply function element-wise</p>
<pre><code>&gt;&gt;&gt; df.applymap(f)</code></pre>


<h3>Internal Data Alignment</h3>
<p>NA values are introduced in the indices that don't overlap:</p>
<pre><code>&gt;&gt;&gt; s3 = pd.Series([7, -2, 3],  index=['a',  'c',  'd'])
&gt;&gt;&gt; s + s3
a     10.0
b     NaN
c     5.0
d     7.0</code></pre>



<h3>Arithmetic Operations with Fill Methods</h3>
<p>You can also do the internal data alignment yourself with the help of the fill methods:</p>
<pre><code>&gt;&gt;&gt; s.add(s3, fill_value=0)
a    10.0
b    -5.0
c    5.0
d    7.0
&gt;&gt;&gt; s.sub(s3, fill_value=2)
&gt;&gt;&gt; s.div(s3, fill_value=4)
&gt;&gt;&gt; s.mul(s3, fill_value=3)</code></pre>
</div>


https://www.datacamp.com/community/blog/python-pandas-cheat-sheet

A quick guide to the basics of the Python data analysis library Pandas, including code samples.

A quick, free cheat sheet to the basics of the Python data analysis library Pandas, including code samples.

Pandas Cheat Sheet for Data Science in Python

https://www.datacamp.com/community/blog/from-predictions-to-decisions

Dan Becker deep dives into the intersection of decision sciences and machine learning, how data teams can go from experimentation and deployment to providing value at scale for organizations, and more!

community

datacamp

Official Blog