paid course

Cleaning Data in Python

This course will equip you with all the skills you need to clean your data in Python.

4 hours
17 Videos
58 Exercises
46,412 Participants
4,700 XP

Create Your Free Account

By continuing you accept the Terms of Use and Privacy Policy, that your data will be stored outside of the EU, and that you are 16 years or older.

Course Description

A vital component of data science involves acquiring raw data and getting it into a form ready for analysis. In fact, it is commonly said that data scientists spend 80% of their time cleaning and manipulating data, and only 20% of their time actually analyzing it. This course will equip you with all the skills you need to clean your data in Python, from learning how to diagnose your data for problems to dealing with missing values and outliers. At the end of the course, you'll apply all of the techniques you've learned to a case study in which you'll clean a real-world Gapminder dataset!

This course will equip you with all the skills you need to clean your data in Python.

Course Outline

So you've just got a brand new dataset and are itching to start exploring it. But where do you begin, and how can you be sure your dataset is clean? This chapter will introduce you to the world of data cleaning in Python! You'll learn how to explore your data with an eye for diagnosing issues such as outliers, missing values, and duplicate rows.

The ability to transform and combine your data is a crucial skill in data science, because your data may not always come in one monolithic file or table for you to load. A large dataset may be broken into separate datasets to facilitate easier storage and sharing. Or if you are dealing with time series data, for example, you may have a new dataset for each day. No matter the reason, it is important to be able to combine datasets so you can either clean a single dataset, or clean each dataset separately and then combine them later so you can run your analysis on a single dataset. In this chapter, you'll learn all about combining data.

In this final chapter, you'll apply all of the data cleaning techniques you've learned in this course towards tidying a real-world, messy dataset obtained from the Gapminder Foundation. Once you're done, not only will you have a clean and tidy dataset, you'll also be ready to start working on your own data science projects using the power of Python!

Here, you'll learn about the principles of tidy data and more importantly, why you should care about them and how they make subsequent data analysis more efficient. You'll gain first hand experience with reshaping and tidying your data using techniques such as pivoting and melting.

Here, you'll dive into some of the grittier aspects of data cleaning. You'll learn about string manipulation and pattern matching to deal with unstructured data, and then explore techniques to deal with missing or duplicate data. You'll also learn the valuable skill of programmatically checking your data for consistency, which will give you confidence that your code is running correctly and that the results of your analysis are reliable!

1
Exploring your data
Free

So you've just got a brand new dataset and are itching to start exploring it. But where do you begin, and how can you be sure your dataset is clean? This chapter will introduce you to the world of data cleaning in Python! You'll learn how to explore your data with an eye for diagnosing issues such as outliers, missing values, and duplicate rows.
View Chapter Details Play Chapter Now
Tidying data for analysis

Here, you'll learn about the principles of tidy data and more importantly, why you should care about them and how they make subsequent data analysis more efficient. You'll gain first hand experience with reshaping and tidying your data using techniques such as pivoting and melting.
View Chapter Details Play Chapter Now
Combining data for analysis

The ability to transform and combine your data is a crucial skill in data science, because your data may not always come in one monolithic file or table for you to load. A large dataset may be broken into separate datasets to facilitate easier storage and sharing. Or if you are dealing with time series data, for example, you may have a new dataset for each day. No matter the reason, it is important to be able to combine datasets so you can either clean a single dataset, or clean each dataset separately and then combine them later so you can run your analysis on a single dataset. In this chapter, you'll learn all about combining data.
View Chapter Details Play Chapter Now
Cleaning data for analysis

Here, you'll dive into some of the grittier aspects of data cleaning. You'll learn about string manipulation and pattern matching to deal with unstructured data, and then explore techniques to deal with missing or duplicate data. You'll also learn the valuable skill of programmatically checking your data for consistency, which will give you confidence that your code is running correctly and that the results of your analysis are reliable!
View Chapter Details Play Chapter Now
Case study

In this final chapter, you'll apply all of the data cleaning techniques you've learned in this course towards tidying a real-world, messy dataset obtained from the Gapminder Foundation. Once you're done, not only will you have a clean and tidy dataset, you'll also be ready to start working on your own data science projects using the power of Python!
View Chapter Details Play Chapter Now

Daniel Chen

Data Science Consultant at Lander Analytics

Daniel is a Software Carpentry instructor and a doctoral student in Genetics, Bioinformatics, and Computational Biology at Virginia Tech, where he works in the Social and Decision Analytics Laboratory under the Biocomplexity Institute. He received his MPH at the Mailman School of Public Health in Epidemiology and is interested in integrating hospital data in order to perform predictive health analytics and build clinical support tools for clinicians. An advocate of open science, he aspires to bridge data science with epidemiology and health care.

Course Instructor

Daniel Chen

Data Science Consultant at Lander Analytics

Collaborator(s)

Hugo Bowne-Anderson
Yashas Roy

Join over 3,210,000 others learning to leverage the power of data with DataCamp!

Start Course For Free

If you type... We will search for...

Press enter to see all search results

Courses

Tracks

Instructors

Cleaning Data in Python

Create Your Free Account

Course Description

Course Outline

Exploring your data

Diagnose data for cleaning

Loading and viewing your data

Further diagnosis

Exploratory data analysis

Calculating summary statistics

Frequency counts for categorical data

Visual exploratory data analysis

Visualizing single variables with histograms

Visualizing multiple variables with boxplots

Visualizing multiple variables with scatter plots

Combining data for analysis

Concatenating data

Combining rows of data

Combining columns of data

Finding and concatenating data

Finding files that match a pattern

Iterating and concatenating all matches

Merge data

1-to-1 data merge

Many-to-1 data merge

Many-to-many data merge

Case study

Putting it all together

Exploratory analysis

Visualizing your data

Thinking about the question at hand

Assembling your data

Initial impressions of the data

Reshaping your data

Checking the data types

Looking at country spellings

More data cleaning and processing

Wrapping up

Final thoughts

Tidying data for analysis

Tidy data

Recognizing tidy data

Reshaping your data using melt

Customizing melted data

Pivoting data

Pivot data

Resetting the index of a DataFrame

Pivoting duplicate values

Beyond melt and pivot

Splitting a column with .str

Splitting a column with .split() and .get()

Cleaning data for analysis

Data types

Converting data types

Working with numeric data

Using regular expressions to clean strings

String parsing with regular expressions

Extracting numerical values from strings

Pattern matching

Using functions to clean data

Custom functions to clean data

Lambda functions

Duplicate and missing data

Dropping duplicate data

Filling missing data

Testing with asserts

Testing your data with asserts